Designing Effective Online Surveys
Arguably the key benefit of conducting online quantitative surveys is that they are a highly effective way of collecting valuable data from a targeted number of respondents in a relatively short period of time.
However when designing an online survey, one needs to keep in mind that it is a self administered interview from the respondents perspective. It therefore follows that a well-structured questionnaire will help you to obtain the robust measures and actionable insights you are seeking.
Let's check out some good housekeeping tips when it comes to online survey design!
Define your objectives and keep your focus
First and foremost, you need to clearly define your research objectives, having well-defined objectives will help you to determine which questions should make the cut and more importantly those that should be discarded, thus keeping the survey as concise as possible. Define your exact hypotheses and then develop a question set to test and confirm it. When designing the questionnaire, it is also important to visualise in advance the pending data outputs and results, this allows you to hit the ground running in your reporting as soon as you are in receipt of your data file.
Diversity of the sample selection
The current penetration rate for internet usage in Australia stands at 89.8% (Source : Internet World Stats), however it’s safe to assume the vast majority have yet to register themselves to become members of a market research only panel. So to get the most out of your online sampling, it is important to understand the online behaviour of your target audience. The general demographic of internet users in countries with high internet penetration rates would likely be closely aligned to that of the general "offline" population. Use of a larger sample is wise if a suspected diversity exits between the general and online population. To ensure your findings are more statistically sound, specific quota control should also be applied to ensure that the final sample accurately mirrors the key demographics of your target audience.
A courteous welcome and thank-you page
"What's in it for me?" - Is the question many respondents would ask before deciding whether to accept and complete your survey. Therefore from the very first page we need to engage the respondents, and make them feel appreciated. Start by thanking them for their time, identify who you are and then clearly set out the reasons why the research is being conducted, including also the expected interview length. Although appropriate incentives might entice them initially, they will be more motivated by knowing that their opinions are being listened to and could ultimately make a difference to how that product or service actually develops. Don't forget a simple "Thank You!" message upon completion of the survey, including those who are screened out or at quota full.
Getting your respondents into the flow of the survey
Maximise respondent engagement and stimulate a genuine interest by asking interesting questions early in the survey. Ease the respondents into the survey by starting with some broad, generalized questions as a form of 'warm-up'. Whilst always ensuring that any screener questions are inserted at the very beginning, and from here funnel down to ask the more specific qualifying questions. Front load any important questions that require more considered responses, as there could be the potential for the respondents to become disengaged or fatigued during the later part of the survey if it proves too long.
The sequence of the research topics is equally important. Having a logical flow for all questions or topics can avoid the respondent being confused about the direction of the interview and thus becoming less engaged. It is also best to place profiling demographic questions at the back end of the survey as these questions required little effort to answer.
Keep your survey as concise as possible to avoid respondent fatigue and increase completion rates
The average online attention span of an adult is considered to be less than 15 minutes. Therefore it's always advised to be honest about how long the survey will take to complete, stating it clearly in the email invitation or in the welcome page. Stating a range (for example 10 – 15 minutes) is advisable when there are questions that involve routing or piping. If you find yourself with a long survey (25 minutes +) due to the inclusion of multiple research objectives, it is worth considering a multiple of shorter surveys as an alternative, also by evaluating each subject separately you will optimise the respondents focus on that topic alone by reducing the known issues associated with respondent fatigue.
It is considered best practice to use only open-ended questions to capture spontaneous feedback where it matters most. Although open responses can elicit extremely valuable insights, it is best to limit the number of open-ended questions as they always take more time and energy for respondents to express their opinion, so again use only where they are truly needed.
Simplify your survey – brevity , clarity and consistency
Make your survey easier to understand
- Construct clear and direct questions by using the language that respondents will understand.
- Use shorter sentences and make your questions as specific as possible. When there is more text displayed on the screen, the fewer words respondents actually read.
- If you wish to focus respondent attention on certain key word(s), consider highlighting them by bolding it or using different font and colours.
Provide a self-explanatory, clear and jargon free answer list for easy completion
To pre-define relevant response options for each question: avoid providing too many choices; ambiguous (overlapping categories like "1-2 units only" and "2 – 5 units"); or 'double barreled' answers (such as "friendly and knowledgeable customer service staff") as this could confuse the respondents and slow down the completion time.
If relevant, leave open the possibility of other response options, such as "other please specify" to find out what you don't know or allow an opt-out response (e.g. "prefer not to answer", "don't know" or "does not apply") where respondents are unable to respond in the way that's been prescribed. Any forced answers introduce agreement, dissatisfaction and therefore inaccuracies in your final data set.
Use consistent rating scale throughout the survey
Another key point is that unbalanced scales are inappropriate and almost always lead to biased results, i.e. presenting rating scales in different directions and of different sizes (positive to negative or vice versa) within the same survey. For example:
- In one question, using a 10-point liking scale starting from 'Do not like it at all' as code 1 to 'Like it very much' as code 10; and
- In another question using a 5-point rating scale starting from a positive to negative statement, i.e. from 'Extremely appealing' as code 1 to 'Very unappealing' as code 5.
With rating scales, it's also best to avoid neutral response options where possible, or not allowing a respondent to choose a "not applicable" or "don't know" option. Design a good question with valid and thorough response options, this ensures your data is valid and representative at the back end.
Avoid cumbersome and complex question types
Although grid questions are commonly used to ask a series of repetitive questions, it can be tedious and complex for respondents to fill out. Respondents can easily become disengaged when presented with a long battery of questions and response options. This can lead to contrived agreement as they will just click through as quickly as possible to get it finished, resulting in the straight lining of responses and ultimately junk data.
There are ways to make it easy for respondents to complete a survey:
- Brand logos, product images can be used to make questions more visually engaging and intuitive.
Also the use of dropdown questions is ideal for a very long choice list such as countries or postcodes. It also works well for displaying correlated questions together on the same screen. For example:
- Asking respondents to indicate the make, model year and type of the car
- Determining respondents' socio-economic status by asking them their highest level of education, income level and occupation
These dropdowns give the impression that the survey will take a shorter time to complete as it reduces the number of 'clicks' to continue to the next survey screen, and is more visually appealing as it removes clutter from the screen.
Survey Engagement – Enhancing the Survey Experience
Visual layout and formatting
The physical layout of a question, and the number of questions displayed on screen at one time are also important considerations as they can affect interview completion rates and ultimately data collection. One question per screen is always recommended as it prevents respondents from reading ahead; avoiding the possibility of them altering their responses after seeing subsequent questions and before submitting their response.
Depending on the respondent's browser and screen resolution, they may need to use the scroll bars (vertical or horizontal) in order to see everything. It is essential to choose the question structure, limit the amount of text and/or images of the question and avoid requiring respondents to scroll horizontally (left to right) in a survey screen. They can get confused or frustrated and then dropout since they see it as a daunting task to complete.
Use a variety of question types
Like it or not there is a high potential for respondents to become bored when taking your survey, thus they begin to respond in a repetitive manner, meaning the response become ill considered with data quality the first thing to suffer. Keep your respondents engaged by utilising a variety of question types and where possible presented in a visually engaging way.
Minimise potential bias in your research data
Knowing any potential bias enables you to eliminate them before launching the survey.
- Online surveys allow you to set randomisation rules with ease across questions, response listings and concepts, thus eliminating order bias.
- Use skip logic and conditional routing to ensure the relevancy of your questions, this can be easily and efficiently programmed when using an online survey to collect your data.
Creating a great survey is an skill that requires attention to many details. How questions are being asked and the survey layout may influence how people understand and answer your questions. Engaged respondents will answer your questions fully and accurately, thus resulting in better insights and a higher quality of data. This is because they feel that they are genuinely contributing to something meaningful and sharing their opinions in a constructive way, not just contributing their personal time for a quick reward.
To conclude, below is a summary of some important considerations to keep respondents engaged throughout the interview process :
- Communicate to respondents that their answers matter
- Keep the language plain and simple, jargon free and avoid ambiguous questions and instructions
- Make the survey as short as possible, consistent with meeting the clearly understood research objectives
- Relevancy and accuracy, avoid the temptation to ask 'nice to know' questions
- Wording style, type and logical question sequence are important to achieve
- Asking questions in an unbiased way will give you reliable data to analyse
- Check for bias, randomizing the order of your questions and response choices
- Remember, keeping your respondent engaged is the key to better research data
For more advice simply get in touch with our Client Services Team (firstname.lastname@example.org) who can provide you with consultation and support on constructing an effective online questionnaire and research design.
Author: Megan Kuek – i-Link Research Solutions