How to Collect High-Quality Data in Survey Research: A Comprehensive Guide

Obtaining high-quality data during survey research is paramount to drawing accurate conclusions. But how can you ensure that the answers provided by your respondents are accurate and true? In this guide, we delve into the significance of quality checks, optimized survey design, and how representative sampling and ethical survey practices enhance data quality. Furthermore, this article provides valuable insights into what reliable data is and how it can be obtained from both paid and free sources.

What does “reliable” or “high-quality” data look like?

When it comes to survey research, “high quality” data refers to information obtained from survey participants that is representative, valid, and reliable.

• Representativeness: The extent to which the characteristics of your sample accurately reflect the characteristics of the wider population from which it is drawn, is referred to as “representativeness”. Data is considered representative when it adequately represents the full range of variation and diversity within your population of interest, allowing your findings to be applied beyond the surveyed population.
• Validity: Validity refers to the extent to which your survey measures what it is intended to measure. Valid data accurately reflects the concept or phenomenon that you are studying, and ensures that your survey accurately captures the intended construct, leading to meaningful and actionable insights.
• Reliability: Reliability refers to the consistency and stability of your measurements. A survey’s data is considered reliable if it yields the same results under consistent conditions. High reliability ensures that your survey produces stable and consistent results, which is crucial for tracking changes over time or comparing different groups.

These key attributes constitute high-quality data, and will ultimately lend credibility to your research findings. Now, the obvious next question is, “How can I collect high-quality data in survey research?”

Adding Quality Checks

The first and arguably most important way to attain high-quality data is by adding quality checks to your survey. Quality checks are essential safeguards that shield your data integrity against bad actors and careless respondents.

1. Qualification Checks

Qualification checks protect your data quality from individuals with ulterior motives. These bad actors may attempt to manipulate paid survey panels for financial gain, present themselves as someone they are not, or provide false answers to gain access to your incentives.

Types of qualification checks:

• Qualification Funnels

Offer various multiple choice questions with 10-12 options to see whether a participant truly qualifies to answer your survey.

Example: You’re looking for Dutch healthcare workers to answer your survey. Firstly, ask your respondents which country they reside in. List the Netherlands as an option. Secondly, ask the participant which languages they speak and include Dutch among your list of other languages. Then, ask which industry respondents are currently working in, and add healthcare as one of the answers. If the participant chooses all three of your desired answers, they qualify for your survey.

Additional Tips:

1. Ensure that respondents aren’t able to guess what the correct answers are from your survey’s title. For instance, don’t use the title “Survey for Dutch Healthcare Workers”, as this could lead to you screening in bad actors anyway, defeating the purpose of the qualification funnels.
2. Randomize the position you place your correct answer options in to prevent respondents from guessing the correct answer. For instance, if Question 1’s correct answer is ‘C’, make Question 2’s correct answer ‘A’.
3. If possible, randomize what your answer options (e.g. ‘A’, ‘B’, and ‘C’) represent for each participant that takes your survey. Some survey builders allow you to randomize your options per question, so that your answers will be displayed in a different order for every participant.

• Sample-Specific Questions

Ask questions that are designed to test your target audience's specific knowledge. 

Example: You’re looking for a United States citizen to take your survey. Ask your respondents to identify the non-state amongst a list of US states. If the participant incorrectly identifies a list item as a US state, they don’t qualify to answer your survey, as we expect a US citizen to have specific knowledge about the United States.

PRO TIPS: 

1. Add your qualification questions earlier on in your survey to prevent sincere participants from taking the time to answer questions, only to be disqualified in the end because they don’t match your selection criteria
2. If you’re setting up your survey with Qualtrics’ survey builder, make use of CAPTCHA verifications to keep those robots at bay.

2. Attention Checks 

Attention checks ensure that respondents are taking the necessary time and consideration to answer your questions to the best of their ability. As the name suggests, these checks assess whether a respondent is paying attention and not merely speeding through your survey for financial or other gain. 

Types of attention checks:

• Red Herring (“Trap”) Questions

Present questions with obvious correct answers to filter out disingenuous respondents.

Example: 

Question: “What color is a polar bear?”

A) Red; B) White; C) Black; D) Green

Disqualify any respondents who answered anything other than B) White.

• YEA-saying Counters

Include questions that require a negative response to identify participants who mindlessly agree with all statements.

Example:

Question: “To what degree do you agree with the following statement: ‘I can name every person who ever lived on earth’”

1 - Strongly Agree; 2 - Neither Agree nor Disagree; 3 - Strongly Disagree

Any person who chooses either option 1 or 2 should be disqualified from your survey results.

Did you know? You could also explicitly mention that the question you’re asking is an attention check. E.g., “This is an attention check. Please choose ‘Strongly Disagree’ as your answer.” If the participant fails to follow your instructions, they fail the attention check.

• Sand Traps

Incorporate open-ended questions to assess genuine engagement. Take note: Unless you’re making use of AI prompts to analyze your data for genuine answers, this is only recommended for surveys with smaller samples (100-200 respondents) to avoid you having to check a vast number of open-ended responses.

Example:

Question: “In your own words, how would you describe your experience with online grocery shopping?”

Use your own logic and discretion to determine if the respondent provided a sensical, thought-through answer. If the participant answered something like “Yes”, it’s safe to assume they were not paying attention when taking your survey.

• Manipulation Checks

Use questions related to your survey material to ensure participants are attentively completing your survey.

Example:

If you asked participants to read a short text titled “Dave and his dog, Spotty”, ask them what Dave’s dog was called. Anyone who fails to answer “Spotty” was not paying proper attention to the text.

PRO TIPS: 

1. Distribute your attention checks across your survey to check attentiveness at different points in the questionnaire, instead of clustering them altogether. 
2. Use a combination of attention check types to ensure only sincere, qualified respondents get to participate in your survey.

Optimizing Survey Design

A clear and well-structured survey design is fundamental to obtaining reliable data. Survey design refers to how and in which order questions are set up, as well as the format you use when doing so.

1. Align Questions with Your Research Goals

Survey questions should be designed with your specific research goals in mind. Think about the graphs, tables, or test results you want to report and what they might look like, and pose your questions to fit that end goal.

Example:

Research Goal: To understand consumer preferences for soft drinks among different age groups.

Reporting Goal: A bar graph showing the distribution of soft drink preferences among different age groups.

Relevant Questions: 

1. “Please select your age range: 18-25, 26-35, 36-45, 46-55, 56-65, Other (please specify)”
2. "Which of the following soft drink brands do you prefer? Please select all that apply: Coca-Cola, Pepsi, Sprite, Fanta, Other (please specify)."

Pro Tip: Ask the most important questions early in your survey, when participants are at the height of their attention span.

2. Maintain a Consistent Question and Answer Format

It’s also important to keep your questions and their formats consistent throughout your survey to prevent contradictions and ensure logical alignment. Maintain a uniform response format as far as possible. For example, if you’re using Likert scales to measure agreement or frequency, keep this consistent across your survey.

3. Pose Clear and Specific Questions

Furthermore, make sure that your questions are unambiguous, and avoid potential sources of misinterpretation. Use precise language and provide clear response options to reduce errors in participants’ responses.

Example:

Vague Question and Answer Options:

“How often do you exercise?” 

A) Never; B) Sometimes; C) Often; D) Always

The question above provides too much room for interpretation. What constitutes “exercise”? Some participants may believe that a 15-minute stroll counts as exercise, while others draw the line at a 60-minute circuit training session. Similarly, the answer options are very vague, as statements such as “sometimes” and “often” are very subjective and can easily lead to inconsistent data. 

Improved Question and Answer Options:

“How many times a week do you engage in at least 30 minutes of moderate-intensity exercise* (*at 50-70% of your maximum heart rate)?”

A) Less than once; B) Once; C) Two to three times; D) Four to six times; E) Every day

4. Collect Complete Answers

To prevent an incomplete dataset, make survey questions mandatory. Most survey builders (like our own free survey building tool) allow you to mark a question as required, meaning respondents cannot leave it blank. This ensures the completeness of your dataset. It’s also useful to include progress trackers to help participants track their completion status and prompt them to fill in any missed sections before submission.

Maintaining Transparent and Ethical Practices

Building trust between you and your respondents can greatly impact how truthful they are when participating in your survey. To ensure reliable data, clearly communicate your survey's purpose to respondents, assure their answers will remain confidential, ask for their consent, and adhere to ethical standards in survey research. This not only fosters a trust between you and your respondents that encourages them to take part in your study, but also encourages honest responses, which improves your data quality.

Making Use of Representative Sampling

Ensuring that your target group is representative is crucial to obtaining reliable and generalizable results. The following sampling strategies can improve your data’s representativeness:

1. Random Sampling

Utilize random sampling techniques to ensure that each member of the target population has an equal chance of being selected. This minimizes selection bias – the systematic exclusion from certain individuals or groups from a sample – and ensures a diverse representation.

Example:

To survey university students’ satisfaction with their study material, assign each student a unique number and use a random number generator to select participants for your survey. This guarantees that every student has an equal likelihood of being chosen to participate.

Take note: Achieving a perfectly random sample is often impossible due to inherent selection bias. This bias arises from individuals who voluntarily engage in online surveys, have the necessary access and technological proficiency to participate, and are motivated by financial incentives or other gains to take part.

2. Stratified Sampling

Divide the target population into relevant subgroups or “strata” based on key characteristics (e.g., age, gender, location). Then, randomly sample participants from each stratum to ensure representation across all of your desired demographic categories.

Example:

In a nationwide survey about healthcare, stratify the population by age groups (e.g., 18-24, 25-34, 35-44, etc.) Then, randomly select participants from each age category to obtain a diverse sample.

3. Quota Sampling

Set quotas for specific demographic groups within your target population to ensure proportional representation. This method allows you to control the composition of your sample based on predefined criteria.

Example:

When conducting a product feedback survey, set quotas for age and income groups (e.g., fifty low income 18-25 year-olds, fifty low income 26-35 year-olds, fifty high income 18-25 year-olds, etc.) to obtain representation from various demographics. Once your quota is met for a particular group, stop respondent recruitment for that category and focus on getting more participants for the following category.

Pro Tip: Aim to create a census representative sample that mirrors the distribution of the entire population, as outlined by the office of national census data.

Choosing Between Paid Data vs. Free Data

Many researchers choose to make use of paid survey panels to obtain their responses. Those who work without a budget, on the other hand, opt for free survey exchanges. Whichever boat you’re in, be sure to consider the following:

Paid Data

When you’re going about buying survey respondents, remember that your payment is likely to reflect in your data quality. If you underpay your respondents, you’ll probably end up with lower data quality. On the flip side, fair payment is likely to produce higher-quality data. 

Another thing to keep in mind is that your payment should reflect what your target audience’s time is worth. A financially-challenged individual might be satisfied with a minimum wage payment for 10 minutes of their time, but a high-earning CEO most definitely would not. Adjust your payment according to your desired target audience’s professional profiles.

Free Data

Survey exchange platforms and reputable online social groups are the way to go when you’re on the hunt for free, higher-quality survey data. Why? Because the other users on the platform are looking for reliable data, too, and are therefore more likely to provide you with the same. As a plus, most reputable survey exchanges already have quality measures in place to ensure the responses you collect are real and of high quality too.

Examples of free survey exchanges:

• Exchange platforms like SurveySwap (Yep, that’s us!)
• Facebook groups dedicated to mutual survey exchanges
• Relevant LinkedIn groups for survey exchanges

Closing Thoughts

Implementing the right strategies and precautions during survey research can lead to valuable insights from genuine respondents. You can choose to employ these strategies on your own, or reach out to experts for help. 

And remember: While they do exist, bad actors are the exception, not the rule. With the right approach, reliable, high-quality data is within reach. 

Curious about what the experts can do for you? Get in touch with us for personalized survey research solutions. 

FAQs

1. How can I ensure my survey data is reliable?

Use quality checks such as qualification and attention checks, optimize your survey design to align with research goals, and maintain consistent question formats. Additionally, ensure your sampling methods are representative and adhere to ethical practices.

2. What are qualification checks and why are they important?

Qualification checks ensure that only participants who meet specific criteria can complete your survey. They protect your data against bad actors who may manipulate survey results for financial gain. Examples of qualification checks include qualification funnels and sample-specific questions.

3. What are attention checks and how do they work?

Attention checks verify whether or not respondents are paying attention to your survey questions and ensure that you only get thought-through answers. Examples of attention checks include red herring questions, YEA-saying counters, sand traps, and manipulation checks to filter out inattentive participants.

4. Should I use paid data or free data for my survey research?

Both have their advantages. Paid data often results in higher-quality responses if respondents are fairly compensated. Free data from survey exchanges or reputable online groups can be just as reliable, especially if the platform has built-in quality measures and works on a mutual-exchange basis.