Protecting Data Quality: The Importance of Qualification Checks in Survey Research

Your survey’s success greatly depends on the measures you take to ensure the integrity and reliability of your collected data. One such measure is the addition of quality checks to your survey. Quality checks are essential safeguards that shield your survey from bad actors and careless respondents, both of which pose a serious threat to your data quality. This article focuses specifically on qualification checks which, when used alongside attention checks, ensure high data quality in survey research.

Did you know? Studies suggest that, without qualification checks, up to 30% of survey data can be considered low quality or irrelevant. This means that if you have 100 respondents who take your survey, chances are you’ll have to discard 30 of those answers if you don’t implement the necessary checks.

Importance of Qualification Checks in Survey Research

The integrity of your survey’s data hinges on the quality of the responses you collect, which is why it’s so important to be sure your respondents are who they say they are and, more importantly, actually qualify to take part in your survey. Essentially, qualification checks are safeguards against data contamination through individuals who misrepresent themselves or manipulate surveys for personal gain. These checks prevent you from collecting inaccurate or biased information, which could undermine the validity of your findings.

Take the following scenario:

You are conducting a survey to explore the attitudes of medical students in the USA towards the Covid-19 pandemic and the effect it had on their academic performance. As an incentive for survey participation, you offer each respondent an entry into a prize draw for a free gym membership.

Motivated by your reward, the following candidates all participate in your survey:

• A final-year medical student from Phoenix, Arizona
• A third-year medical student from Dallas, Texas
• A 16-year old high school student from Germany
• A prospective medical student from Fort Worth, Texas
• A final-year medical student from Cleveland, Ohio

The 16-year old high school student from Germany is not a medical student and does not reside in the USA, so you can easily pick them out and disregard their answers when sorting and analyzing your data. If you were very cautious, you’d also realize that, not having studied medicine yet, the prospective medical student from Fort Worth would not have experienced the effects of the Covid-19 pandemic on their academic performance, and therefore should also be disregarded as a respondent. In a batch of 5 respondents, it’s quite easy to spot those who don’t qualify to participate in your survey. In a batch of 200 respondents, however, you would have quite a hard time sorting through all your respondents to try and determine who qualifies to have their answers included in your study, and who doesn’t. Even more so if any of your participants were insincere and misrepresented themselves as a final-year medical student from Phoenix…

Cue the qualification check.

Different Types of Qualification Checks

We’ll be doing a deep-dive into two types of qualification checks: qualification funnels and sample-specific questions. It’s important to include both types of qualification checks to your survey to ensure that you’re only screening in participants who truly qualify to answer your survey.

1. Qualification Funnels

Qualification funnels involve the posing of various multiple-choice questions to determine a respondent’s eligibility. They typically offer 10-12 answer options, with only one being true. Survey participants who answer all the qualification funnel questions correctly are eligible to answer your survey, and their data may be included in your dataset if they pass the other quality checks too.

EXAMPLE

You’re looking for Dutch marketing students to answer your survey. Pose the following questions:

Question 1: Which country do you reside in?

Answer Options:

1. United States of America
2. United Kingdom
3. Germany
4. The Netherlands
5. Belgium
6. France
7. Hungary
8. Italy
9. Romania
10. South Africa
11. Namibia
12. India

Question 2: Which of the following languages is your mother tongue?

Answer Options:

1. English
2. Italian
3. Spanish
4. Mandarin
5. Hindi
6. German
7. Dutch
8. Afrikaans
9. Portuguese
10. Swahili
11. Japanese
12. Arabic

Question 3: Which of the following is your area of study?

Answer Options:

1. Computer Science
2. Engineering
3. Education
4. Business Administration
5. Marketing
6. Economics
7. Linguistics
8. Sociology
9. Biology
10. Environmental Science
11. Nutrition and Dietetics
12. Psychology

ADDITIONAL PRECAUTIONS

• Ensure that respondents can’t guess what the correct answers are from your survey’s title. For instance, don’t use the title “Survey for Dutch Healthcare Workers”, as this could lead to you screening in bad actors, defeating your qualification funnel’s purpose.
• Randomize the position of your correct answer options to prevent respondents from guessing the correct answer based on previous answers. For instance, if Question 1’s correct answer is ‘4’, make Question 2’s correct answer ‘7’, Question 3’s correct answer ‘5’, and so on.
• If possible, also randomize what your answer options (e.g. ‘A’, ‘B’, and ‘C’) represent for each participant that takes your survey. Some survey builders allow you to randomize your options per question, so that your answers will be displayed in a different order for every respondent.

2. Sample-specific Questions

Sample-specific questions are questions that are designed to assess a participant’s knowledge or characteristics that are specific to your survey’s intended audience. The correct answer to the question is typically very obvious to a respondent who belongs to your target audience, but not so much so to someone who falls outside of this group. Respondents who get these answers wrong do not qualify to participate in your survey and should be excluded from your dataset.

EXAMPLE

You’re looking for frequent Facebook users to answer your survey. Pose the following questions:

Question 1: Which of the following actions cannot be performed on Facebook?

Answer Options:

1. Like
2. Share
3. Comment
4. Ping
5. React
6. Follow
7. Unfriend
8. Search
9. Save
10. Report
11. Post
12. Tag

Question 2: What is Facebook’s Messaging App called?

Answer Options:

1. Facebook Messenger
2. Threads
3. Facebook Chats
4. Chatbox
5. Instant Message
6. DM
7. Inbox
8. The Hub
9. WhatsApp
10. Slack
11. Ping
12. The Chatroom

Question 3: Where would you sell an item on Facebook?

Answer Options:

1. Facebook Shop
2. The Group
3. Marketplace
4. META
5. Buy & Sell
6. Store
7. Yaga
8. The Exchange
9. Gumtree
10. Amazon
11. Zuckersell
12. Facebook Purchase

ADDITIONAL PRECAUTIONS

• Randomize the position of your correct answer options to prevent respondents from guessing the correct answer based on previous answers. For instance, if Question 1’s correct answer is ‘4’, make Question 2’s correct answer ‘7’, Question 3’s correct answer ‘5’, and so on.
• If possible, also randomize what your answer options (e.g. ‘A’, ‘B’, and ‘C’) represent for each participant that takes your survey. Some survey builders allow you to randomize your options per question, so that your answers will be displayed in a different order for every respondent.
• Be careful of what knowledge you assume your target audience has. While frequent Facebook users may know what the ‘Share’ button does, not all of them may know what the maximum number of characters are when a post is being created. Again, the correct answer should be fairly obvious to your intended audience.

BONUS Qualification Check

If you’re creating your survey with Qualtrics’ survey builder, you should definitely add CAPTCHA verification to keep those robots at bay. When adding a new question to your survey, simply choose “CAPTCHA verification” as your question type. This will require all respondents to be verified in order to proceed with your survey, and keep any survey bots at bay.

Closing Thoughts

In conclusion, qualification checks play a crucial role in safeguarding the quality and validity of your survey data. By employing these screening measures, you can enhance your data’s reliability, and ultimately derive more meaningful insights from your research efforts.

P.S. We reject the idea of throwing data away post-analysis, and believe that data quality should be guaranteed before responses are even collected. Allow us to do that for you: Get high quality survey data now.

FAQs

1. What are qualification checks in survey research?

Qualification checks are measures implemented in survey research to ensure the integrity and reliability of collected data. These checks verify whether respondents meet your specific criteria to participate in a survey, preventing data contamination from individuals who misrepresent themselves or manipulate surveys.

2. What types of qualification checks are commonly used?

Two primary types of qualification checks are often employed: qualification funnels and sample-specific questions. Qualification funnels involve multiple-choice questions with only one correct answer, assessing eligibility based on specific criteria. Sample-specific questions target respondents' knowledge or characteristics relevant to the survey's intended audience.

3. How do qualification funnels work?

Qualification funnels typically consist of multiple-choice questions designed to verify respondents' eligibility to take part in a survey. These questions have a single correct answer among several options. Respondents who answer all qualification funnel questions correctly are deemed eligible to participate further in the survey.

5. What are sample-specific questions?

Sample-specific questions assess respondents' knowledge or characteristics specific to the survey's target audience. The correct answers to these questions are often obvious to the intended audience, but may be challenging for others to answer correctly. Respondents who fail to answer these questions accurately are excluded from the dataset.