Protecting Data Quality: The Importance of Attention Checks in Survey Research

Your survey can only be successful if you take the necessary measures to ensure the integrity and reliability of your collected data. One such measure is the addition of quality checks to your survey. Quality checks are essential safeguards that shield your survey from bad actors and careless respondents, both of which pose a serious threat to your data quality. This article focuses specifically on attention checks which, when used alongside qualification checks, ensure high data quality in survey research.

Did you know? Studies suggest that 10-20% of respondents fail attention checks under normal survey conditions, whereas 20-30% of respondents fail these checks in highly-incentivized survey environments. This suggests that the same percentage of responses are likely to be discarded during data-analysis due to the absence of attention checks.

Importance of Attention Checks in Survey Research

Attention checks, also known as “validity” or “trap” questions, are designed to assess whether respondents are actually paying attention to your survey questions rather than responding haphazardly. They help you to filter out respondents who speed through the survey or provide random answers, whether this be for personal gain or simply because the participant is not attentive. Answers provided by such careless respondents could negatively impact your data integrity.

Take the following scenario:

You are conducting a survey to investigate the potential interest in a new product you’d like to add to your business catalog before your next busy season. You want to survey 500 respondents. Due to the large number of responses needed in a short amount of time, you decide to incentivize participation: Each respondent will get a 10% discount code to spend at your online store.

Because of the desirability of your incentive, you quickly find 500 willing respondents through an online survey panel. Unfortunately, you quickly realize that not all of these participants have good intentions – many of them are merely looking to get access to your discount code as quickly as possible, without attentively reading through and answering your questions. Of course, this poses a massive threat to your data integrity. Ideally, you would only collect responses from individuals who spend the time and energy to carefully read through your survey and answer it truthfully, to the best of their abilities. In other words, you only want the responses of those who are actually paying attention to your survey, and disregard those who merely speed through it to get to the reward.

Attention checks to the rescue…

Different Types of Attention Checks

We’ll be focusing on four types of attention checks: Red Herring Questions, YEA-Saying Counters, Sand traps, and Manipulation Checks. It’s important to remember that these attention checks are most successful in filtering out careless respondents when used together, scattered randomly throughout your survey.

1. Red Herring (“Trap”) Questions

Red Herring questions (also referred to as “trap” questions) are designed to filter out respondents who rush through a survey without properly reading a question, by asking respondents to select the fairly obvious answer.

EXAMPLE

Pose the following question and provide the answer options that follow:

Question 1: Which of the following cannot be categorized as a fruit?

Answer Options:

1. Banana
2. Apple
3. Monkey
4. Orange

Filter out any responses that do not provide Option 3 as the correct answer, as these respondents could not provide the correct answer and therefore failed the attention check.

2. YEA-saying Counters

YEA-saying counters are used to identify respondents who mindlessly agree with all statements made to rush through surveys. These questions require a specific response to determine if participants are actually paying attention to what you’re asking, or if they’re simply selecting options without paying any mind to what is being asked.

EXAMPLE

Pose the following question and provide the answer options that follow:

Question 2: Can you name all people who have ever lived on Earth, from the beginning of time?

Answer Options:

1. Yes
2. No
3. Possibly

Any respondent who answers anything other than “No” (Option 2) has failed the attention check, since it is impossible for anyone to name all people who have ever lived on Earth from the beginning of time.

Another way to perform a YEA-saying Counter, is by explicitly mentioning that the question you’re asking is an attention check, and requesting participants to provide a specific answer.

EXAMPLE

Pose the following question and provide the answer options that follow:

Question 3: The following question is an attention check. Please select “Disagree” as your answer.

Answer Options:

1. Strongly Agree
2. Agree
3. Neither agree nor disagree
4. Disagree
5. Strongly disagree

Again, if the respondent doesn’t select your desired answer as requested, they have failed the attention check and should be disqualified from your survey.

3. Sand Traps

Sand traps involve incorporating open-ended questions to assess genuine engagement. These questions require respondents to provide thoughtful answers rather than selecting predefined options. Responses that lack coherence or depth, such as a simple "No," indicate the respondent wasn't paying attention.

EXAMPLE

Pose the following question and provide an open field for respondents to provide their answer:

Question 4: In your opinion, how can universities best support students who are having academic difficulties?

If a respondent fails to provide a genuine answer (e.g. they simply answer “yes”), they have failed this attention check and should be excluded from your study.

Take note: Unless you’re making use of AI prompts to analyze your data for genuine answers, this is only recommended for surveys with smaller samples (100-200 respondents) to avoid you having to check a vast number of open-ended responses.

4. Manipulation Checks

Manipulation checks ensure that participants are attentively completing your survey by incorporating questions related to the survey material. These questions are designed to filter out any respondent who isn’t truly engaged in your survey.

EXAMPLE

Provide respondents with a short text related to your survey topic. Thereafter, pose the following question and provide the answer options that follow:

Text: 

Should children be allowed to own pets? Greg’s pet tortoise, Speedy, says “no”. According to Speedy, Greg continuously mistakes Speedy for a sea turtle. Speedy has had enough of the daily “swims” in the bathtub and wishes to be placed in the care of an adult owner.

Question 5: What was Greg’s pet tortoise called?

Answer Options:

1. Turty
2. Snail
3. Shelly
4. Speedy

Respondents who attentively read through your text would know that the correct answer to the question is Option 4: Speedy. Any participants who answered incorrectly did not pass this attention check and do not qualify to be part of your study.

PRO TIPS

• Make use of various types of attention checks to ensure that you’re only screening in participants who are truly engaged in your survey. A respondent who passes all of your attention checks is likely also providing thought-through answers.
• Scatter your attention checks across your survey to test respondent engagement at various points in the questionnaire.
• When making use of attention checks in the form of multiple choice questions, randomize the position of your correct answer options to prevent respondents from guessing the correct answer based on previous answers. For instance, if Question 1’s correct answer is ‘4’, make Question 2’s correct answer ‘2’, Question 3’s correct answer ‘1’, and so on.
• If possible, also randomize what your answer options (e.g. ‘A’, ‘B’, and ‘C’) represent for each participant that takes your survey. Some survey builders allow you to randomize your options per question, so that your answers will be displayed in a different order for every respondent.

Final Thoughts

In conclusion, attention checks are crucial additions to your survey to ensure the quality and validity of your data. By continuously testing participant engagement and filtering out any insincere respondents, you are enhancing your data’s reliability, and will ultimately derive more meaningful insights from your research efforts.

P.S. Data should never be discarded post-analysis. That’s just wasteful. Let us help you collect high quality survey data in a time- and cost-effective manner.

FAQs

1. What are attention checks in survey research?

Attention checks ensure that respondents are actively paying attention to your survey questions and providing thought-through answers. They help maintain the integrity and reliability of the data collected by filtering out participants who are speeding through your survey or giving random answers.

2. Why are attention checks important in surveys?

Attention checks are crucial for maintaining data quality. They identify and exclude careless or disingenuous respondents who might negatively impact the validity of your survey results. This ensures that the insights derived from your survey are accurate and reliable.

3. What are the different types of attention checks?

There are several types of attention checks used in survey research, including:

• Red Herring (“Trap”) Questions
• YEA-Saying Counters
• Sand Traps
• Manipulation Checks

4. How can I use attention checks in my survey effectively?

To use attention checks effectively:

• Incorporate various types of attention checks throughout your survey.
• Scatter them randomly to test engagement at different points.
• Randomize the position of correct answer options to prevent respondents from guessing based on patterns.
• Randomize what answer options are presented to each participant.

5. What should I do with respondents who fail attention checks?

Respondents who fail attention checks should be excluded from the final dataset to maintain the reliability and validity of your survey results, ensuring high data quality in the process.