Understanding Satisficers

Humans are fickle creatures. Regardless of our capabilities, oftentimes we just want easy-peasy, quick, and instantly gratifying tasks — including participation in survey research. Despite our best intentions and efforts, when asked a series of questions in quick succession, many of us engage in behaviors that reduce the mental effort needed to complete the survey. 

Unfortunately for survey researchers, this means that participants are sometimes dishonest in their responses, inattentive while answering, or in a hurry to get through the questions. These survey respondents ultimately provide poor-quality answers, which leads to poor-quality data, which leads to poor-quality insights. This behavior is called satisficing. 

Read on to explore how satisficing impacts the results of your survey efforts — and how Civis Analytics is innovating to mitigate the satisficer effect and more accurately determine the opinions of audiences.

Researchers have grappled with how to address satisficing^* — i.e., the attempts that respondents make to get through a survey while reducing the mental burden of the survey process — since the inception of survey research and polling as means of collecting first-person data. 

When a respondent is satisficing, they employ certain shortcuts during the survey-taking process. These shortcuts can include: straightlining^* through matrix (or grid) questions, speeding (or going extremely slowly) though the questions, answering randomly, or giving into social desirability^* or acquiescence bias^*. Either intentionally or unintentionally, they are putting in the minimum effort required to get through the survey and get that bread.

Satisficing behaviors are further exacerbated by increasingly frequent participation in survey research, which is especially common among those who voluntarily opt in to participating in an online, non-probability based panel^*.

These panelists^* are in the habit of regularly taking surveys, and therefore even more likely to be fatigued by the survey process. Online panelists engage in satisficing behaviors at a higher rate than what researchers have historically observed among respondents in other modes of survey research, such as face-to-face interviewing and phone surveys. 

Although traditional polling methods are less likely to lead to satisficing, response rates in phone surveys have fallen dramatically over the past decade, leading to increasingly expensive and slow-to-field surveys. To adapt, organizations and research firms including Civis have turned to online panels or panel marketplaces^* as their sample sources, requiring innovative and effective solutions to remove satisficers from the sample so we are able to provide accurate, high-quality data and insights to our partners. 

Civis’s state-of-the-art quota^* and weighting^* algorithms ensure that respondents match the population of interest along key demographic characteristics, giving you confidence in the inferences you make about your intended audience. Importantly, our extensively tested in-survey data quality checks ensure that only respondents who meet our quality standards complete the survey, guaranteeing the resulting data and insights are accurate, precise, and actionable.

Civis’s survey scientists, a team with backgrounds in survey methodology and experimental research, consistently test and implement novel ways of identifying these problematic respondents. At the time of this writing, we’ve tested a variety of measures of identifying and removing satisficers from online survey samples, including: 

In addition to these in-survey factors, we regularly compare panel data quality so we can identify panels that send us a disproportionate share of satisficers — an essential step in improving data quality. This iterative process ensures that we are able to take actionable steps to block problematic panels from future samples, while allowing for the high-quality panels to send us more survey participants. We incorporate these findings into our survey templates so our data scientists always have ready access to the most up-to-date, rigorously tested, and methodologically sound ways to identify low-quality respondent data.

Figure 1 (above) displays a comparison of estimates for one of the satisficer-detection questions used in our data quality module and looks at how the sample changes when the people that we flagged as satisficers are removed from the dataset. The y-axis for this graph is the reported frequency of these behaviors in Question 1 (where we gauged participation in unlikely events), and the x-axis is our acceptance threshold for the satisficers in the sample. As the x-axis increases, we are more and more inclusive of satisficers in the survey. A score of 1 on the x-axis means that the resulting sample has excluded anyone who satisficed one or more times, and a score of 10 means that only those who satisficed 10 or more times were removed from the sample. 

The satisficing score, shown on the x-axis as the “Acceptance Threshold,” was determined by the number of times respondents satisficed during the survey. We used the assumption that high-incidence activities should have nearly a 100 percent incidence rate (IR), meaning that nearly everyone should have done these activities once over the course of the past 30 days, and conversely, the low-incidence activities should have close to a zero percent incidence rate, because most people would not have done these activities in the past month. (Please note that due to disabilities and other factors, it’s possible that some respondents might be outliers in how they responded to these questions; therefore, we use a threshold of two or more incorrect responses before we terminate respondents from the survey for satisficing.)

The plot on the left looks at the reported frequencies of high-incidence activities, like drinking water, milk, or juice, or watching an hour of video. The plot on the right looks at the impact on the reported frequencies of some of the low-incidence activities (like riding a hot air balloon or a submarine) when these satisficers are removed from the survey sample. 

As you see, as we are more inclusive with the satisficers in the survey, the reported frequencies of high-incidence behaviors decrease, and the low-incidence behaviors increase slightly. This demonstrates that the stricter we are with cutting off satisficers, the closer we get to measuring the true frequency of these behaviors. 

The question wording is as follows:

We’re continuously testing and refining our methods, and one way to ensure we’re getting accurate results is to compare survey estimates to known, external benchmarks. To determine our level of precision, we asked a series of questions about a variety of topics that were asked on benchmark surveys or other creditable data sources. These questions included things ranging from cat ownership (whether or not the respondent owned a cat), driver’s license possession (whether or not the respondent had a valid driver’s license at the time of data collection), and airplane travel (whether or not the respondent traveled on a flight between October 2020 and November 2021).

After filtering out satisficers according to our standard process, we compared the weighted estimates from Civis survey questions to a known benchmark^* for the same question (sources: TSA, Bureau of Transportation Statistics, ASPCA). These benchmark estimates were pulled from well-respected, high-quality tracking surveys, the values of which are often considered to be a “True Value” for these questions. For most of the comparison questions asked in the survey, our weighted estimates after the removal of poor-quality respondents were extremely close to the benchmarks!

We noted that our satisficer checks contributed to the accuracy of our survey estimates in many of these questions, with the confidence intervals of our estimates often containing the True Value. Even for the estimates that didn’t fall within the confidence interval of the True Value, filtering out satisficers from the sample always got us closer to that estimate.

Satisficers will learn and adapt to the questions being asked in a satisficer-detection module, particularly those who participate in online panels and engage frequently with surveys. Figure 3 demonstrates this concept for one of the questions in our module — a simple attention check question that instructs the respondent to choose the response option “C” in a randomized, multiple choice list of four letters. 

However, despite the simplicity of this question, we see that at the peak, nearly 14 percent of respondents in the sample answered incorrectly. Although there are some peaks and troughs in average satisficing over the first half of the waves, after March 2021 we see a definitive decrease in average satisficing on this question. This demonstrates that over time, savvy satisficers will learn to recognize these questions and adapt their behaviors. We recommend regularly checking on average rates of satisficing, and changing your satisficer-identification methods and questions when necessary in order to continue properly flagging poor respondents.

Note: This data was pulled from Civis’s “always-on” tracking survey, the Omnibus, which runs weekly waves of surveys (including the satisficer-detection module) with a sample size of about 2,000 people per wave. The tracking survey began in May 2020, so this data includes respondents from every wave from that time through March 2022.

For Figure 3 (above), every survey from the four waves of the Omnibus fielded were consolidated into monthly waves.

During this experimental process of finding the most effective satisficer-detection module for survey research, we’ve learned many important lessons about the world of survey respondents and the measures necessary to address increasingly poor respondent data quality. 

First and foremost, we’ve learned that there will always be people who are engaging in these types of satisficing behaviors regardless of the data collection mode used for the research, although it is more common in some modes of data collection than others. Since satisficing behavior is endemic to survey research, researchers should ensure the survey is designed succinctly, simply, and engagingly to foster respondent attentiveness and accuracy throughout the survey. 

Expecting perfection and full attention from respondents is proving to be more and more unattainable. But employing effective warning messages and motivational messages, in addition to well-tested satisficer-detection questions, can go a long way towards meaningfully improving data quality. By remaining vigilant about data quality and designing simple, engaging surveys, we can deliver precise, accurate, and representative insights to our clients and partners. 

Additionally, this research highlights the importance of constant experimentation around methods of identifying low quality respondents, and the importance of regularly updating these findings. Satisficers are savvy! Survey respondents adapt and find new ways to “game” the survey system, and get around outdated versions of data quality checks. Panels are continuously recruiting new participants into their panels, so the quality of their data is also subject to changes based on these uncontrollable recruitment methods and respondent composition. 

Regularly investigating the quality of data from panel providers is essential to ensuring that you are making informed decisions on the composition of your sample, as is only using panels that send high-quality respondents to your surveys. This iterative process ensures that we never accept a one-and-done solution to a constantly changing problem. We encourage organizations to conduct similar experiments on their samples to determine what works best in their particular field, and to continue to do so regularly, since respondents will never stop shifting and adapting to outdated checks. 

Lastly, we found that knowledge sharing is crucial as we determine even better methods for identifying poor-quality respondents. We hope other organizations apply our findings and insights to address similar concerns about data quality in their data collection modes. 

Civis researchers have presented many of the findings cited here at survey conferences such as AAPOR and MAPOR; disseminated this information to partners and clients; and externally documented much of this research to audiences like you through posts like this. Transparency and open lines of communication across organizations are key factors in ensuring that research teams are up to date on the most recent findings in this field, giving audiences deep trust in the accuracy and quality of our survey research findings.

Collaboration and transparency, regularly adapting our research methods, prioritizing respondent experience, and constantly learning about the novel methods implemented at other organizations help us combat quality concerns and produce accurate research and insights. We encourage you to reach out to us with any questions, concerns, or suggestions about identifying low-quality respondents.