Survey Response Biases in User Research

Surveys can be a great user research method because they’re often quick and relatively cheap. However, surveys are based on user perceptions and there’s a lot of talk around biases that surround surveys. Today, I want to focus on three types of response bias, acquiescence, social desirability and recency. But before I get to that, I want to point out that surveys can be qualitative or quantitative and both are subject to response bias. Now, when I say response bias, I’m referring to a group of biases that result in our tendency to inaccurately respond to survey questions, even fantastically written survey questions. With that point made, let’s talk acquiescence, bias. This is our tendency to check agree as we run through a list of survey questions. Acquiescence, bias is basically a tendency to agree or say, yes, let’s imagine we’re running a system usability scale questionnaire using a five point scale where one is strongly disagree and five is strongly agree. Generally speaking, respondents will have a tendency to check, agree or strongly agree as they go through a questionnaire which would result in positively skewed results.

Moving on, it’s no secret that we like to project our best selves, but this want to be perceived as our ideal image results in a response bias the social desirability bias. This bias is the tendency to over report socially desirable behaviors and characteristics, but under report socially undesirable behaviors and characteristics. Think of a survey question like approximately how many hours per day do you spend on your mobile device? Respondents are more likely to underestimate the amount of time they spend on their phones because excessive phone usage is often perceived as socially undesirable. Finally, let’s talk recency bias. Human memory is flawed and our short term memory has a limited capacity.

When answering survey questions, respondents will often exhibit the recency bias. In other words, respondents will have a tendency to give more weight to recent experiences rather than their overall experience. So let’s say we ask the question how likely are you to recommend this website to a friend? Although this question might be targeting the overall Web experience, respondents are likely to rely on their most recent experience to make a judgment. And this might positively or negatively impact your data. To conclude, I want to leave you with three comments. First off, surveys don’t measure objective performance.

They measure user perceptions. Second, response bias is one of the reasons why we need such a large sample size for surveys. Large sample sizes help cancel out random variations and ensure that our findings would be seen and our entire user population. And finally, knowing that these response biases exist doesn’t mean you can prevent them in yourself or others. However, being aware and educated about these biases can help you better evaluate your research findings and lead to greater design insights.

Leave a comment

Your email address will not be published. Required fields are marked *