If you’re interpreting quantitative user research data, you should be familiar with the concept of a confidence interval to explain why confidence intervals are important. Let’s think about an example. Let’s say I’m running a quantitative usability test and I want to know how long on average it takes my population to complete a checkout process. We can think of the big group of people, the entirety of our users as the population.
When we’re collecting numerical data, we can’t usually collect that data from every single one of our users. I can’t kidnap each one of my customers and force them to participate so my population value the average time on task for the entire group of users. That’s unknown. What I have to do instead is take a sample. My sample will be the participants in my study. So maybe I run my study and I find out that the sample value, the average time on task for my participants is four point three minutes. Here’s that same four point three minutes shown as a bar chart. I could just go ahead and report that sample value that four point three. But let’s think about this for a minute. Remember that we took the sample because we can’t test the whole population. We want to use the sample to estimate the value that we might see if we could test the whole population. So is it likely that we just happened to get the same exact value with our sample that we would see in our population? No, it isn’t likely. If we recruited the right users and we ran the study carefully, it’s probably very close to that population value, but it probably isn’t exactly the same.
So instead of reporting that single value, that four point three minutes, it’s better if I present a range of values that makes it more likely that I’ll actually cover that unknown true population value. Imagine that we have a goal to make sure that this task is so easy. It takes people on average less than five minutes to complete it. If my confidence interval is three point six to four point nine minutes, that means I can be very confident that the true population value that actual amount of time it takes all of my customers to complete the task is less than five minutes. So it looks like we’re achieving that goal. Using statistics formulas, we can calculate ranges like this one for each set of quantitative data we collect.
The exact size of each confidence interval depends on the individual characteristics of the data set. So how many data points you have and the variability of those data points in particular, if you weren’t the one running quantitative studies, the analysis the most simple.