You Can’t Test Everything, So What Should You Test

Time is a valuable commodity and in short supply for most product teams, it would be great if you could test every feature journey or piece of content many times, but that’s often not possible due to time, budget and resources. Inevitably, some journeys, task screens or functionality.

I’m not going to be tested as many times as you would like. If you’re in a situation where you have limited time to run usability testing, how do you decide which aspects of the design get more rounds of testing than others? When we have a long list of products, features, tasks or journeys to test, the first thing we should really be doing is consider the potential to make changes to our design after we carry out our usability testing.

There’s no point choosing something to be tested in our upcoming sprint. If we know the usability issues we uncover are going to sit in a backlog for six months or 12 months, it’s probably better we focus on another area of the design where we can have the greatest impact for our users and also for the business. Then we should consider factors like what’s the risk if we don’t test this particular aspect of the design? What’s the worst that can happen? Features that we’ve done before and have existing components for in our design system will inherently be less risky than completely new features where we’ve not performed any design or research. Other aspects we should consider is the impact of a specific feature product task or journey on the user and the business. Some things will be used or valued more highly by users, and some will be more profitable for the business. Let’s look at a simplified example.

Let’s imagine we have three different aspects of a design we need to test and we’re not sure if we’ll be able to test them all. Let’s also imagine it’s impossible to test them in one study. So we need to decide what’s the priority now, the way that you do this is up to you and of course, could be done in many different ways, but you could use something like a prioritization matrix. Here we have three columns for risk, business impact and user impact. Down the side. We have three things that need to be tested. The sign up process log in and authentication edge cases and changing subscription preferences. Now, let’s imagine we’re going to give each of these a score. So one is high, two is medium and three is low.

Starting with the sign up process. The risk here is very low. We have an existing sign-up process for our other products. We’ve done user research on this before. So in this case, I’m going to give this a three for risk business impact. This is going to be one if people can’t sign up for a project that’s lost sales for user impact, I’m going to give that, too. It’s a problem for users, but they’ll either move onto a competitor or us if they want the product. OK, let’s do login and authentication edge cases. Let’s imagine we have these inbuilt security features and these are our existing users who need to log in.

Now, we’ve not done research on this before. This is a new feature. So we’re going to say high for risk. Business impact is also going to be high because if people can’t log in, that’s going to push a lot of calls to the customer contact center and we could lose some customers as a result. User impact is also high because this is really a problem. If people are having difficulty getting into their account, that’s going to impact their experience using our service a lot. OK, finally, let’s consider changing subscription preferences. Risk is low.

We’ve done this before. There are plenty of design patterns available. So we’ll give this a three business impact that’s also low. The business doesn’t care if it’s a little bit difficult for people to pause or check their subscriptions. For the user, it’s slightly higher impact. It’s going to be annoying if they can’t figure out how to do this themselves. But it’s not a showstopper because they could email or call the contact center. So now we have these goals. We can add them up. And looking at the lowest total here, we see we have a natural priority for these. The thing we should really focus on testing for our next round of usability testing is to log in and authentication edge cases. Now, calculating priority can be done in any way that you choose. The key important thing here is that we should focus our efforts on ensuring we have the greatest impact for users and for the business in making prioritization decisions for usability testing.

Leave a comment

Your email address will not be published. Required fields are marked *