AB testing is a big part of how modern products are developed and optimized, and there’s a good reason for it when it’s done well, you compare one or more possible changes in the design that you’re considering against the existing version. And you see which one performs better, gives you really good large sample size data about how people behave. But it’s not the whole story because you’re blind as to why something worked better, how people felt about it. If AB testing is the only technique that you use to test a design, you can end up with an experience that’s worse than the sum of its parts. Have you been into a clothing store lately, like an actual physical clothing store? What happens first? Someone pops up and goes, Hey, how’s it going?
Can I help you find anything I need?
You try and dodge them because you’re already been to two other stores today and you don’t want to have to have the same conversation again where you go. Oh, I’m fine. Just browsing. Then you walk forward a few feet and it looks like the coast is clear. So you start OK, some of the clothes on one of the racks and then suddenly someone else is there. Hey, can I help you find anything? So you do a little fake you out and you try and get past them to an area where they’ll just leave you alone and let you shop for clothes. But another person is waiting to tackle you from behind a display. So you turn and you dodge them and then they end up chasing you around the store and so you run away never to return. This is a terrible experience, but it’s pretty universal nowadays. How did this happen? Well, it wasn’t someone purposefully designing a terrible, stressful and end experience. This came from a bunch of little experiments. So having a person greet you when you walked in resulted in more sales. So that became the new standard. Next, they tried doing the same thing, but divvying up the store into several zones with an employee and each one that bumped up the numbers more than having just one person do it. So that became the new standard. And then there were experiments about what they should say and how much time they should wait before engaging you and so on. So we ended up with retail stores filled with employees on headsets, chasing customers around in order to lift a sales metric slightly. But overall, the whole experience became kind of stressful and unpleasant and kind of dehumanizing for everyone involved. Each little change lifted a metric, but it made the experience a little less pleasant. Each one on its own wasn’t a deal breaker. But combining them led to a cumulatively terrible experience. It became worse than the sum of its parts. And now retail stores are losing a lot of ground to e-commerce. And it’s not hard to see why e-commerce is more convenient. Sure, but we also don’t have to have these unpleasant interactions where we feel pressured to make a snap decision, except we’re kind of turning e-commerce into the same nightmare. Pop ups on top of pop ups, chat bots, banners, flashing animations. They’re all grabbing our attention. This all came out of the same practice, optimizing via AB testing. We’re testing to see which version of something performs better, but with only a limited understanding of why it does. The problem is that this is happening without actually spending time with users to find out why they do the things they do, how they feel about it. So all we know is that if we put a modal pop up that pressures people to sign up, then we might see that number go up a little bit. But we forget about how we’re destroying our brand goodwill on the process. So don’t ab test yourself off a cliff. When we look only at a little sliver of something, we don’t understand how all the pieces we put together make the whole experience worse. You can stepwise optimize things to be worse than when you started because each little experiment on its own might work in isolation. But together they’re a bad combination. It’s important that you combine your AB testing with some qualitative research where you can find out why your design changed those metrics.