You Can’t A/B Test Taste

Inspired by Ken Kocienda

“A team at Google couldn’t decide between two blues, so they’re testing 41 shades between each blue to see which one performs better.” — Douglas Bowman

How would you feel if you worked on that team? Empowered or a slave to data? More importantly, is choosing a shade of blue what really matters? I guess at Google scale these types of decisions are meaningful, but for most teams, this seems like a model to avoid.

The attraction of testing is certainty, but it poses more danger than is often talked about. A/B testing is fast to say, but even with modern tooling, it’s not free. Designing the test, coding it, running it, reviewing the results, and potentially re-running it with new options — all this effort can be more distraction than elixir. Taken too far, an A/B testing mindset can infect your team and burrow you into optimizing a small, localized area of the product. A/B Testing comes at a price — you lose sight of the big picture.

“The opportunity cost of running all the trials meant there was less time available for everyone on the development team to dream up a design that people might like two, or three, or ten times more. A/B tests might be useful in finding a color that will get people to click a link more often, but it can’t product a product that feels like a pleasing and integrated whole.” — Ken Kocienda

Ken makes a deep point here about taste, about how the overall product feels. This is one of the intangibles in building products people love. The product we rave about feels thoughtfully put together. No seams — everything within reach, just where it should be. If you’re working on optimizing a single isolated color, button, or text element, you’ll likely miss zooming out and checking in on the whole of your creation. A/B testing can be a tantalizing dead end.

“When it came to choosing a color, we picked one.” — Ken Kocienda

There is another hidden risk in rotating too hard in the direction of data. In a data-driven culture, you will undervalue intuition, and therefore, lose your ability to discern taste. Use it or lost it as they say.

If data dominates your culture, product experience will fade to the background, subservient to an all-powerful numerical score. What can be measured, will be. The artists will leave, the taste muscle will atrophy, and you’ll just be left with the data. Did Michelangelo A/B test the David? Did Hendrix A/B test Little Wing?

You can’t A/B test taste.

Like articles on building product? Subscribe to receive by email.