Measurable Design

Faster is better. That much is obvious. How much better? That we can find out. For people preoccupied with performance, numbers can tell a clear story. But how can you determine success for design changes? “It just looks right” isn’t going to be enough.

You can’t embetter what you cannot measure. So let’s look at how to make design measurable. Which metrics matter, and which can be ignored. A better looking hamburger won’t improve your navigation, but there might still valid reasons to change it.

Two problems:

  1. Design decions are hard – and there are times people basically just guess!
  2. Design starts out great and slowly unravels – you can’t manage what you don’t measure..?

People love numbers. People like magic numbers. Some things like sales and performance have clear numbers. But what would that be for design?

There is a DOW average – it’s a measure almost nobody really understands, but it gives a number. We are seeking the DOW of web design (great in-joke!).

Start on the inside… “I don’t want to talk to users, they’re gross!”

How much is your design system worth? How many engineering hours is it saving? Look up the design debates people were having in PRs, or planning sessions, that aren’t required any more.

Physical measurements:

  • contrast is easy to measure but we keep getting it wrong!
  • Signal:noise ratio can be measured for typography and charting (signal is relevant information and noise is stuff other people made you shove in there).
  • Line length – “studies have shown”…? actually there really are studies and they’ve been reproduced! But you can work out what works for your product then measure if you’re sticking to it.
  • Golden ratio

Measurement systems

  • Pulse metric – page views, uptime, latency, seven-day active users, earnings. But being alive or dead is not a good measure of the quality of your life. It’s just not very expressive.
  • Heart metric – happiness, engagement, adoption, retention, task success (the methodology suggests you pick two to focus on)

Goals → Signals → Metrics

Goals should not contain metrics! Keep the goal broader, to avoid making it self-fulfilling.

People should be happier → measure the number of one-start reviews → reduce the number of one-star reviews

Engagement isn’t a great goal but it’s easy to measure so it’s popular.

Simon went with happiness and task success. These are really good metrics, but they are hard to measure.

Happiness – could you measure twitter outrage? Over time you’ll see people go through change aversion responses after every relaunch. It seems the best way to measure happiness is still….sadly….surveys.

  • ask one single question. If you want to do more you will get far fewer people filling them out.
  • how you ask questions matters – ask neutral, reproducible questions
  • how you collect the answers matters – free text vs multiselects etc

Measuring task success

  • Analytics work for some tasks
  • User tests work best for measuring task success

Take feedback, but don’t let them tell you what to do! Get feedback but don’t literally do what they say would solve the problems they’re raising.

You can’t measure your way to new ideas. Data only tells you about the things you already have.

Data is not insights.

Engineering term: AM/FM. Actual Machines/F***ing Magic. The things we have now vs the things that are coming next year that are going to solve everything… like magic. Sadly the design number is still FM.

So don’t let data make your decisions. Measuring design is more than just numbers.

Not everything that counts can be counted. Not everything that can be counted counts.