Design by Numbers: How to Practice Effective Data-Informed Design

“Data-driven” is the adjective on everyone’s lips—but can this truly apply to the creative space? Can we solely rely on numbers and evidence-based decision making to shape great product experience?

This talk will cover the two sides of data-informed design—both qualitative and quantitative approaches—with practical insight into key techniques including MVT and Design Thinking principles. Through a product and design lens, we’ll discuss how to achieve user-centred outcomes, deliver compelling experiences and collaborate with creative teams on product optimisation.

Have you ever played “two truths, one lie”? You say three things, two are true and one lie… Lucinda gives some examples, but a truth is she speaks Spanish. She learned a lot about words we use in English by learning Spanish – the way words combine makes sense when you pull them apart to learn another language.

Data is useful, but we can easily get into a situation where our data is telling us two truths and one lie.

Three things to help us think about using data:

  1. Data can help us understand how a user ‘uses’ the product.
  2. Data is based on what you’ve already got
  3. Metrics define what you value

Supposing is good, but finding out is better. – Mark Twain.

People say they like to read world news in a newspaper, but actually they read entertainment stories.

Remember the four types of data:

  • qualitative
  • quant
  • attitudinal – what users believe and say
  • behavioural – what users actually do

Getting good data:

  • clear metric – defined success point
  • sufficient users/traffic/data
  • acknowledge systematic bias – focus on the micro, not macro
  • data-informed – give thought to the short and long term tradeoffs

When to A/B or multivariate test:

  • Optimisation – making a specific thing better
  • Validation – understanding wider impact of design decisions
  • Stress test – sanity check, test assumptions

Use the PIE framework (rate the three out of ten and you get a PIE score):

  • potential (how much can you improve something)
  • importance (is it valuable to work on this)
  • ease (how hard to test).

When the Fairfax team redesigned the SMH people said they wanted the exact same experience on all devices. In reality, engagement was low on mobile. They did an A/B test removing summaries, to make things cleaner and easier to choose what to read. There were upticks on mobile and tablet; with no negative impact on desktop. In the long term there is some feedback that by hiding summaries, some users felt the tone of the publication changed.

Metrics are merely a reflection of the product strategy that you have in place. – Andrew Chen


  • Micro-optimisation – particularly to the detriment
  • Undefined hypothesis – unclear goal of research
  • Inapplicable metrics
  • Local maximums

Hypothesis statement

We believe that
   (doing this)
   for (these people)
   will achieve (this outcome).
We will know this is true when
   (we get customer feedback).

Be data-informed, not data-driven. Use data to support and test intuition.