The Continuous Discovery paradox: rigour vs. agility

Many of us have been there. We recommend that we need research and are told it's not going to tell us anything new. It won't be of any value to the business. And mostly that it just what Stephanie was talking about, It takes too much time. We trust, though, that good research saves time by avoiding cost, by avoiding bad choices.

But often when research is needed, it's already too late. So in continuous discovery habits, Teresa Torres has tried to change what time means to research. I'm Eri. I'm here as a collaborator, as someone who believes research is fundamental to design and to product, as someone who believes that research drives a lot of purpose in teams and, of course, differentiation for business, And because I think work is much more effective and much more fun when it's cross disciplinary.

So when this approach kicked up a bit of a stir, and it came onto the scene with some controversy, I got really curious about it. Today, I'll provide an overview of continuous discovery, address some of the criticisms, share some lessons from a pilot, and considerations for establishing an always on program.

So what is it? Continuous discovery is an approach that embeds learning into product development, connecting business goals to customer insight and product choices. It was developed by Teresa Torres, who defines it as and this is her quote and her visual of the whole framework at a minimum, weekly touch points with customers by the team building the product, by the team building the product, where they're conducting small research activities in pursuit of a desired product outcome.

The framework involves many techniques. It's like a collage. It's like a kind of collection of lots of things that are familiar to us that Torres calls habits. It takes an iterative and incremental approach. That is an agile one to product discovery. It involves a product trio. That's a product manager, a product designer, and an engineer on the team. And it's about getting these different lenses to the problem with someone on design and usability, someone on technical feasibility, and overall business viability too, all working together, looking at research together and thinking through it. This team defines product outcomes that generate business value and can be measured. They draft an experience map to baseline what is known and where to start.

This is a bit of a living artifact. They conduct customer interviews. If anyone's kind of reaching up at the thought of that, we'll talk about it. They conduct customer interviews regularly with the interviews following a story based format. So it's really about getting that participant to recall experiences rather than project into the future on something speculative.

An overview of the session is captured as a participant snapshot. So this is an artifact that's designed to overcome that amnesia where details can fade over time. Customer pain points, unmet needs and desires these are what we're looking for. They're mapped onto an opportunity solution tree. The team then prioritizes the most important things to act on.

Ideate solutions. It's sounding like a pretty typical design process at this stage. Breaking them down, though, then into underlying assumptions. And then early in the cycle, they test those assumptions. They test prototypes. So a variety of means to test that design concept or early ideas.

And then later in the product cycle, they run experiments to measure impact and evaluate their solutions. So there's some stuff that's a bit familiar there, and some stuff that might be slightly new. The opportunity solution tree is the central artifact for thinking about these trade off decisions. And it's a really key artifact to the whole process. It starts with the business outcomes.

That means your OKR or your KPI. What's that thing that's generating business value? Opportunities that are uncovered into the researcher mapped to it. And then, of course, I think you get the point that solutions or potential solutions, ideas, are mapped to those pain points.

And then they're really broken down, teased out, using techniques like story mapping and going from there. So you can see that when this is all put together, it helps the whole team and anyone else they're working with see all of their thinking. So we'll consider a little example here, a simple one from e commerce.

So pretend you're about to go shopping. The OKR is to increase people adding an item to their cart from the catalog. So customer interviews uncovered a lot of problems with the shopping experience. So people negotiating an overwhelming catalog, unclear how something would look, confused by size charts.

Customers find photos like really uninformative and unhelpful. It might be hard to imagine how something that looks really great on a model will look on your body type. You know, the research might have also uncovered that a lot of people need some inspiration for what to wear. That they're tired when they're looking at that vast catalog of seeing repetitive products. And then others are really frustrated, wasting their time browsing.

Not repetitive products, but irrelevant products. So some ideas to solve for this. Maybe showing some curated edits. Solving for repetition by consolidating color variations of a product into one listing. Enhancing personalization, you know, maybe looking at someone's purchase history, and enabling people to filter out options.

And some assumptions to test around those ideas could be that shoppers will spend extra time setting filters. That they'll know when they need to, how to undo those options in case they change their mind. So there's some kind of UXE type assumptions. This last one is a bit different, because the team is looking at this through a business lens also.

So they also want to test that showing fewer irrelevant products won't impact revenue by, for instance, hiding products, hiding high margin items, or items from sponsored brands. So hopefully that gives you an idea of how it all might come together. Continuous discovery is more than just a process.

It's a mindset shift. And these mindsets are from Therese herself. So she wants to move teams from building the plan to constantly delivering to asking, is this actually right? Is this assumption sound? So the six mindsets that underpin it are collaborative and continuous, you know, speaking to some of its agile principles and that cross disciplinary perspective that it's customer centric and visual, speaking to its design principles and techniques that help teams show their design thinking in prototypes, but also their product thinking in the opportunity solution tree, and speaking to product principles, that it's experimental and outcome oriented.

Now this is about value. And it's worth thinking about the context that many product teams find themselves in. They might have preset roadmaps. They're working on ideas that are not their own. They're inherited from their stakeholders. And success is rewarded by what's shipped, not necessarily tracked back to value. In continuous discovery, the focus is on outcomes and, you know, this change that we want to see for the business and this positive change that we want to see in customers' lives.

So it's less about did we build the thing and more did the thing we build actually achieve the desired result. I would add three additional principles as key to understanding this framework. These being ruthless scope within the product team's control. This is what they are accountable for and what they can deliver.

Action through ambiguity. So looking for early signals with initial interviews and tests before large scale experiments. And my team really might get sick of me saying this all the time, but exploring optionality. From which opportunity to pursue, to generating lots of ideas to explore, showing all the possible paths, and having those trade off discussions.

I really like Goats, so I put one in my presentation. This method has gained traction in product management, but it has faced a lot of criticism from the UX community, the UXR community. When I first started exploring it, the criticism was palpable. And with this backlash, this method rightly or wrongly became a scapegoat for the many job layoffs across tech.

I think there was a lot of bad timing as well with when the book was released, when it gained traction, and what was happening. Overall, the criticisms of the approach focus on the pitfalls of democratising research. However, they don't acknowledge continuous discovery as a wider product operating system. So let's unpack what some of them are.

The first one is democratizing research leads to flawed research. That non researchers are prone to bias, you know, leading questions, that sort of thing. That they're only capable of superficial synthesis. They're not learning from Steph. That the approach devalues researchers and their skill set.

And that it's asking actually just too much of that product team, that cross disciplinary product team. It's asking too much of their skills and their roles. Another critique is the perception of flawed recruitment practices and research sampling that there's insufficient guidance on how to recruit, that too small a sample size provides false confidence, and that it encourages convenience sampling.

Another criticism is that it's tactical and not strategic, that it is overly focused on features. Then a slightly different one is that a focus on customers and observing them doesn't result in breakthrough ideas. That might come from some more, you know, product people. Some product people. And then if strategy and road maps are set, there's no need for discovery. Ideas from new research will only bloat an already large backlog, and speculative, proactive research may not be the best allocation of resources. I saw someone shaking their head, I I know this one.

I I I agree. It's a funny one. And lastly, you know, particularly from the teams that have practiced it, that the regular practices, particularly the cadence of customer interviews and that expectation set in the in the book of weekly interviews is just unsustainable over time and can fatigue teams.

So there's a lot of logistics involved, that discovery activity contributes to additional pressure alongside delivery, that over time, the central artifact, that opportunity solution tree, gets bloated over time, cumbersome to use, and that the context of research, that original research, is lost over time. Okay. So let's address these criticisms and see if we can find some balance. On the matter of democratizing research.

I feel like this is the second wave of this. I went through the first wave of democratizing research with design thinking and, yeah, other other stories. On the matter of democratising research, the target audience of the approach is companies where there are no research roles. So should they be doing research?

And I think about this too, about what information do we want them to have and to use? There's also a plus side to this too. And it's teams who find continuous discovery valuable but cumbersome are actually learning what the labor of research is. And this criticism also may be overstated.

Not everyone is undertaking this without research partners. So there are teams that are including UXR, data scientists, and other partners in the business in their product team. On the criticism about recruitment, it's a continuous process.

So the research sample and evidence base is cumulative. Opportunities are also triangulated with other evidence, like surveys, customer service feedback, analytics. On it being tactical, not strategic. Also, not everything can be strategic. Let's just start there.

Sometimes you just gotta do some tactical work. There's no hierarchy. There's no good or bad. But I would say strategic questions are part of the approach. Opportunities are evaluated against customer, company, and market factors. And tactical improvements, you know, they can be really great. It could be a usability win or an important optimisation for the experience. It also doesn't exclude separate strategic studies from being run.

But maybe you don't need discovery at all. There are tools and techniques within the framework that are still worth looking at, and that guide good product practices and scrutiny on testing ideas. And they're really worth looking into. But if that's true and that's where you're at, then ad hoc usability studies, rapid iterative testing and evaluation, or an always on program might actually be more appropriate for your team.

Is it sustainable? Is it fatiguing? Lengths of customer interview sessions are meant to vary and be short. This is one that really challenged me, and the book talks about having interviews for as short as, like, five minutes. But the thing to understand is not every interview in this process is designed to be a sixty minute deep dive.

The framework necessitates automation of logistics, using tools for scheduling and recruiting users from inside your product. On the criticism of the opportunity solution tree getting bloated over time, it's meant to be a living artifact that is pruned over time, that's edited, that's worked through as the team learns.

And teams are finding the regular cadence that works for them. So it seems in the real world that weekly is rarely doable, and people are finding what works. So there is a bit of a paradox here. If we compare continuous discovery with established methods, we actually find that there are a lot of similarities.

Continuous discovery shares the same characteristics and the same potential shortcomings as familiar discovery and testing methods. Like design thinking, it's generative and democratises research. It shares quite a few characteristics with the Google Ventures Design Sprint approach. Both are user centred, both are lean, both tease out ideas using story mapping to make them more robust and uncover assumptions, Both democratise research to a cross functional team, and use pretty low sample sizes for validation and customer testing.

But continuous discovery is much more than an ideation exercise or testing technique. It's a whole product discovery framework. But what is most paradoxical is that the very approach that is advocating the hardest for customer centered discovery at this point in time is the approach that has faced most scrutiny from the UX and UXR community.

I thought the best way to scrutinize the approach and sorry, I'm being really fussy with my glasses because I'm a new glasses wearer, so when I have them on, I can't see you. Anyway, forgive me. So the best way to scrutinize approach is to test it. And I'll take you through what I learned in a pilot. And this isn't going to be one of those case studies where everything goes really perfectly.

Our continuous discovery program sought to inform the direction of one of our core categories. And I wanted to start with some really basic fundamentals. Because in our business, we do a lot of research in product. We do a lot of evaluation. But we haven't actually explored generative discovery research.

So I wanted to understand is, is this sort of research even needed for our business? I work in streaming media. It's not the most complex product space, and we have a lot of feedback to act on. I wanted to prove that generative research would produce new insight. I wanted to tease out the story based interviewing technique, that it provided contextual and behavioural insight, that findings would be actionable, inspire ideas, and provide strategic direction compared to existing methods that we used, even if that was just people's ideas of what we should do.

Our business also has no shortage of ideas. I work in media. It is full of creatives. So the process needed to demonstrate that it could validate and invalidate assumptions, as well as inform prioritization and increase our confidence in the road map. And in a consumer facing industry that we're in, where everyone is the customer, I also wanted to see if it influenced us and my colleagues to reference the research and rationalize decisions through the voice of the customer rather than their own experience.

So how did we do? Okay. Assumptions were validated early, but a good body of research was needed before new insights emerged. On the key product question of whether we could inform better priorities, reality hit, and competing priorities took over. That never happens in product. Right? Now football fans in this audience may know what I'm referring to. Secondly, I wanted to understand if continuous discovery was the right method for us. Would it be useful?

So the criteria for this was whether we could establish a more regular research cadence, whether the opportunity solution tree was helpful for mapping the product space and prioritising work, and if the team generated new and varied ideas to address the opportunities. On the product side, success looked like more time and scrutiny of underlying assumptions, and if tested, shifted to challenging assumptions rather than testing usability of solutions, and if the activity for the whole team was also regular and sustainable.

What we found? The research was intense. It was the right size for strategic research, but too big for this process. That's my fault. That's how I designed the study, so that's my lesson there. Creating the Opportunity Solution Tree took some practice, and I personally, especially, fell into traps of affinity diagramming.

And they're similar, but they're not the same thing. The Opportunity Solution Tree is more of a filter to actionable insight, and the detail of the context is best kept with that participant snapshot if you want to be true to this method. On the product front, same competing priorities took over, and this effectively paused our pilot. Thirdly, on some practical aspects that are operational but really crucial for something designed to be always on.

Did we have the right research tooling in place to scale and sustain regular interviews? And whether AI moderated interviews, which we did try out in addition to complement our human interviews, human facilitated interviews, if they were of comparable quality but at a lower cost and time investment. Okay. What did we find?

The report card. Yes. But automated scheduling would have been really helpful. And the findings from the AI moderated interviews were better than expected, also terrible. But they are lessons for another talk, and I think Steph actually just gave that talk before, so that really worked out well. So overall, the report card had mixed results.

The practice requires a dedicated team, and we need to wait until our competing priorities settle to create the right conditions to try new ways of working. In the meantime, the opportunity solution tree will stay, and we'll explore how we use this further. So those were the lessons from the pilot. It's a really dense framework.

There's a lot to learn, and maybe, for me, some things to unlearn as well. But there are really valuable practices within it to incorporate into, you know, how we work today. What was clear, though, through continuous discovery and the always on research part is that it builds UX research practices and people into the product process. So if you're considering an always on program of any kind and here, I'm not advocating for the democratizing research part.

I'm not advocating against it either. But if you're considering an always on program, treat the program like a product. Begin with discovery to understand business needs. And importantly, understand the skill gaps in your team, because it is a fool's errand to try and upskill people in a new framework that comprises lots of skills all at the same time.

You really need to break down how that learning is going to happen. Set new expectations of people's time and collaboration, and start small. Your role as a UX designer or UX researcher is facilitate the program, frame and prioritise research questions collectively with your team, actively highlight the opportunities for new research, and facilitate debriefs and prioritization.

I really believe that this framework provides an opportunity for UX and UX research to become a partner in the decision making process. And improve the program, you know, through retros and feedback, streamline what you can and test with emerging technologies, and test and adapt what works for your context.

And with that, I wish you a happy discovery.