(Re)Designing Research: How AI is shaping research practices

Hello, everyone. It's great to see you all today. I my name is Stephanie Moss. I'm the head of design and research operations at SEEK. And today, I'm here to talk about redesigning research, how AI is shaping research practices. So this is a great talk, very fun, to be delivering after Christmas talk. There'll be some complementary ideas as well as some contrasting ideas.

I also hope you all like pink. There's gonna be a lot of it. So before I get started, I do wanna do an acknowledgment of country. So I am grateful to be standing today on the traditional land of the Wurundjeri Wurrung people of the Kulin nation. I pay respects to their rich cultures, to their elders' past, present, and future, and their continuing custodianship of the land, waterways, and communities on which we all rely.

So to start this talk, I did want to talk a little bit about why I'm specifically here on this stage talking to you about AI. There's a lot of people saying a lot of things about AI for a lot of different reasons. And my reason specifically is very simply efficiency. When I was a designer, I was the type who wanted to design a form really well and then never do it again.

I wanted to move on to bigger and more interesting problems. It got me into design systems really early on, and then it's led me into design and research operations. So efficiency is a very much a core motivation of mine. And so when AI came along with its sweet, sweet promises of efficiency, I was very intrigued, and so I'm doing my best to learn about what this is all going to mean for the practices that I work in.

There are a couple of topics I won't be covering today, and I do wanna explicitly call them out because they are really important, but I'm not gonna be covering them in the thirty minutes that I have. So one is I won't be predicting the future. Not something I can do, and the singularity is gonna have all bets off at that point, so I'm gonna be talking very much about the capabilities of the here and now.

I'm not poking fun at AI. It does make mistakes, and there are some admittedly very funny ones, but I'm not attempting to mock what its capabilities are. And finally, I'm not making a judgment on if it's good or bad. As humans, we do have a lot to reckon with when it comes to AI, legally, ethically, environmentally, and those are all really big, important topics that deserve talks of their own.

But for this particular one, I'm not making a morality judgment on AI itself. So efficiency, my beloved. This is the formula for efficiency. It's a bit scary, so for this talk, I'm going to be talking about like this, where efficiency equals value, so things like quantity, quality, impact, over the cost, such as money, time, and effort.

When we think about efficiency, both these things are equally important. When it comes to conversations about AI, I am finding there is a lot of conversation about the cost factor, reducing cost, especially time and effort. And that is all very valid. But I am finding less conversation about the other side of the equation, which is what is the value that's being provided and what is the impact?

So this is a graph some of you may recognise. Speaking back to Christopher again, this is from IBM. It is from about 2010. It shows the cost of fixing security defects, actually, across the software lifecycle. And it's really quite a basic concept, which is the later you fix something in the stages of delivery, the more expensive it is to do. Makes a lot of sense.

Right? And I'm very familiar with this because when I was very early on as a designer, I remember asking an engineer, can he fix a small thing in the UI? Wasn't a huge deal. Would improve the user experience. And he turned around and said, it's gonna take three hours to deploy to the monolith.

I was like, do know what? Don't worry about it. That has changed over time. So there has been a downward pressure on technology in the cost of delivery. And a large part of this is actually engineering practices themselves focusing on efficiency in how they do things. There's been huge amounts of evolution in DevOps, in continuous delivery, that has reduced the cost of delivering products.

In this time, design and research have seen some change from their technology, in particular around tooling. But how we do things hasn't dramatically changed. Some of the result of this is that we have seen things like dual track discovery and delivery, trying to fix in design and research into the agile workflow somehow, as well as things like continuous discovery, where we'll do much smaller pieces of research very rapidly in order to try and keep up with delivery.

So already, there's a bit of a gap here in how we do things between design and research and the delivery side of things. Then AI comes along. Guess what it's doing? It's creating even more downward pressure. Over time, it's gonna take less engineers, less time to do more.

Now this is also going to impact how we do research. There will be AI benefits there. But what we are looking at is this decrease in return on investment. It's suddenly not as important to learn things from research because the cost of fixing them further down the line is not as big as it used to be.

Still there, but not as big. And so over time, what we've seen is things like AB testing or multivariate testing, and sometimes just the good old let's ship it and see. So it's just become much easier for people to put things out, and people aren't always willing to wait for things like research. This might seem like a really, really bad thing, but there is a part of this equation that's coming from AI, which is companies are gonna start doing this, and they're gonna look look at how much we're shipping, look at how productive we're being, look at how efficient we're being.

It's going to be an absolute smorgasbord. We're suddenly able to do so much more with the same. And that's an interesting thing to look at. I really wanted to look at this in terms of that efficiency equation. So cost? For sure. Provided that the AI tooling itself isn't astronomically expensive, that it outweighs everything else, there is a reduction in time and effort required to do a lot of the things that we're currently doing.

The question mark I have is around the value. Because rubbish delivered quickly is still rubbish, And this rubbish is going to accumulate very quickly in our experiences if we're not careful. Now some organisations might be cool with that, and they're just willing to take the risk, but there will be a lot of other organisations which are like, it's just gonna start costing stuff over time.

So the value that is coming there isn't a value in the equation for efficiency for it to really matter. And so the value of research, again, is going to be helping organisations identify what are the things that matter, what are the things that are actually going to add all this additional value?

And so the intention behind research hasn't really changed in this model. It's still about identifying what are the right problems and what are the right solutions. What has changed is the scale and the speed of which these things need to happen, which is quite dramatically different. And so I spoke a lot about why businesses might care about research and what kind of role it plays in how we're gonna be delivering faster in this world of AI evolution.

But there is this human aspect as well that I think we will all connect with, which is fundamentally making sure our research insights drive decisions is the right thing to do for the people. When we gather insights from people, they want us to use them. They're giving them to us for a reason. We could do the best research in the world, and if it's not changing the products or the services that we're delivering, it's not going to matter. So the right thing to do for the people we do research with is to make sure that we're able to do it efficiently so it can have impact.

Now, I'm going to talk a little bit about the practical side of things. I've talked about our cause, the why. Why why do we care? Why does this matter? Now I'm gonna be talking a little bit about the how. And the reason I'm saying re or designing research is because I think we actually do have to quite dramatically rethink how we do things. It hasn't changed a lot over the recent history, but AI is quite a catalyst, and there is this big opportunity for us to be doing more, which ultimately helps the people we do research with. So I'm actually going to start with a very, very controversial topic, synthetic uses, which makes a lot of people do this face.

And I was the same. I was like, no. No. No. No. But when it comes to AI, I have been trying to quite deliberately step back, not knee jerk, and think about, okay, what's the capabilities? What does this mean? How does this fit into my world view of how things work? And so look at let's look at this in terms of the efficiency equation.

It does reduce cost, provided, again, it's not super expensive. It does reduce time and effort to do research. You would fundamentally have these users at the tips of your fingers. There would be no need for recruitment. They're just there. Again, the question mark is around that value. So what value is being provided here?

And I think this is where a lot of us are obviously picking up on some really good questions. So because the way AI works and the way it does statistical probability, we're not really likely to learn anything particularly new from synthetic users because it's going to be sharing information that it already holds and usually the most common information that it holds.

Obviously, there's a huge risk of misinformation and bias. We already know AI models have bias inherent to them, and it can obviously tell us falsehoods. Unless you're working for American user bases, it is not likely to be sampled on your user base. For SEEK, for example, our user base is Australia as well as Asia Pacific. And so we're not gonna see our people, the people we're building products for, the people we do research with, in these models as dramatically as some other groups.

Fundamentally, and this is a big one, you're not empathising with a real person. It's the same as empathising with a character from a book or a movie. It's a fictional character, which is fine. We can empathise with that. We're able to do that with fictional. But it's not a real person. And so I did sit back and ask myself the question, okay, how would I increase the value of a model like this?

So a really obvious one is to use your own user data. So making sure that the model is based on real people that are relevant to your company. You'd probably do something like create segments and groups, you're making sure that you're covering relevant groups that your company served, especially underrepresented groups that you wanted to make sure were covered.

You define motivations and behavior. So, for example, with seek, job seeking, hiring, that's what we would be looking for. And you'd model context and variables such as, like, these they use phones. They live in this country. These people have poor mobile broadband. And what this all made me think of fundamentally is an interactive persona. Now, a persona is perfectly valid.

It's a tool that's been around in UX for a really, really long time, and you could push the functionality of it quite far with AI. You could have it talk, respond, you'd be able to move the mood around, lots of really cool stuff you could do. However, our question is still the value. Was this a problem that needed to be solved?

Was there a lack of value in personas to begin with? And by doing something with AI, have we increased the value? There may be an opportunity here. I'm not particularly convinced of it, but it may be there. I do, however, believe there are some much more interesting and bigger opportunities we can focus on. And I'm gonna be talking about those now.

So this is the current process for research at SEEK. It is quite detailed, but you may notice it is very long. And there are legitimate pain points and inefficiencies all the way along this process. And so at SEEK, we've been looking at this and going, okay. How do we make this better? How is technology changing how we may think about some of these things? And what are we doing for the scale of our business, especially since we do research across many different markets, different languages?

What are all these things that have changed over time that we need to take into consideration? And so the process is starting to evolve into something a little bit like this, and there are three areas that I wanna talk about specifically today. They're not the only opportunities, but they are three quite interesting ones. One is information retrieval, one is AI moderation, and the final one is synthesis.

So diving first into information retrieval. Tell me if this sounds familiar. Hey. What do you know about this particular topic? And you go, let me go look, followed by the longest sigh ever, because you know you're about to spend half your day trying to dig up this information to answer this question.

There is no doubt there's a massive amount of value in insights. They can drive decisions. They have impact. They are a core part of how organisations can flourish. But there is a huge cost in information retrieval. And the reason that there's such a huge cost is because information is stored all over the place.

The larger your organisation, the worse this is going to be. If you are lucky, you will have access to where that information is stored. If you are unlucky, it's stored on somebody's desktop. There's a very high amount of effort required to curate repositories, which have been, to date, our solution for managing this type of information.

Typically, a lot of the tools had very low functional search. So unless you very specifically remembered the name of a piece of research, good luck trying to find it. And finally, no summarisation. So even once you did find pieces of information, you had to read through all of them to decide whether or not they were relevant. Again, very, very costly in terms of time and effort.

AI has come along and reduced some of this cost. It is quite good at information retrieval. It's one of its strongest capabilities. It is very efficient at summarisation. It can read and go through things quicker than we ever could, and then return information to us.

And it does have the ability to simultaneously draw from multiple data sources. Again, something that we, as humans, would find quite difficult to do. So the cost has improved. We've reduced how much cost it takes. It's less effort, less time. The question mark again is around value. So there was high value. What are we doing to make sure that we maintain that high value? We don't wanna lose value because then we lose efficiency.

And so some of the things oh, so this is a really good example of why you don't want people just going off and doing information retrieval. AI can do some very, very strange things if left to its own devices. And there's a lot several different reasons why this happens, but it just fundamentally underlines the point that AI may be wrong.

And so I, in design and research operations, I'm not the one who's probably responding to these requests. What I'm trying to do is design a system so that everyone at SEEK is able to do this efficiently. So I'm much more concerned on not how one person will do this, but how hundreds and thousands of people will do it.

And so I don't want people running around writing their own prompts. People are typically not very good at writing prompts at this point in time. And so there would be a huge amount of lost efficiency if I just let everybody do whatever. And so the way we are solving it currently with our current capabilities is through agents.

And agents are quite good because they allow us to design a workflow. Several prompts along the way where we can guide a user who's using AI to do a thing to get the right type of outcome as best we can. So we're putting boundaries and guardrails in place. So some really good fundamentals, like, you can make agents as complex as you like, but some good fundamentals are make sure that you're guiding the person using the AI to give the most important context.

AI thrives on context. If you take it away, it gives poor responses. Secondly, you might wanna actually direct it to certain data sources. There might be certain data sources that are more accurate, more up to date, more relevant. So you might not want it searching the entire Internet. You might want it to search particular things.

Then you'll be wanting to tell the agent, once you've found all these things, this is how you need to think about it. Here's the context. Here's how you should be breaking it down. We should be restructuring how the agent then responds back to the user. There's an example there that Wade and Lee in my team has done, which is an excellent part where we have made sure that it responds by covering user behaviour, trends, sentiment, common pain points, differences across different regions.

You can do all of this to make sure that when it is providing an output, there's a consistency to it, but you're also making sure that it's very rich in data and easy for the person to consume at the other end. And finally, and I do believe this is incredibly important, which is make sure it provides citations.

Regardless of how well we set up agents, it can still be wrong. And citations allow people to check not only if it's correct, but how important is it. Where did this data come from? Did it come from Slack, or did it come from a research report? That matters. The second one I'm gonna be talking about is AI moderation.

And AI moderation is a very intriguing one for me, and I'll go over why. So there's a huge amount of value in moderated interviews and research. Huge. Like, nothing can replace us talking to people. And so there's a really good question of, okay, why would we use AI moderation? And the reason for that is because it is quite a high cost attached to doing moderated interviews.

And so there is a reasonable question of, like, can we increase the efficiency of how we do this? Now, interviews are time consuming, but that's actually not where I'm most interested. I think there's some lower hanging fruits here. And that's fundamentally around more recruitment. It's actually quite hard to reach the the right participants.

Depending on which type of industry you're working, this can be very, very difficult. At SEEK, we have hires who work our clients. We also have candidates of the people working. It can also be very difficult to schedule time. Funnily enough, our work hours overlapping with their work hours suddenly doesn't play out so great when you're trying to schedule an interview.

We also have things like time zones come into play where there's very little overlap in terms of our working hours with the times that they're available. And, of course, there's this really high cost of no shows, which are incredibly disruptive, and I'm sure all of you have faced this frustration in the past where you schedule an interview, you're all organised, and they don't turn up.

We've actually done research on no shows and also why people are motivated to take part of research. And so it's not often that they're rude. It's they've got conflicts. The thing that you're doing isn't as important to other things that they're doing. They've had a meeting scheduled over it. There's a lot of different reasons. And so the question is, what can AI do for us?

AI is interesting because it doesn't have fatigue. It doesn't get hungry. It can keep doing research for as long as it runs. It is available at any time. It doesn't have working hours. It does not need to sleep. It's able to recognise language and respond, which is something that it has over surveys. It's able to do things like transcription and translation, which is really interesting because not a lot of people can speak all the languages that a user may be speaking in in an interview. And so cost is being reduced.

Excellent. How do we make sure we keep that value, though? Because interviews are very, very valuable to us. If we give it to AI moderation, what happens? I think the best way to think about this is that it's not a replacement methodology. It is a new methodology. It is a methodology that sits somewhere in between interviews and surveys.

It has disadvantages against both those types, but it also has its own advantages. And so when I think about redesigning with AI moderation in consideration, I would recommend that we actually consider it to be a new methodology and use it as such. Leverage it as something that is additional to your interviews or your surveys. Use it to talk to participants that you can't reach otherwise, people outside of your time zones, people who are not available when you're available, people who need to do things at their own time.

Use it to fit in with the participants' schedules. This is actually really great for their user experience of doing research. I don't have to fit in with when you are needing to do this interview. I can do it whenever I have the time to do so. And you can use it to scale discovery insights quickly. So let's say you have done some moderated interviews, you have some findings, and you're wondering, I wonder how many people think this, or I need a little bit more to back this up.

You can use this type of methodology to gather more data at scale without necessarily investing the time to do thirty, fifty, whatever odd interviews. So there is this really big opportunity there where we can start balancing out this cost reduction with making sure we're maintaining this value that we have.

The final example I have is synthesis. Synthesis is a really interesting one. I think it's a little bit unique in a way, and I think it goes a lot to Christopher's talk where there is actually quite a massive difference between a human synthesising something, having information go through your brain, you connecting the dots, versus an AI doing it.

So there's huge amounts of value. We know that doing synthesis is incredibly enriching. It's how we get it's how we make meaning of the research we do. It's how we make transform it into data into insights. So huge amount of value, but very, very, very costly.

Synthesis is very time consuming. I think the old baseline used to be three times longer than whatever you spent on doing the research itself. It has a lot of manual activities attached to it, such as manual tagging or coding, which requires people to go through transcripts and individually pick out things. Valuable but costly. And because of those two things, there's actually a lot of insight wastage.

So what typically can happen in a lot of teams, especially when you have cross functional teams doing their own research, is they're gonna do the research, then they're gonna take the insights that matter to them, and they're gonna run. And they're going to leave a lot of insights on the cutting room floor, unsynthesized and unreusable by anybody else.

Hugely inefficient. Research itself is quite costly to do, valuable, and then we're just chucking out a whole bunch of insights because they don't matter to that particular team at that point in time. So we're looking at how AI can reduce that cost, and it has a lot of really key benefits.

This is pretty much everyone has used AI for synthesis of some sort. It's one of the first activities you are likely to do with an AI. It's very efficient, a summarisation. It can read through many, many, many transcripts and tell you what's the general summary of what people talked about. It's quite good at identifying patterns, sometimes potentially better than we are.

And it can generate model formats. So rather than you having to create a Confluence page and a PowerPoint and this and that and whatnot, it can just generate all of this stuff. It does it doesn't matter to it. So we've reduced the cost through AI. Things have become easier and faster. But there is this really interesting question about what value could be lost through this process, especially in the cases where you might hand over all synthesis. So the way that I recommend thinking about it is that it's not something where the human totally does it or the AI totally does it.

It's a partnership, a collaboration where there is a natural back and forth. So the idea is that the human should be leveraging what we do best while using the AI to do what it does best. So a researcher might be picking out their initial themes, the AI scales it, so on and so forth.

And so when it comes to designing your collaboration with AI, you need to think about spending time still absorbing the research. Do not lose that value. I think if we lose that value, the efficiency equation doesn't work anymore. We've actually probably lost efficiency. Treat it as a back and forth. So rather than giving the entire task to AI and then wandering off with it, go back and forth with it across the several different tasks you need to do for synthesis.

Definitely use the AI for manual tasks. Low level tasks like coding and tagging, the very repetitive stuff, get it to do that, is somewhat a waste of your time and effort and life. And finally, there is this opportunity to start exploring how we scale experience. There's this really interesting thing where synthesis is incredibly valuable, but it only really goes through one person's brain, maybe a couple people's brains.

Not particularly efficient. And so is this an opportunity to use AI to make synthesis matter to more people, to help walk them through the process of connecting the dots? So my final say here is that I think we're barely scratching the surface of some of the opportunities here. I'm actually quite excited about the opportunity that we have for creative problem solving and changing how we work.

Just because we are reducing cost doesn't mean we don't care about value. Just because we care about value doesn't mean we can't try to be more effective and efficient. I, myself, love efficiency. And by caring about both sides of the efficiency equation, cost and value, I am very ready to take on the design and research challenges that are coming our way.

And I hope all of you too have now got a bit more information on how you yourselves can approach this big change that is impacting us all. Thank you very much.