Real-World Vibe Prototyping at Google Maps

Thank you. Please join me. Happy to be here. Thanks, everyone. Behold my low resolution animated GIF. So if you've opened design social media anytime in the last twelve months, you would have been inundated with AI thought pieces. So they're generally of the same kind of theme, like frighteningly simplistic, the new natural language is the new design tool, UX is dead, you just prompt, vibe a bit, and a flawless interface pops out the other side.

For the last year, we've been absolutely intoxicated with vibe coding and AI velocity. I've seen product managers skipping the design process totally, prompting, building prototypes, front end code with natural language, And for a moment there, we thought our jobs were gone. But the good news is I'm here to tell you, vibe coding is already dead. The thing we're finding in the background at Google Maps is that generating scrap UI from thin air is a prompt is is sorry, is is a pilot trick.

It's magical the first time you see it, but as soon as you try to scale it to real nondeterministic users, and we have 2,000,000,000 of those, but you can either with smaller subsets, it starts to fall apart. Just to be clear, the act of writing lines of code and basic UI layout has been completely commoditized.

So the currency of UXs now is no longer the static asset. The Figma link, the deck of insights, the prototype, it has no value. So I recently read this great book by the legendary physicist Robert Richard Feynman, and he famously said, what I cannot build what I cannot create, I do not understand.

And so his genius was around active creation over passive consumption. So he believed that if you wanna really understand a system, you have to build it from the ground up. As UXers, our roles are shifting. So we're moving away from being creators of static artifacts to being product builders, to being strategists and creative directors.

The hardest part that I've seen as we've been going through this AI transformation in Google Maps is literally the first step, like how do you get over the hump. So in late twenty twenty four, when Andre Kapathi coined the term vibe coding, my team dived into it.

We'd already been working with AI pretty heavily within our products and, like, prototyping and designing AI for our products. But recently, we've moved away, and we're now prototyping AI for our workflows, for our daily operations. And when we started doing this, the thing I started to see as soon as the broader UX team, we have about 250 UXs on the team, we fell into what I call the whirlpool of AI process confusion.

So there are three steps to the whirlpool of confusion, and the first step is literally the hardest, just getting over the hump. So we're being inundated with AI noise. It relates to AI paralysis, but you just need to get over that hump. You need to start prompting, understanding how prompting works and prompt engineering, and then building a prototype.

Then once you're over that step, then you move into the second phase, which is how do I integrate AI into my personal workflow? So how do I use it on a day to day basis? And when you mastered that, then it's like, how do I make it repeatable for my team, for a larger work stream, for an organization? And that's the really messy bit.

So that's the three steps of the whirlpool of AI confusion. I don't know what happens next. There might be a fourth step. It could be a 12 step program. I don't know. I talked to my therapist about that one. So processes are evolving. We're hearing a lot from our industry about we gotta throw everything out the window. For years, we've been worshiping the like, processes like the double diamond. You know, people are saying it's dead.

People are saying it's not dead. People are saying we're all gonna die. The truth is, from my perspective, we used to spend weeks in sprints playing Crazy Eights, like debating whether a sheet should have rounded corners or square corners, and at the end of the sprint, we ended up with pretty much the same thing we started with or a direct copy of our competitor.

So we know that innovation doesn't happen in these kind of forced workflows. Like, it happens at the intersection with sparks. So we've gotta be kind of agile enough to be able to move with different processes to be able to get over that first hump. So in the double diamond, we have divergence and then we have convergence. AI blows apart the divergence part immediate immediately.

It compresses the timeline. So you want 50 variations of a UI layout, round corners, square corners, you got it, ten seconds, and it's done. So AI accelerates divergence but human judgment still rules the convergence stage. And that's where we're always gonna need a human in the loop, in the convergence stage.

And it means that human the human act of curation is the only thing now in the process that carries the true value. And this shifts our daily metric of labor entirely. So I used to spend 80% of my time writing front end code and designing and about 20% of my time does push sorry, in strategy and planning about the thing, like how am I gonna do it. That's totally flipped now.

I've seen this personally. I've seen it on my team, and I've seen it across the industry. We're spending 80% of our time in strategy and planning with agents and about 20% with manual execution, so pushing pixels and writing code. For me, that's not a crisis. That's an opportunity. Right? That's an upgrade. The actual manual labor of building the tactical assets, being a tactical asset monkey, has totally been removed.

And that's what we've always wanted. Like, we wanted a seat at the table. So we're becoming true product builders, strategists, creative directors, but the challenge is if we want to do that, we have to own the architectural layer. And I mean that, like, in our day to day work, but also as an industry, we need to define that that's where UX sits. We own the architectural layout because if we don't do it, someone else is gonna do it.

Software engineers are gonna do it, product managers are gonna do it, and they're gonna define it for us. Oh, I had a slide on the printing press here. I might skip this one. It's just like an analogy. You guys are smart. You get it. So I'll just get into the nuts and bolts of how we're doing this in Google Maps.

So how do we become the product builders? My advice is don't build prototypes, build tools that help product teams understand the problem. On my team, we call these data sketches, and you can see a few of them here. And so this is when we start to move this to the second phase of AI confusion where you're moving it into a workflow.

But absolutely, don't just build prototypes. Your goal is to kind of explore the landscape. Let me just oh, look. I can come over here and press play. Thank you, YouTube. Let's explore the landscape. Expose the data. Build parametric sliders and knobs and dials to let people understand the boundaries of what's possible.

So you wanna expose the data, like all of the things, the LEGO pieces that are available, and then you wanna assign it to a user need. You wanna apply that and align it to a user need. And that's the real function of prototyping. It's not building a fake version of the product. It's understanding the constraints and the capabilities and all the cool shit we have to play with and then blending that with a user need.

But I regret to inform you that all of this has been totally commoditized. So anyone can now do this. Right? This used to be my bread and butter, so it makes me feel a bit sad. Build anyone can build a parametric slider and build data, but it's still a really important step in kind of the overall prototyping process.

Like, you still need to do it. So don't just build prototypes, build tools that help your team understand and explore the space that you're working within. And so the next part of the second step of the the whirlpool of confusion, and it's getting complex, it's called planning and orchestration.

So we have the ability to create an end to end product. If we want, we can do that. We can vibe code it. But how do you get there? Don't build a prototype. What you need to do is plan it out, and it's really important to have a strict human in the loop planning methodology.

So whenever I have a complex problem to solve, I don't just get AI to write the code for me. I'll ask the AI to create a plan of how to approach the problem. And so first, I'll work with it to create a plan. I'll give it use case. I'll give it a problem statement. I'll get it to create some flows.

I'll give it a bunch of context, so PRDs, give it briefs, I'll give it, like, foundational research, inspiration, design principles, and then I'll get it to clearly state the plan in plain English. And then I'll also get it to state how it's going to incrementally go through and solve that plan, how it's gonna execute on it. And then I'll get it to create the plan.

I'll review the plan, and usually it's a bit thin. So I'll kind of dig into it and say, okay, cool. We were trying to go deeper here, and so I'll go back to the kind of the plan of the planning stage. And sometimes I have like a plan of a plan of a plan, and I'll get it to go deeper where I need it, and I'll be iterating like this.

And this is how I spend most of my time now, on the planning stage. And then when we have a solid plan, I'll get it to output a prompt to create that plan. So I never create a prompt myself anymore, even for like, you know, menial tasks. I'll always get the AI to create the prompt for me. It's much better at writing prompts.

And then I'll execute the prompt and I'll look at the output, and usually it's not right. So I'll go back to the plan and we'll iterate on the plan until the output is looking good. And so this is the human in the loop process. And so what we're trying to do here is we're trying to close the intent to output gap.

So we're trying to, like, what my intention of what I want the system to do versus what its actual output is. We try and get those two things as close together as possible, and that's the tricky part. So I'm spending most 80% of my time doing this. And so once you master the planning loop, then you can move on to a really interesting stage, and we're still in kind of the second phase of the confusion, and this is the multi agent orchestration phase. So it's like having a powerhouse design studio at your fingertips twenty four seven.

And that's what we're building at Google Maps. And I say building because it's never like a process that's done. It's just a a system in flux. Right? It's like a big machine. We need to keep tinkering with it, taking pieces out, putting pieces in. It's different for every person. It's different for every team, every, like, work stream that we're working on.

And we use this pattern called multi agent orchestration. So we have a primary orchestrator, we can call them the boss orchestrator, and they spawn spawn a team of sub agents. And the sub agents will do something, say, like run a competitor competitive analysis, some kind of competitor audit. So then the orchestrator agent is kind of the head design audit agent and we give them skills that we've written.

We give them a persona, and then they will spawn a team of highly specialized sub agents who go around and do the task. And we've written skills for each of them. They have rules and tools. And so we might have a UXD, we might have a tech lead, we might have a UXR, we might have a strategist.

And then the boss orchestrator sends those sub agents off, say, if we're doing a competitor analysis or some kind of research, some kind of early stage discovery stuff, and they will kind of go search whatever they have access to, like blog posts, websites, you know, design reviews, the actual product itself.

And then the sub agents come back to the Orchestrator agent, and they give it their findings, their analysis. And this is the really cool part. So then we get the orchestrator agent to run a critique on each of the sub agents and give them feedback based on criteria that we've given the orchestrator agent. And then we run this system three times, and that's the magic number. So this is a trick that one of, like, a very very good AI software engineer at Google told me.

If you run it one time, the data gets very it's kind of thin, but if you run it more than three times, it starts to regress towards the mean. So we run it three times and we get pretty good data. And then we're doing this in parallel with like a standard traditional UX cycle, whether it's a sprint or a research study or whatever it is, like, your task is.

So we'll give the the Orchestrator agent will take the findings from the sub agents, we'll synthesize it, and we'll give that back to the team. And so the team's doing this in parallel as well, but we've been able to do this really quickly. And then they'll fold the findings from the agentic system back into their work, and then we'll take their output and put it back into the system, and we've got this team in the loop.

And we've found that we're able to explore much further breadth much more quickly than we otherwise would have. So I'm just gonna show you what this looks like in the real world. So this is when we start to get into stage three of the AI confusion, when we start to integrate AI into an actual work stream, like an actual organization.

So on the Google Maps team, we run a lot of, like, extensive quantitative research, testing, validating end to end user journeys. We have 2,000,000,000 daily actives. We have 4,000,000,000 users in total, and they're incredibly complex real world scenarios. So I'm gonna read one because it's long and I can't remember it. So can a user successfully navigate an unfamiliar multilevel transit station in Tokyo during peak hour?

And then we can add more to the tail. So and then they need to go to the toilet, and then they need to grab lunch, and then they need to friends ad nauseam. Right? And so we'd usually do this. We do this with thousands of users, all different personality types from all around the world. And traditionally I don't really have any more slides to show you for this story. I'm just gonna talk it.

So we this would be a massive logistical nightmare. So we'd have to hire an external vendor agency. They'd have to coordinate participants. We'd have to run hundreds of permutations of this, And then the vendor agency spends months build create synthesizing the data. They create a deck, throw it over the fence, and then no one actually reads it because we've lost interest by that point.

So instead, my team built an agentic simulation system. We're building because it's still in progress, and it does most of this. So we have specific user personas. We're able to place them directly in a simulator, so whether that's the iOS simulator, the Android simulator on the web, and we can run these automatically anywhere in the world, like Mumbai, Jakarta, Tokyo, Sao Paulo.

And the result is we're able to run hundreds of these journeys in a really short period of time. It's allowed us to free up a massive amount of vendor budget, and it's also allowed the researchers to target, like, higher value user empathy work, and it's also allowed PMs to instantaneous almost instantaneously get feedback on where the bottlenecks are and like the pain points in the app for the long tail of users, of the 2,000,000,000 users that we don't usually contend with.

So I just wanna be really clear here. We didn't automate away real humans. We've automated away the menial repetitive task of finding predictable bottlenecks in kind of huge amount of use cases. So all of us have this system ready for us right now. The tools are ready.

The only question is how are you gonna orchestrate for your needs? But I just wanna be clear, this isn't paradise. There's a lot of danger here. When there's anyone can build a prototype, there's an immediate push for velocity. So the need to ship skyrockets, and that's a trap.

Prototypes aren't product requirement docs. Just because you can build something doesn't mean it's the right thing to build. AI velocity can turn into an engine for product and shitification. As the cost of production moves to zero, the volume of digital noise skyrockets to infinity.

And as UXs, our superpower was never the manual labor of creating the asset. Right? Our superpower was the relentless focus on user centricity. And that superpower is more important than ever now. So it's our job to own the architectural layer and defend it against product and shitification.

So the way we're doing that at Google Maps is through rigorous, centered, grounded user data. So it's grounded in factuality and real people. So we need to have a human in the loop at every step of the AI process. And if we don't do this, someone else is gonna write the prompts for us. And I promise you, they won't be thinking about the user.

Alright. I'll start to wrap this up. So there's three stages to the whirlpool of AI process confusion. The first is just you just need to write a prompt in a prototype. You just need to get over the hump. If you're not there already, just do it. It's not hard once you start doing it. And the second is integrating it into your personal workflow.

So it's not just building some random experiment. It's like, how do I actually use this in my design process? And then the third, which is the messiest part, is how do we make a repeatable process so we can scale it for a larger organization or even a team or your partners or a work stream. Oh, wow. I had separate slides for each of these.

I'm sorry. So that's them. So I imagine I I this is what I've observed. I'm not sure what happens after, but this is kind of the layout. Right? You're either starting with AI, you're using AI, or you're trying to integrate it. It's a business transformation thing.

Alright. So how do you actually take this and go to work on Monday morning without having a mental breakdown? I have two things that I suggest and maybe a third which is see a therapist, which I've been doing and it's been good. But the first is piggyback on a low low risk work stream.

So don't wait for an exec to tell you what to do to mandate it. Choose a boring medial task that's been on your plate for

a

while, and run a parallel agentic work stream or even just an AI work stream and see where the AI can you can insert it. And just kick the tires, see what works, what doesn't work. And then the second thing is become a vibe champion, become an advocate. Be the one person on your team who gets over the hump and does the thing and stands up for the users.

So over the last twelve months, the narrative has been terrifying. People have been shouting from the rooftops that design is dead, UX is dead. We all thought our careers were over, but it's not true. It's simply evolving. UX didn't kill design. Sorry. Vibe coding didn't kill design.

I'm dyslexic. I actually am. It liberated it. So the parlor trick of vibe coding is at an end, and the era of systemic design orchestration and curation is here. So stop making manual assets and start directing the and curating the the the behavior of these systems.

Cool. Well, thank you. That's me.

Thank you, Sam. We have, time for some questions for Sam. We have a question here and then a question here. So we'll start here.

I'm not an expert. Alright.

Let's go.

I'll try. I might have more questions for you.

Sam, thank you very much for

the presentation. I had a question because you work specifically in Google Maps and there's, so for example, a peak in, on a Monday morning in Melbourne will be quite similar for a lot many users who will be coming into CBD for work. Yeah. From a data perspective, do you create synthetic data or digital twins to curate user journeys?

Sorry. Could you just go back a step? Are we talking about live data or are you talking about data in kind of a UX and research capacity?

Both perspectives.

So we analyze real time data, and there are kind of issues that happen with problems at scale. So if there are roadblocks or traffic, then we will detour people, and then that will create more issues with, you know, scale, and so there'll be a roadblock over there. Is that and then you're but you're asking, do we actually test that?

No. I wanted to understand. So you know how there's a lot of use of synthetic data to kind of predict behaviors

Yeah.

Model behaviors.

Yeah.

I wanted to know if this is something being done in the design of

the procedure. It is in a way, but we're treating it as one signal. We're not taking synthetic users seriously. We're using it as a signal to kind of understand predictable outcomes for 2,000,000,000 users so then we can focus the team on the actual the real work of empathy. Yeah. Yeah.

Hi. How's it going? So I wanted to ask I feel like it's a really big fundamental question a lot of people are asking themselves at the moment is, are we over Figma? Are we moving away? I know it's if there's anyone in Figma here at the moment, I'm sorry.

There you

There's this big question. There's this big question I think we're all asking is do we go from conceptual straight into these built prototypes or are we asking a really big question of are we keeping Figma at the core of our design and the core of our processes? And I think it's a really fundamental question we're all asking ourselves right now.

Yeah. I don't know the answer, but we're in the churn.

Tell me.

Yeah. No. I I don't know, but

I mean, I used to be a Flash developer, and I was an iOS developer, and, you know, before that, I was a web developer. So we've kind of seen these changes previously. I think the tools are changing, but the kind of the foundational, the fundamentals haven't changed. I So don't know. I think there's a place for everything.

Like, I still use Photoshop, but I don't know if UX is Figma. Right? I don't think those two things equal each other, and I don't think they ever have, but there's been a kind of a misunderstanding from cross functional teams that that's where we sit, and that's why my point is that we need to start owning the architectural layer as an industry because product managers there's a street fight going on right now.

I don't know if you guys are seeing it, but like everyone is trying to work out who that triad is, what are those smaller jobs, like there's things are gonna change over the next twelve to eighteen months and you need to be on the ground. If your skill set is Figma, then I'm not sure I've never really used Figma that much, so I I don't know.

Like, it's it's just it's gonna be challenging if that's what you're left with as opposed to being the builder. Yeah.

I remember standing in the audience listening to a talk at the IA conference in, god, Memphis in 2009, and one of the talks was talking about the the the role of wireframe person.

Right? And that if your job was to make wireframes Right. Then essentially, needed to reskill because creating wireframes as a role Yeah. Was going to change and was potentially gonna disappear and was potentially going to you know, like, you'd you'd be out of work, basically. And it was put in the same category as being a person in the architectural industry when CAD came along.

Tools continually evolve and change and the way in which we use tools continue to evolve and change. I think to your point, Sam, if your value is your ability to drive a piece of software, then that is always precarious.

But it's an interesting point because the goal of a wireframe was always just a simplistic way of conveying the functionality. It's a huge tool. Yeah. So now it's easier to do in other ways, and then we've kind of gone almost to the opposite end of the spectrum where it's now you can just create the finished product, and so we're kind of there is a space in the middle that we need to be owning and we need to be thinking about where Indeed. We're solving the problem.

Hillary. Thanks.

Thanks, Sam. That was really fascinating. So the multi agent orchestration piece that you're doing

Yeah.

Is really cool. How mature are you in that space at the moment? Are you still experimenting? You're rolling it out in earnest? I just wanna get a sense of how

Yeah. For sure. So I'm leading digital AI transformation for GeoUX, and so we've got multiple teams using a system like that with a team in the loop, but it's hard. And I think it's more of a cultural issue than anything else. It's like people kind of reticent to move on to new ways of working, particularly in kind of a team dynamic.

So we have lots of individuals who are kind of working, integrating it into their workflow, but then once we all start working together, even if it's like a group of three or a group of 10, we're it's a new space. So I think we're now our VP of design, I think he described it really well. He said this is a multi year journey.

It's not a destination, and so we need to kind of pave the way along that journey, and I don't know if it's right or wrong. We're just kind of making mistakes and learning things right now. Yeah.

Just on that point, Sam, culturally, how patient is the leadership at Google for you to try and learn and find your way?

They're very patient. Yeah. Yeah. It's really they've given us a lot of space. I mean, it's the only way forward. I think that they're the particularly UX leadership are very empathetic, and so they understand that, you know, they've been around the block. This is like a new phase, and you can't just rip things out of the roof.

So there's either it's kind of it's biz organizational change management, right? And so there's two ways you can do it. You can do it top down or you can do it bottom up. And so we've chosen approach which is top down has given us some guidance and then they wanna see how we organically work from the bottom up and so because we have it's 250 person org, we have smaller teams and everyone works differently.

And so if you're mandating a top down system for working, that's never gonna happen. Right? So yeah. So we're kind of working away as we go and I think that's the way it's always been at Google. It's like let a thousand flowers bloom, everyone we have smaller pods and everyone works differently, but you have to do it.

That's the mandate. Right? You have to do it and we've carved out space, so managers like myself will we've deprioritized certain tasks, certain work that is kind of lower value and then make space for this this work for for learning and educating, but I think the challenge is now's the time to learn it. Right? Now's the time to get on board because if you don't do it really soon, it's gonna accelerate and you need to get to step two pretty soon and then step three is gonna happen when all of the juniors are gonna come through, all the new grads, and they're just gonna be AI natives, right, and so

Excellent. Down the front here, one last question. Think maybe we'll get to yours, but down the front here, please.

With that

level three where the AI Level three autonomy. Yeah. Yeah. There any tools yet that are really good for that? There probably are, but we're forced to use Google tools. So we I like antigravities. So Yeah. And I use antigravity almost every day or every day.

Yeah. But it's not built for change management. And that's I mean, there's lots of business tools. Right? So this is kind of like a an executive level thing of, like, how do we do this? And it's also kind of, like I said, a cultural thing. It's also about, like, bringing people together. It's a management thing. So there are tools, but I don't know if they're software tools.

Thanks. Alright. We do have

time. Alright. Thanks. I've got a clock.

I just got a question about well, not a question, more observation about mental models. And some of us are like spatial thinkers and some of us are more like gifted with words. And given your work with maps, I was just wondering how your approach is different because you're dealing with spatial information wayfinding?

Yeah. That's the big question. Right? That is, like, literally the $20,000,000,000 question. We can talk about this afterwards. I'd love to talk about it, but it's it's a huge space, and I'm not an expert there. But it's really interesting. Different people use maps in different ways. So in particularly in different cultures and different countries, I don't wanna pick on, say, India, but in certain areas of India, certain people, and also other regions are more visual, and so we have a different skew of Google Maps for transit and for directions and active navigation where we'll use more imagery, and so instead of saying turn left at, you know, in 200 meters, we'll have a photo and say turn left here, that's not and then they can switch that off. And so but then also other people like myself are dyslexic, and so we read I read a map in a different way.

Some people like the map facing north up all the time. Some people like it to move around with them. Some people like north up at the start, and then when you go into active navigation, then they like to have directions. It's a really incredibly interesting space, but, yeah, happy to talk more about it. But then that kind of ties into AI and one of the conversations that was happening before in one of the previous presentations where it's like, do we have text boxes or do we have more of a visual, like, interactive drag and drop interface?

Yeah. It's interesting. This is kind of the the thrashing that's happening. Please join me in thanking Sam. Thank you.