Why Designers Are Accidentally Breaking Customers’ Trust in AI
An Emotional Reckoning with Past AI Work
The speaker opens by describing a personal crisis of conscience triggered during an AI ethics lecture at LSE in January 2024, where they recognized harmful patterns in four AI projects they had worked on over eight years. This realization—that good engineering intentions couldn't guarantee against bias or privacy harm—led them to take six months of gardening leave to research AI ethics and safe design practices, setting up the talk's exploration of harm-minimizing AI design.
From Deterministic Machines to Learning Systems
The speaker shares their 20-year design background and introduces the core premise: AI represents the first new interface paradigm in 60 years, shifting design from controlling deterministic, hard-coded machines to designing for two learning systems interacting and adapting together. They explain that unlike traditional software, AI experiences change over time as both user and system learn, making scripted, predictable design impossible—much like a marriage or friendship.
Five UX Principles Flipped Upside Down
The speaker details how five foundational UX principles—user experience, consistency, hierarchy, user control, and frictionlessness—are being fundamentally inverted by AI. They explain that design must now focus on enabling two-way collaboration between human and AI partners, helping users build new mental models for variability, determining which partner leads at given moments, replacing control with transparency and steerability, and introducing 'strategic friction' as moments for trust calibration rather than removing all friction.
Trust as the Operating System of AI Design
The speaker presents findings from roughly two dozen studies showing transparency as the universal driver of trust in AI, and reveals that Australia ranks fifth-lowest among 47 countries in AI trust, with 64% of Australian users starting from a trust deficit. Despite this, 83% of Australians say structural safeguards would increase their trust, suggesting an opportunity for brands that prioritize trustworthy AI design to gain competitive advantage through customer retention and acquisition.
The Four Components of Building AI Trust
The speaker breaks down four non-additive components essential to building trust in AI systems: competence (communicated capability and resulting mental models), reliability (repeatable patterns users can rely on), transparency (microcopy and microinteractions explaining AI decisions), and value alignment (respecting user boundaries and cultural differences, such as contrasts between Australian and Singaporean or German user expectations).
The Non-Linear Trust Journey
The speaker outlines trust as a non-linear journey through four stages: trust formation (initial mental models), trust testing (early experimentation with low-stakes use cases), trust cultivation (building reliable patterns through ongoing use), and trust breaking/recovery (addressing the inevitability of AI errors as a core part of the experience rather than an edge case).
Practical Design Levers: Onboarding and Graceful Failure
The speaker identifies onboarding and graceful failure recovery as the two highest-leverage design investments for building AI trust. They recommend accurate mental-model setting, granular privacy controls, and constrained initial feature access during onboarding, alongside a four-part failure recovery approach—acknowledging mistakes, explaining causes, offering user choices, and demonstrating learning—advocating for shifting design effort from a 90/10 split favoring core experience toward a more balanced 60/40 split that accounts for error states.
Closing Argument: Why Design Matters More in the AI Era
The speaker closes by addressing common career anxieties about AI's impact on design relevance, arguing that design is more essential than ever because every AI interaction creates a unique, evolving experience that someone must monitor and repair. They emphasize that designers' core purpose—understanding human responses to technology—remains vital, and end with a call for community collaboration, referencing a human-centered AI community waitlist and QR code for further engagement.
Good day, everyone. So I wanna start by talking about something that is very often, chronically missing from AI discussions, and that's the emotional roller coaster that I've been on for the last couple of years as a result of AI. And I'm sure most of you are probably feeling the same. So is this conversation, by the way.
So for me. So so in it started in January 2024. I was sitting in a lecture on the ethics of AI at the London School of Economics and Politics. And about halfway through the whole course, it started to really dawn on me as the lecturer went through case studies of different types of AI harm. And after a couple of them come through, I started to sort of think back to the work that I had been doing, and I started to recognize signals in there.
Not the exact projects, but the actual patterns, the decisions, the shortfalls, the things that I had signed off on in good conscious across four AI projects that I'd worked on over the preceding eight years. Three of them were even in regulated industries, and I couldn't hand on heart say that the work I'd produced hadn't caused harm.
I couldn't say for sure whether it was bias against some people or not, and I couldn't tell you that we protected everyone's privacy. And that made me feel sick to my stomach. It wasn't from bad intent.
I mean, I was sitting in a lecture about AI ethics. I was trying to do the right thing. I knew that this is where I was going, and I wanted to actually, you know, make sure that I was prepared for this next era of my career when I was gonna be doing you know, working more embedded in AI in everything I did, leading teams across this. And it wasn't bad engineering necessarily, and it was just good design applied to the old paradigm that AI fundamentally doesn't sit within.
And so I never wanted to be in that position again. So I spent six months on a very long gardening leave, post redundancy, researching, learning, talking to experts around the world because I wanted to ensure that I understood exactly how I could work within AI and do it so safely from a harm minimization perspective.
And so that's what I wanna talk to you about today, so what I've learned through that period. Quick context. I've been working in design for about twenty odd years in human centered design, leading teams here in Australia, Berlin, Amsterdam, and Sweden. I have a master's in human computer interaction. You'd think that would have helped.
Right? And a certificate of ethics of AI. And for the last eighteen months, I've also been sharing some of these lessons that I've been learning along the way with other designers so that they didn't feel that same sickening feeling about the regret of their own work potentially causing harm. So let's get cracking.
First and foremost, you know, let's just look at where we've been. So for the last thirty odd years, we have been mastering design. We've been designing the user flows. We controlled the output. We minimized errors. We removed friction. The user navigated through what we had designed for. The product did not change while they were using it.
And we got real good at it. Consistency, hierarchy, user control, frictionlessness, these were the foundations. But Nelson Norman recently said that AI is the first new interface paradigm in the last sixty years, which was kinda shocking when I first saw it. But I was like, hang on. We're with designing different, you know, interfaces.
No. It's a totally different, you know, way of designing for this particular interface. So we started we have been designing for hard coded, predictable machines, and now we're designing for learning systems, learning systems that are collaborating with other learning systems. Right? So we've got two different forms of intelligence, each making decisions, taking action, and learning from the interaction that's happening in with each other. So where the same user comes back to the same system and doesn't have the exact same experience, doesn't get the exact same output because they both moved on.
They both learned from it. So even from your own experience, if you've used a prompt and it kind of got a really good outcome, the next time you try to do that task, you're like, well, I just remembered that one. That that that new trick I had actually worked a treat. Let me see if I can actually use that now first time instead of the fifth, you know, prompting.
That is what I mean by, like, we're all learning and adapting as we're going and trying to find things that actually work reasonably consistently over time. And so just like a friendship or a marriage, you can't script it. There'd be a lot less divorce. It turns out you can't actually fully script an AI experience either.
So there's five core principles of user experience that we've been leveraging for the last, as I said, like, thirty odd years that aren't just bending a little with AI. They are fundamentally flipping entirely upside down and breaking the way that we have been designing. So first and foremost, a pretty central one, user experience.
We're no longer designing a one way single actor experience. We're designing between two learning systems that are working in partnership towards a shared goal. So the question stops being, does what does the user need? And has to start moving towards what conditions make it viable for these two different partners to work together to get where they wanna go. So what are the mechanics, the micro interactions, the information, the the the tools that they're able to use, and how they're able to actually build that trust and interplay between them?
Now we're not all creating chatbots, and that's not what I'm talking about here. Obviously, lot of us have that in our mentality of what AI is, but in a lot of AI systems, we are still interacting with systems that are learning and adapting as we go. So the design challenge is designing for conditions that, yeah, two way collaboration becomes an effective partnership towards a shared goal.
Sorry, I have dyslexia as well. Someone said it before. Like, it might not say what it says on there. Okay? I might not say out loud what it says on there. Roughly the same. Okay. So the second one is consistency. So it used to be that the same input got the same output. That was the contract. Right? And so with generative AI, as we know, it breaks it on purpose. Variability isn't a bug.
It's part of the design. But the problem is is that your users have the mental model that has been built on decades of consistency, on understanding that whether they're using a remote control to turn on their television, it doesn't turn on the oven. Right? Like, every piece of technology we've used has actually been deterministic until this point. And unless you design for that mental model change, they're gonna be disappointed.
Because when AI behaves differently each time, they don't think, how clever. No. Once they start you know, once they get past that initial thing of how clever, they start to think, oh, how can I actually rely on this? How do I know if it's true? And if they can't genuinely work that out, if they can't see how they can get the outcome that they're looking for, they're gonna churn. And there's higher churn from AI products and services than there is from non AI products and services.
So the design challenge is to help users build new mental models to handle variability as part of the design, not to fear it. The next one is hierarchy. This one was pretty set. Humans judged, decided, tools executed in that order.
Now AI reasons, infers, sometimes leads, because it does have genuinely different capability than we do. And at other times, we lead because we have the capability it will never have. The goal isn't humans in charge or AI in charge.
The goal needs to be designing for the right partner to be leading at the right moments. And this becomes exponentially more important when we're talking about organizational design, where that rigor needs to be there in some way, shape, or form, but even more so now that AI agents are entering the the the the remit of individuals, workplaces, products, services.
Because that hierarchy, the the more automation and autonomy you give AI agents, the less you actually see their decision making, the more obscure that is. And so we're having to actually really, really think through this. And it's flipping everything from, yeah, the way we design, the way we actually, you know, as leaders, we'll very soon be leading teams that have AI agents and humans in it.
And so this is very fundamental to that. User control. Yeah. The users aren't in fully in control. We aren't as designers, and that's really uncomfortable. We've been design we've been designing the experience, and now we don't even fully have control. So control has to get replaced with transparency and steerability because the same thing isn't sorry.
The same input is not gonna get the same output, so you need the ability to at least understand what's happening, be able to make an informed decision about what you do next. And so can the user understand what it's doing and how they can steer it towards their end goal? That is the new form of control.
And frictionlessness became what I call strategic friction. So over the last decades, we've been removing steps, reducing friction, and aiming for seamless experiences. But now adding friction, it turns out, is actually the precise moments where humans can calibrate their trust.
If you observe someone interacting with an AI system, you'll notice a moment, Understated yet profound, where they hesitate before accepting the AI's recommendation. That pause represents the most valuable element of human and AI collaboration, Trust assessment happening in real time.
Can I believe what it's saying? Do I understand it? What do I wanna do about it? And how do I respond? So the pause is actually where comprehension, decision making, and your creativity lives, is in that pause.
And trust isn't a feature. It's the operating system that everything else runs on because it comes down to one fundamental truth. Trust is the very bedrock of humans' willingness to want to collaborate with other people, and as it turns out, artificial intelligence as well.
So I looked across roughly two dozen independent studies on how AI and sorry. How humans need to build trust with AI. They disagreed on a lot, but there was one thing, a single through line that every single one of them had. The number one driver of trust is transparency. And transparency for reference, which I'll go into a bit more, is almost entirely in designers' hands.
Not product managers, not engineers, you. That inter that interface between the the the user's experience and the AI's explanations and reasoning, that is entirely in your hands. And it's the single biggest lever for humans or customers building trust. Bad news, though. Australia's got the fifth lowest out of 47 countries in AI trust. So we're a skeptical bunch.
I think Robodebt did a bit of a number on our collective psyche. So put it another way, though, 64% of Australian users are arriving at your AI experience not trusting it at face value. That means that when a new user arrives at your AI, there is likely a trust deficit that you have to build out of, not a level playing field on which to build.
So it's not all bad, though. That same study, by OKPG, and Uni of Melbourne, found that 83% of Australians, said that they would trust AI more if there was structural safeguards. So while we're currently in a pretty poor position when it comes to trust in AI, it is the mechanism by which adoption and retention are built. The good news is for Australian brands who actually take this moment to build trustworthy AI products and services, not only will they keep their own customers, they will start to actually pull away their competitors' customers.
Because if everyone's got access to the same AI technology you do, it is in the whether or not someone trusts your AI over theirs. And for that reason, I think design is actually more important than ever before. So how on earth do we build trust?
If all this is going to to to crap, how do we do that? So there's four key there's four components, and they're not additive. They should all be present through the process, but at varying degrees of importance or strength, essentially, at each moment. So first and foremost, competence.
Importantly, this is not just capability. It's actually what it can do, literally. It's how well you've communicated that, so what you've actually communicated it can do, and what is the resulting perception that your users have now. So what's that change in the mental model? How do they understand it to be able to work?
Because if that is broken, if that doesn't work, kinda all bets are off about how successful they're gonna be using your AI. Reliability. It is more difficult to find reliability now, but what we're looking for is whether or not your users can build repeat or sorry. Build or find repeatable patterns that work to get them to where they wanna go for each thing.
Transparency. So this is fundamentally the microcopy, microinteractions that help your users understand what AI is doing, what decisions it's made, what data it's relied on to make those decisions. Also, they can understand whether or not sorry.
To trust the output. And lastly, value alignment. Does this adapt to the way users actually want to be interacted oh, sorry, collaborated with? Doesn't respect boundaries. Have you designed an experience that asks people whether or not they want to have privacy controls?
And have you designed the deprecated experience for when they choose privacy over full capability? And just one note on the value alignment, this is very, very dependent on culture. So if you've got Australian based audience, obviously, looking at Australian behaviors, but if you've got a more global reach, what works in Australia in a very, you know, less hierarchical culture will actually turn off people that are from Singapore, Bangalore, or Germany.
And so it's really important that you understand what is that, from your particular user's case. So trust is not binary like me. So it's a journey. And like all journeys, just just just so you know, like all journeys, it's not straight. Right? Like, no journey map is ever really been straight. Right? So then neither is this.
Also like me, just quietly. So
so
first and foremost, trust formation. Right? So, yeah, how how is their mental model? Like, what is the mental model they have? Either what they arrive with and how how much you've been able to actually change it to see whether or not it aligns with what you've got. Secondly, they're actually testing out whether or not that can work. Like, the initial first couple of tries.
Right? You can obviously relate to your own use of AI. Like, if you're trying a new tool, you're giving it a bit of a crack. Usually, on on more sort of, you know, lower end, you know, use cases, etcetera, you don't go for the, like, the most complex one when you're first starting. You kind of start out lower, and you're testing it out.
You're seeing how it can work. Does it can it do what it says it can on the box? Can I, you know, work out how to do this? Does it have a huge learning curve? All those things. So you're really making those judgments. And And then as they're starting to use it more, you're going into trust cultivation. So you this is, like, the more sort of, like, ongoing use. And, obviously, this is, as I not a a linear path. So this is where they're trying to build reliable patterns that not only work in the moment, but that they can maybe learn to use in different moments as they're as they're sort of maturing in their use of your AI.
Part four, what most AI products are currently missing. The the only thing you can guarantee from predictive machines, at some point, they will be wrong, just like the weatherman. Okay? And so, errors are no longer edge cases that we try to control down and minimize. They are literally fundamentally part of the core experience.
And so if you're not designing for that breaking of trust and how you're going to repair it or what that response is, from day one, you will have an entirely leaking bucket of churn of people not coming back. And trust recovery is kind of just that like, trust breaking is in the moment and what's you know, exactly what happens straight after it, to trust recovery is kind of more of a longer tail kind of, like, how do we build it back and go back into that sort of cycle around.
So you can't design for all of those things on Monday. Right? So let's focus on a couple that you can kind of focus on, and the ones that are the biggest levers, to be honest with you. So first and foremost, the early stage. So the biggest lever for adoption is investment in the those two stages. And so what this fundamentally comes down to for me is onboarding. Right? So, in the first stage, it's like really, you know, focusing on competence. What can your AI do?
What can it do is so important to put in there. Can they achieve the outcome they want? Can they, through that that early experimentation, work out roughly how it works or can see where see value in it? Can they work out how to steer it, and can they judge when it's right and wrong?
Problem is generative AI sounds just as confident when it's entirely made up as it does when it's actually right. And so you have to try and work out what can help your users work that out for themselves. Onboarding. So mental model accuracy, we wanna make sure that it aligns with what your tool can actually do.
This might have to have some collaboration with marketing to make sure they don't overpromise and under deliver in the product experience. As I said, trust deficit is something that we really have to consider when we're designing for this. So the next couple are really about I I wouldn't say they're nice to have, but, like, if you were gonna the giving people nuance control in that onboarding, right, you're actually showing them that you respect their privacy. You're actually putting, you know, money in the money box of trust so early in the experience that you're actually getting some trust equity.
And then also, the second part of the onboarding experience you might wanna consider is actually constraining the use cases or the features you give them access to initially and not really sort of giving them all the whiz bang beta, you know, more unreliable, uncertain kind of things. Because you're trying to build trust in this early stage.
Right? And so if they can actually use something reliably, you can you can then move them, you know, into those more, know, edge casey kind well, not edge casey, but you know what I mean? More sort of difficult or variable use cases. So we're looking to build trust equity. Invest in that. That would be my primary thing I would invest in tomorrow.
The second one is the trust breaking, trust recovery element. Right? As I said, prediction engines, they are guaranteed to be wrong and not in the same place either, which is really fun. So in order for that, we're trying to design for graceful failure. And so what does that really look like?
Well, first and foremost, acknowledging that something went wrong and saying sorry. We all know people who don't apologize for something they've done and how that makes us feel. Right? Similar thing for AI. Right? So think about it from that perspective. Second thing is explain what happened and why so that not only are they being able to understand what's happening now, but the why helps them understand how I might avoid this in the future.
Right? Offer them choices. So this is, again, one of those things that really is feels so doesn't seem like that big a difference, but in that moment where they're explaining what they can do, if you give them two choices or more, right, but, you know, a set of choices instead of just saying, I'm gonna fix this now, right, you're giving them back some agency.
You're giving them back a sense of control where they make the choice about what happens next. And in doing that, it means that you're actually, with something so small, actually giving them that chance of, again, building that back that equity and getting getting back on that pathway. And then lastly, depending on the the experience as well, showing that it's learning. So if if, you know, if you've only got it wrong, I can fix this for you, great.
Now I put that in my back pocket. I'm gonna try and avoid it in the future from the AI's perspective. Okay. So for me, as a design leader, I would actually be wanting to advocate for changing our design focus from being 90% core experience to 10% hap you know, error state to really looking at, like, maybe a sixty forty split.
Because if you've done all this work getting users in, convince them the capability's brilliant, and then it breaks, and there's nothing but, like, a trail of people leaving. That's that's absolutely useless. Right? So this is stopping the bleeding. Okay? So I wanna finish with some questions that I think all of us at different stages and potentially doing right now, walking around with some questions in mind.
What do I focus on with AI and design and my career and what I have to learn? How do I stay relevant? How do I make the case that design sorry, for design when AI seems to be able to do so much in minutes that used to take me days. Here's what I really believe, and this isn't just, you know, hype, you know, talk. I think design is more important in the AI era, not less. Because if every single use of AI system, not every user, every use is unique experience that emerges through use, then someone has to understand how that's evolving.
Someone has to notice when the relationship between the person and a system is quietly breaking down, and someone has to figure out how to fix it. And that someone isn't an engineer. That someone isn't a product manager. That someone is a person who's built their career on paying attention to how people respond and work with technology.
What it feels like to be human trying to get something done, to be disappointed when it goes wrong. That's you. And so I think design has a single purpose still living. It's the same one. But our craft has to adapt, and it has to do it bloody quickly. I believe we're essential, and I think it's gonna be easier for us to actually continue to make the case to learn and adapt and actually be there for each other if we, you know, come together in places like this where we can actually talk about it, where we can share what's working and what's not.
And so I wanna say thank you, and that's it. And if there's any questions, be all men's. QR code to my website, and I've looking at starting a human centered AI community, so there's a wait list there if that's something of interest as well.
Concepts & Methods
- AI Agents
- Generative AI
- Graceful Failure
- Human Computer Interaction
- Mental Model
- Onboarding
- Robodebt
- Strategic Friction
- Trust Breaking
- Trust Cultivation
- Trust Formation
- Trust Recovery
- Value Alignment
Organisations & Products
- London School of Economics
- Nielsen Norman Group
- OKPG
- University of Melbourne
For thirty years, we’ve been designing one-way USER experiences. Now we are designing two-way Human+AI experiences. We had established principles we designed 1 user experiences with – consistency, hierarchy and removing friction. We got very good at it. And now, quietly, that mastery may be the most dangerous thing we bring to AI design. Because friction, it turns out, is precisely how humans calibrate trust. That moment of slight resistance before accepting a recommendation. The pause that lets a person feel they have agency. The explanation that slows things down but makes them feel seen. We’ve been trained our entire careers to sand those moments away when the experience is based on consistency, but doing so, we’ll be building AI experiences that feel effortless, but remove users agency and but cannot be trusted.
This talk began not with research, but with regret; recognising my own work in a case study of AI harm during an ethics lecture at the London School of Economics. That discomfort became two years of asking other designers whether they recognised it too. Most did. Drawing on 240 interviews and eight frameworks built from that listening, this session invites designers to examine the most confronting possibility in their current practice.
