Humans in the Loop…But Where? Lessons from AI in Education, Retail, Financial Services, Utilities, Government, and Health
The Luddites Reconsidered: Lessons from the Industrial Revolution
The speaker opens with the historical story of the Luddites, reframing them not as anti-technology but as skilled workers resisting a shift in power from craftspeople to factory owners focused on profit. She notes that those who survived didn't just resist machines but identified what machines couldn't do, leaning into hand stitching and bespoke craft to retain their livelihood and social esteem. This sets up the talk's central theme: finding where humans must remain in the loop as AI reshapes work.
The Mass Production of Answers and the Value of Struggle
Drawing on Walter Benjamin's idea of lost 'essence' in mass-produced art, the speaker argues that today we face mass production of answers via AI rather than art. She warns that outsourcing problem-solving to AI robs us of the 'messy middle' and the euphoria of solving problems ourselves, citing Darwin and Einstein's habit of walking to think through problems as an example of valuable cognitive struggle.
Twenty-Five Years of Human-Centered Design Experience
The speaker reflects on her 25-year career working with diverse people—from convicted murderers to surgeons and parliamentarians—to understand how design must adapt to unique human contexts. She introduces the structure for the rest of the talk: five case studies demonstrating where AI has been incorporated and why human-in-the-loop involvement remains essential.
Case Study: Zero Gravity Bookings and Emotional Sensitivity
The speaker describes working with iFly, a zero-gravity flight experience company, to implement AI for answering customer questions. She recounts two critical examples—a woman asking about wearing a hijab and a grieving father seeking a refund after a family death—where AI responses were rerouted to humans to ensure cultural respect and emotional compassion, illustrating the need for human judgment in sensitive situations.
Case Study: Financial Forensics and the Problem of Intent
In a government financial forensics project, the speaker explains how AI was used to detect patterns and flag irregularities in financial documents, freeing specialists to focus on legal judgment calls. She emphasizes that AI struggles to interpret intent, meaning human oversight is crucial to avoid unfairly penalizing people who make honest mistakes rather than intentionally violating compliance rules.
Case Study: Many Rivers and Indigenous Cultural Nuance
The speaker shares her work with Many Rivers, a nonprofit providing microfinance to Indigenous small businesses, where AI helped staff locate documents on SharePoint and saved hours of work weekly. She highlights the challenge that many AI models are trained on North American Indigenous contexts, missing local cultural nuances and oral knowledge, reinforcing the need for human oversight to preserve accuracy and cultural respect.
Case Study: Tacit Knowledge in Complex Consulting Processes
Using the analogy of not knowing which arm you put into your shirt first, the speaker explains how tacit, habitual knowledge is hard to codify for AI. She describes a failed early attempt to automate a consulting firm's complex analysis process, which only succeeded after shadowing employees to identify four distinct types of activities and their proper sequencing—demonstrating the necessity of human-in-the-loop design for capturing unwritten expertise.
Case Study: Utilities Company and Field Experience in Infrastructure
The speaker discusses building a document comparison tool for a utilities company managing over a million power poles, where AI identifies necessary updates from legislative and documentation changes. She stresses that human field experience remains essential for judging context-specific decisions, such as how environmental factors like coastal salt air affect pole material choices.
Human Ingenuity Through History: Empathy-Driven Problem Solving
Shifting to celebrate human problem-solving, the speaker recounts historical and modern examples of empathetic ingenuity: Scottish fishing villages using unique sweater stitches to identify drowned sailors, 1960s wheelchair activists building their own curb cuts, a Seattle book exchange repurposed to feed the homeless, and prams left for Ukrainian refugee families. She emphasizes that AI cannot replicate the lived, embodied empathy behind these solutions.
Closing Reflections: Balancing AI Efficiency with Human Purpose
The speaker closes with a humorous story about a Brazilian jail using geese instead of guard dogs to prevent escapes, illustrating creative, low-tech problem solving. She concludes by cautioning against using AI merely for efficiency given its high resource costs, urging the audience to reserve AI for tasks it excels at while relying on human empathy and 'cognitive sovereignty' to solve complex problems and build a more inclusive, happier society.
So we've been here all day, 4PM, well done. It's kind of a lot. Very exciting things that we've heard so far. We're we're building this plane as we're learning to fly it. It feels a bit overwhelming. But these feelings are not new.
We're going to go back in time, a few hundred years. Where in England, and all over the country, there's highly skilled textile workers, masters of their craft, esteemed in society. Life is pretty good. And then, we've got the industrial revolution coming in.
Absolute transformation, factories being replaced with the highly skilled workers, with lower skilled workers, and bringing in the looms. So as you can imagine, people are not happy. Actually, they're really angry. They start smashing up the machines. All over England, the factories are set on fire.
Some of you are nodding and I reckon you know who I'm talking about. So this group of people, the Luddites. Now, we use that word today, Luddites, to describe people who are against technology, who resist progress, but that isn't what they were. They were challenging this shift in power from the people who knew the craft, the materials, to the factory owners who are more concerned about profit. Now, the people who survived during that time and retained their livelihood, they didn't just resist the machines.
They learned what can't the machines do. And so, they leaned into hand stitching, embroidery, bespoke pieces. And by doing this, they were able to not only retain their craft but regain some of that esteem in society. So, it's very similar to where we are today.
Today, I'm going to share stories of where I've incorporated AI and where we need to have a human in the loop, trying to think about what we have as humans and the value that we bring. So moving forward in the industrial revolution in the nineteen thirties, we had the beginning of art that began to be mass produced.
People could suddenly have a print in their house, and philosopher Walter Benjamin noticed something unusual was happening. There was something that was lost from when you reproduced the art, this essence. So I'd like you all to think about a time when you were in a museum and you encountered a piece of art and you looked at the layers, the brush strokes, this active engagement with that piece of art.
It that moment in time that you saw that piece of art, it had something. It had this capturing. Now, where we are now is not about mass production of art. We've got mass production of something else and a lot of people have referenced it here today. And that is the mass production of answers.
So many answers, answers, answers, answers, answers. We put something into Claude. That's a good answer. We cross check it with Chatty G. We've got six, seven, eight, nine, 10 answers. It's hard to know which answer is actually right. We've moved away from spending time in that problem space in the, already referenced today, messy middle, where we're breaking that problem down into chunks, deconstructing it. Darwin and Einstein had both walking paths near their house, where when they had a problem, they would walk it out.
Thinking about it, turning it over in their mind, really spending time in that space to unpack that problem. And then, when they got that answer, boom, euphoria. Such an amazing feeling when we've all solved problems. Solving a problem is not something we want to outsource to AI.
You can't get the AI to do the problem solving and retain the euphoria of when we get the answer. They're the same process. For the last twenty five years, I've spent time with all sorts of people.
Hold on. Let's go back. Oh, spoiler, spoiler, spoiler. Okay. There they are. All sorts of different people. I've been able to have the opportunity to spend a day with someone who was a convicted murderer serving sentence, a parliamentarian, surgeons, lots of different types of experiences, lots of different ways to solve the problem for them. It's very different when you're designing something for someone to use in the back of a tractor versus a busy GP's office with lots of people coming and going.
And what I've learned from that time working with different people in their backgrounds is always trying to think about how can we get technology to make things easier, but also retaining our unique context and the things that are gonna make it easy to use and better for us. Today, I'm gonna share stories, five case studies of where we've incorporated AI and share where the human in the loop is integral, that lived experience, that empathy to getting solutions for people.
Imagine you've just booked tickets to float. You're about to experience zero gravity. Now, because this is something that is unlike most something people have ever done before, this company iFly who offered this experience, people have a lot of questions calling up on the phone, DMing them through socials, sending them emails, what do I wear, what do I eat, when do you open, can I do this, I have something kind of weird with my knee?
If you can imagine it, someone's asked it. And they were spending so long trying to answer all of these questions, they wanted to bring in AI so that they could have the people on the ground have more time for their customers. So when the team started doing this and putting it together, the first stage was, oh, what do we know?
Let's codify all the things that we know and that are known and common questions, and that was fine. But once they started testing and watching the questions come in, a woman wrote in asking, can I wear my hijab? Totally reasonable question but something that they wanted to make sure was answered answered with cultural respect.
So it was rerouted to a team member. Another one that came in while they were testing it, a father had booked a special day for his family. But in that time between booking and the day, someone in his family had passed away. So he wrote in in grief to try to find out about the refund policy. Now that refund policy was already codified and you could imagine the team were kind of freaking out stopping it and rerouting it to a team member because wanting to make sure that that question was answered in a way that thought about what he must be going through.
Okay. Someone in my family has just died and now I've got to go through and like cancel stuff. Making sure that there is people available to provide that one to one empathy and compassion during a moment in time. People engage with our services in complex lives and things that are unexpected. So whenever we have situations where there's sensitive topics, it's really important to have a human in the loop.
This next case study, was for it was for a government department. I can't share exactly who it is, but it's in the area of the area of, financial forensics. And the challenge that they had was, how do we have these highly trained government specialists, but spend so much of their time combing through PDFs looking for things that aren't quite right.
Trying to see if there's money coming through the organization that could be linked to corruption or other areas that are not correct. So we looked at the end to end cycle, where do we need a human in the loop, and how can we do the things that AI is great at? It's so good at recognizing patterns. It can notify when something isn't quite right and when things don't match up based on the historical data.
And so we use the AI to do all of those rule based things so that the specialists, the people that are trained in in legislation and trained, in legal responses could have everything set up to make the, the call. Because what is really difficult for the AI to understand is intent.
So what we don't want to have happening in this situation is that everyone is painted with the same brush and they receive enforcement language. What we want, especially in government, is to try to ensure that we help people make make their help them through the compliance and then support them to be able to make the right decisions and to make sure that when they are filling in the forms that there isn't something that's just an accident.
Because the AI can't tell whether it's someone with low digital literacy that is struggling to understand what the fields mean versus someone who's actually doing the wrong thing. That's where we need a human in the loop, whenever there is a challenge around intent. If any of you have ever worked in a not for profit, you're probably familiar with doing quite a lot with not very much.
So we got brought in to work with Many Rivers. They're a not for profit that provides microfinance to indigenous, small businesses. Now, the challenge for them was to try to be able to provide their team to have more time to serve customers, and they were spending a lot of time just trying to find things on SharePoint. Highly legislated industry providing finance, lots of areas where information was and very process driven.
And so when people would start at the organization, they couldn't find things and they needed to follow the right process because it's step by step by step. The other thing that had happened when they had tried to do a a I before is that a lot of the models are trained on North American, and they're grounded on North American, indigenous context.
So there was things, cultural nuances that just weren't right. So we work with them to try to understand, again, something kind of small, trying to help them find things, but something that gave four to five hours a week back to their staff members. So I think, you know, something that's important to think about is sometimes the AI isn't doing this amazing innovation and transforming the way we've ever done our ways of working.
Sometimes it's just about trying to solve a simple problem, doing the things that it does well, which is pattern recognition, trying to find, trying to source using code words, and then that overlay of making sure that when we do have those nuances and things that are an oral language that won't live in an LLM, something that is that understanding of terms that are used by people that haven't been documented, it's important to have a human in the loop.
If I ask you which arm you put into your shirt first this morning, it might be difficult for you to tell me straight away unless you do it again. Oh, know? Do you know? Okay.
Left hander.
Left hander. Left hander. Good. I bet that that piece of information isn't written down anywhere. It's something that's just part of you. It's something you do. It's part of your lived experience. It's not an algorithm yet. This is the same problem that we encountered when working with these consultants to try to help them with their processes of a really complex application around, providing an analysis on the consulting.
So what we learned from that is we couldn't just guess what they were doing. Even when we asked them questions, there was a lot of things that were habitual, were intuitive, that they just did by doing it because they've been doing it for twenty years. They didn't even know they were doing it. And when we first started, we thought we had it right.
So we tested it and it was a disaster, total mess. We couldn't just kind of plug everything that they did into one agent and get it to do it. It didn't work. There was four different discrete types of activities that they did And it wasn't until we sat with them, watched them, shadowed them that we could understand what those different types of activities were so that we could get the agent grounded and trained on those activities, and then also the ordering so that one step was done before the other.
So again, this tacit knowledge that comes from the doing, things again that aren't written down but that those specialists know by doing. So it's important whenever you're designing these types of things where there is that tacit knowledge that isn't written down, that you have a human in the loop. This next one is for a utilities company.
So very big responsibility. I don't know if any of you have ever had the power go out in your house. Did you scream? It can be pretty scary. So their role, really important role, keeping over a million power poles active, running, keeping the, the grid going.
So then they sell the electricity to all the retailers which is where you and businesses buy their electricity from. So you can imagine with over a million power poles and all those kilometers of cable, whenever there's a change to any of the parts, change to legislation, all of the documentation, all of the different drawings that that that help the teams decide how to keep these polls going, it's a it's a lot.
It's a lot to keep up with. And so we worked with them to build out a comparison tool. So when there was a new change that it could compare documents, identify where those updates needed to be made, but then having the human in the loop so that they could apply their field experience to know whether that's the right answer or not.
So when you're talking about things like power poles and you've got different materials, it's very different for how you're going to replace that pole if it's near the seaside and you've got that salty air coming in versus somewhere that's out in regional area where it's very dry. So at this point, when you have challenges where you need to make sure that you've got that industry and that field experience, it's important to have human in the loop.
This next section is about how cool humans are. We've always been incredible problem solvers. We're going to go back in time again. A few hundred years ago, all of the fishing villages around Scotland, Ireland, UK, Wales, it was really unusual for men to ever die on land. I know that sounds grim.
They mainly died by drowning, mainly in fishing boats. And if we think about what life was like for them, big open boats, no safety equipment, no radio, no way to be able to get help when they needed it. And the challenge here was, after they drowned, they would often wash up hundreds of kilometers away from their village.
And most of the people in this area were from a Christian background, and they wanted to give them a proper burial. So, god, they're smart. So the women, would have a particular sweater stitch in the waterproof wool and that stitch related to the village that they came from.
And generation to generation, the mothers would change would would train the the daughters. It's kind of the olden days, if, you know, women got stuck doing that. And change train the daughters about that stitch of the village so that when the body washed up, they would know how to return that body back to the village. We're naturally empathetic. In the sixties in California, wheelchair users got tired of lobbying the local council to put in curb cuts, and so they put out a call for action.
People showed up. In the middle of the night, they used jackhammers, sledgehammers, and they busted down the the curbs and then relayed the ramps for wheelchairs. This middle one is near, my sister's house in Seattle. You've probably seen these around. They have It's kind of like a book exchange. Well, people in the neighborhood noticed that they weren't getting used anymore.
The same books are still sitting there with Seattle, so you can imagine they were they're pretty soggy. The neighborhood did notice that there's quite a few people living in the area that were struggling with homelessness. So they filled it with food and that's how it operates now. The last one from a few years ago, you might be familiar with this image, when families were escaping Ukraine and smashing onto trains and buses to try to get towards Poland, Women put prams at the stations, at the bus stations and the train stations, so that when they arrived, they were able to have a pram to to move their family around.
The AI doesn't know what it's like to be hungry and to be able to get access to food with dignity without having to beg. The AI doesn't know what it's like to carry a kid all day long and the weight that that puts on you. That's something that we do as humans.
We're always solving problems in inventive ways. Moving down to Brazil, they had a challenge in a jail. The prisoners kept escaping. So they tried to get guard dogs on. Guard dogs, expensive vet bills and can be bribed.
But there's an animal that hates everyone equally regardless if you're the person who feeds it, and they're really loud, and that's a goose. So they put geese all around the perimeter of the jail and these geese, they don't even sleep. It's like they it's baffling to the the people working there.
And since that time, they're frightened. None of those inmates want to come out and it's all apparently the jobs in jail, it goes like in terms of what they like to do and what they don't like to do, cooking, cleaning the toilets, feeding the geese.
I hate everyone. So AI takes a lot of electricity, a lot of power, a lot of resources, a lot of water. You know, we have a responsibility when we use it to not just make things more efficient.
I don't think that's what we should be aiming for. I think we should be aiming for using the AI for what it's really good at doing. A lot of the stuff that we just don't have the time to do. And you know what? It's kind of boring, so let it do that. And then using our own lived experience, our empathy, staying with our cognitive sovereignty, trying to solve those problems.
Because if we can do those two things, we cannot just do efficiency gains. We can have access to solve the more complex problems, ideally make it a more inclusive society, and big stretch, hopefully have a happier society. Thank you.
People
- Darwin
- Einstein
- Walter Benjamin
Technologies & Tools
- ChatGPT
- Claude
- LLM
- SharePoint
Concepts & Methods
- cognitive sovereignty
- human in the loop
- Industrial Revolution
- Luddites
- messy middle
- tacit knowledge
Organisations & Products
- iFly
- Many Rivers
As AI becomes embedded in decision-making, design, service delivery and operations, the question is no longer simply whether humans remain ’in the loop’, but which loops genuinely require human judgement, empathy and accountability. Building on ideas first explored by Walter Benjamin in The Work of Art in the Age of Mechanical Reproduction, this session explores how technology reshapes human agency, how mediation between people and intelligent systems evolves, and where the tension between automation and perception becomes most critical. What is uniquely human at work today and how do organisations ensure those capabilities are preserved where they matter most?
Drawing on practical AI case studies across education, retail, financial services, utilities, government, and health, this talk contrasts scenarios where human–AI collaboration delivers measurable value with those where misplaced automation introduces risk, bias, or diminished customer experience. These examples highlight alternative futures, emerging governance patterns, and pragmatic frameworks for deciding where humans should lead, guide or simply oversee AI systems. Attendees will leave with grounded insights, transferable lessons and tangible signals for recognising when humans are positioned in the right loops, and when they are not.
