Our AI Hallucinated in Production: How We Fixed It With Evals

Good afternoon, everyone. Let's switch gear a little bit. I know it's been a long day. And let's talk about properties, Australian's favorite topic. My name is Yi Chen Guo. I'm from IEA Group. I lead AI initiative at IEA. We are the company behind realestate.com.au.

I want to start the talk by addressing the number one issue every property hunter has. Actually, can I get a show of hand how many of you are looking to buy a property right now? Oh, yeah. Got a few. Okay. So the number one issue you guys probably have is information overload. If you are genuinely in the market, you're probably not just looking at the five to 10 listings.

You're probably looking at dozens or even hundreds of listings, and each of those listings is packed with so many stuff. There's photos, there's videos, there's a map, there's the price, there is bedroom, bathroom, and there is a big wall of text in the description. And in the age of AI, it might not be surprising that if I tell you real estate agents are using AI to write those descriptions. And you can tell by spotting some of the American spelling in here.

So AI is already in the mix here, but it's not helping with the issue. It's actually amplifying the information overload problem. At IEA, we try to tackle this by introducing a read more button. It's a simple small button where when the user comes to the page, the text is collapsed by default, and the user gets a cleaner decluttered view of the listing.

It's only when they click read more, then they can see the full text. And can I ask the audience to take a guess? What percentage of users do you think would click read more? Do you think it's over 50%? Raise your hand if you think that's over. Okay. Do you think it's under 50%?

Oh, yeah. Cool. Most people think it's under 50%. And you are right. The data shows that only 15% of user click to read more. So that means majority of the people, they just look at the photos, the maps, and they move on. They don't even bother looking at the text.

And this is the problem on itself because there are some important information in the text that you can't get from the photos or the videos, such as the zoning information. So the challenge for us became, how do we give people the appropriate level of information but without overloading their brains?

Then we came up with the solution property highlights. Property highlights is a very quick simple AI summary. It summarizes the main feature for the property, three distinctive features, each with a title and a short description. The idea is simple. User can come onto the page.

They glance at the three highlights to get the main information, and they can move on. And on paper, it sounds very simple as well. You have a big chunk of text. You feed it into AI of your choice, and then it give you back a summary. I think people in this room might be able to hack together a demo in like ten minutes, and it will look great in your stakeholder meetings. However, when we try to ship this to production, we had more issues than we first expected.

So the first thing we need to decide was, do we self host the model or do we use a third party API? There are pros and cons to each option, and we have a lot of internal debates for each. And we actually spiked out and experimented both directions, but it came down to two main factors.

The first one was speed to market. This was one of the first gen AI project at IEA, and we want to ship it very fast and get user feedback as soon as possible so that we can get the iteration loop running. So speed to market allow us to iterate quickly, and the API allow us to do that.

The second thing was the cost. And remember, this was early twenty twenty five. It was, like, one year well, more than one year ago. And at the time, Gemini Google just released Gemini flash model, and it was way cheaper than any other model on the market. It's even comparable to self hosting option.

So the cost side of things was kind of sorted. If we were to make the same decision today, as you know, from today's earlier talk, the token price has increased so much, we might make a different decision. But at the time, we landed on Google Gemini API for speed to market and the cost.

So we quickly put the things together, and we ship it to a small percentage of our users. And then things start to happen. We had some issues reported by user that the model is confidently talking about things that are not true, such as in this example.

You might not be able to see it, but let me zoom in for you. In the highlights, it talks about a stone benchtop. But in the actual listing, the benchtop is made of lemonade. And we've seen quite a few of these kind of cases like that where the AI would make up small details of the property.

And later on, we found out that we had about five percent hallucination rate in front of real users. And by the way, disclaimer here, don't buy a property because AI summary said so. You're signing your life away to the bank, so make sure you go to see the property by your own eyes.

This might sound obvious, but you never know what people would do. So this is a risk for our customers, and this is a risk for us. And also, by the way, don't get me wrong. We didn't ship a demo to production. Before we're shipping it, we have done a lot of prompt iteration. We have got reels in the prompt.

We have done internal testings. We have done spot checks. And it looked great internally. But it's up until we ship it to thousands of users, then the small edge cases started to service. So we realized that we can't just rely on prompt engineering to guard us against hallucination.

We need a real proper evaluation system that is reusable and scalable. And we have a bunch of questions we need to answer, such as which features should be highlighted. Is it too long? Is it too short? Is the summary factual? Is the language accurate? Is it useful? And so on.

And our approach is a three layered system. So layer one is software metrics. Software metrics are deterministic metrics. They are rigid rules such as there has to be three exact three highlights, and the highlight has to be within certain correct limits.

So those things are very easy to test and very easy to verify, and they make sure there's nothing wrong structurally. And the second layer is human judges. We build an internal tool so that human judge human expert can see the description and highlights side by side, and they can label the highlights as factual or non factual.

Then we started to collect those labels. And once we have enough labels, then we scaled it with a LMS judge. So the idea is we aligned the judge LMS judge to the human judgment to make sure they are closely aligned and can make similar decisions, then we can replace the human judgment with LLM. And we were able to ship the LLM judge into our real time pipeline. So when the listing gets uploaded, they will get the the highlight will get generated, and then they will get evaluated in real time. And if there's anything wrong with the highlights, the judge will send it back with their feedback, and then it will get retried multiple times.

If the highlights fails multiple times, we will simply block the highlights so the at the end, the user never sees that problematic highlight. In practice, the the four block situation is really rare. There are very few cases where the listing itself is very confusing, such as the title says there's three bedroom, but the description says there are four bedroom.

And then, like, even a human can't tell how many bedroom there are. So those cases, we don't show the highlight at all. So we shipped the eval system, and the results, we were able to reduce hallucination from five percent to zero percent. Now I know what you're thinking.

It can't be true. 0% hallucination isn't impossible. And you're right. This is only theoretical because we don't know what we don't know. If if the hallucination get past the evaluation system, then we we never know. We have to rely on user reports to find out those issues. But what I can tell you is, so far, we have zero user reported hallucination cases since we rolled out the eval system.

So it's quite effective. And the second thing is we saw a 10% uplift in engagement. This is the business metrics we measure for the property highlights as a feature. Engagement means the users are looking at the photos, they are clicking on different things, or they are looking at the listing longer. So with a 10% uplift, that probably means the users are looking at the highlights a bit longer rather than skipping the whole thing.

And the third thing is we were able to have a clear direction for model improvement and prompt iteration. So every time we change something on the model or the prompt, we can run it quickly through our eval system so that we know whether the the performance is improving or decreasing.

That's the story of us innovating and experimenting with Gen AI in IA. IA Group is a leading global digital business in property. We are on the mission to change the way the world experience property. And it consists of a wide range of different brands as shown in the picture here.

And my name is Yi Chen again. If you have any questions, I'm more than happy to answer those after the talk. Come to find me, and you can scan the QR code to connect with me on LinkedIn. Thank you.