Why Most AI De-Identification Fails in Production, And How We Built One Lawyers Actually Trust

Thank you. I

think I have the longest title of all the talks, maybe. Alright. On the surface, de identify de identifying text seems easy. You find names, other PII's, dates, addresses, numbers And at least that's what I thought when I started building this.

The hard part was not finding private information. The hard part was building a system that a lawyer would trust even after it made a small mistake. Because in that environment, one misname, they don't look at it as a bug, it's like we can't trust the product. So I'll keep coming back to this, but one of the key takeouts through this process for us has been that the downstream model, the LLN should not be treated as a privacy boundary.

It's more your application architecture. So there's two things we've built. One is SmartNode which is like a legal AI assistant, includes a notetaker. But we realized pretty quickly that one of the walls we ran into while building that was lawyers and trust.

And I'll I'll talk to that a bit more as well. So my cofounder Max, he is a family lawyer. He still practices. And the original idea for some of this stuff came from real workflow pain that he faced and he knows other lawyers faced as well. Interesting fact, not really part of this talk, but Max was my divorce lawyer, like, more than ten years ago.

We bonded so much through that trauma that we ended up working on a couple of startups including one successful exit and now this is the one we're we're working on. So anyway, the idea we had was simple. Let's work in the surface where lawyers work, which is meetings with other parties, with clients, for small to medium family law firms.

Let's record and transcribe them or if they have their own call systems or notetakers, we can suck in the transcripts and then let's help with the next steps. So these could be drafting follow-up letters, preparing file notes, affidavits, orders, supporting them with research and other things.

Almost like a junior lawyer or like a paralegal but available to them inside their workflow. So like I said, very early on, before the lawyers asked us what can AI do, it was like, what's the exposure here? And that's where SmartScrap comes in. So when lawyers heard AI, one of the first questions they asked us is like exposure.

And one of the key first factors was just like, okay, will our data be used for training a model? Could it accidentally end up servicing for someone else somewhere else? The other thing was, obviously, where does this data live? Who can access it? What crosses into the public domain, if anything? And then what's the actual risk?

Like, no law firm wants to be the one that hits the headlines because some confidential information ended up in the wrong place. So one thing we saw in existing workflows with lawyers was that they were used to doing redaction in Word documents and documents they wrote masking and things like that. But once we tried to introduce it using AI into their workflow, they didn't like it if it introduced friction or it actually broke the work.

So like I said, our target market was small to medium law firms. And the key thing here, family law, these aren't abstract documents. Most of their documents contain children's names, addresses, schools, medical information, finances, all kinds of allegations, you know, affidavits, chronologies, transcripts in addition to just like your generic PII's.

So and then there's there's additional layers of context. Sometimes just a suburb, if it's obscure enough, might be identifying. Sometimes the school name might be identifying. A file name if it leaked in might be identifying. So the question they asked was not just like, will your model be clever enough to work this out?

But it was more like, will this create a problem for us if something happens? And I think the the key difference was lawyers don't think like IT admins. They think more like risk managers but with billable hours. Now there's two ways that you can build trust in this space we found. One was contractual trust.

And what I'm talking about there is what the enterprise, you know, AI models and plans offer, like data handling, not training on your data, security controls, tenant isolation if they offer it, things like that. That matters, and I'm not discounting that at all.

It is still trust in a vendor. It's trust in a tenant. The second approach though, right, is structural trust. And this is where your architecture prevents raw information from crossing a boundary in the first place. And if your downstream model only sees tokens like person one, person two, then even a bad downstream outcome, it has less risk.

It has less identity to expose. Now, one thing I'm just going to define here when I'm saying LLM, I just mean the downstream model. And the application architecture is everything else that we control, you control around it. So things like ingestion, mapping, storage, logs, permissions, all of that. Now in my opinion, I don't think it's an either or.

I think good legal tech AI often needs both. So this niche, this talk sits kind of at the application boundary. It's not across the whole privacy tech stack. There's been a few other talks that go into all the different methods and things like that.

Nick's talk, if you were at it yesterday, talked about a lot of these things. And these are real techniques. They matter. What we're working on and what we've been trying out is something a bit narrower. It's the application boundary. So practically, like a small law firm isn't gonna jump and adopt an advanced encryption technique in a week.

So we wanted to quickly test something and that's that's how we got here. So our first hypothesis, it seemed like this is something very simple. You detect BII's, you replace them with tokens and placeholders, feel safe enough, you can then send it downstream.

And then on the way back, you just replace it. And this works very clean in a demo. But when you get to the real world and you try and do it with real transcripts with lawyers, some of the things that have we've heard are it missed the nickname. It identifies the person straight away.

You know, this organization also identifies this matter. So we decided that we needed to test more and work with lawyers directly. And also, we decided to use some synthetic data. And we also got a set of a whole bunch of meeting transcripts that the lawyers nominated for evals. So these evals were not just model scores.

We used them to look at missed entities. We looked at a whole bunch of false positives, what the mapping stability was and whether re identifying or hydrating, it still works safely after lawyers did editing. So what we realized is that this wasn't a neat NLP problem, but it was more like a systems problem.

We'll cover a few failure modes we ran into. One of them was over reduction. So at first, this sounds safe. Surely, it's better to over redact than do too little. But over redacting, it destroys meaning. If everything becomes just person, organization, person, now when you send this downstream, there's not enough meaning and structure for a downstream model to do effective work. So privacy is not just removing information.

We took the approach that it's preserving the minimum useful structure for the downstream task to still be able to work. The other failure mode was around names. And here's a good example. And you can all probably relate to this.

Often a person is referred to in multiple ways. So in our case, the same person was referred to as their full name, initial last name, and also a nickname in transcripts. So a human immediately would know, and obviously, it's the lawyer that's working on this matter, would be like, yeah, it's the same person. But a system is not going to figure that out.

So through testing and lawyers' feedback, we learned that we can make the lawyers be able to give it some interventions so that it can realize that for this matter and going forward for matters related to this, that these ways of referring to the person is the same person. Now there's another example, which is what I talked to before, which was more kind of context.

Context. And again, this is very difficult to catch. Let's say an obscure town, there's a incident with a forklift. And if you don't somehow try and de identify that, just that incident itself could reveal a lot about the matter.

So even though the system's technically done its work and removed names, it failed to actually do the de identification. So this is when we started you know, we stopped treating this just purely as a privacy and, you know, token replacement problem, but more as a trust. We needed to allow the lawyers to correct, to teach the system, and give it, like, enough context and be able to give it learnings and proof so that it could work better the next time.

So when we stopped asking, did we just mask stuff? And we asked a few more important questions. That's when we got some real traction. So first, it was really being clear on what are we detecting. Entities, spans, confidence, any context, then figuring out what we're replacing it with. So this is things like stable placeholder mappings, scope.

Then it's like, who decides this? Was this our model? Was this a pattern recognition? We use a combination of these things. Was it a custom rule the firm may have set up? Some firms have specific terminology they use and there's family law terms that you want to preserve and are not identifying. But if you replace them normally, then it loses meaning, like I said before. So and then the last one was, can we safely reverse this on the way back so that it works across editing, exports, and any other downstream AI outputs?

What we ended up doing was we built and I'll show you a quick video demo. We built a UI that the lawyers could use to work on transcripts and the de identification. But then we also built an API that the actual legal assistant product could use. Now we come to the actual placeholders because eventually it actually comes down to text that we figure out what we're de identifying and what we use as placeholders.

We treated placeholders not just as pure string replacements but like contracts. We used new neutral tokens like person one and not like client father or major bank because that's still revealing something. And this was deliberate because we neutral neutral tokens were easier for lawyers to inspect and for auditing and also safer.

The trade off is that they do preserve a little bit less semantic context. So the surrounding system has to carry the stability, the mapping, the re identification guarantees. So that's how we treated placeholders. That's just an example on one sentence. You'll see more in the video later.

Now we were not storing just raw text and scrub text. Once you make the identification reversible, our data model changed. We're storing entities, spans, mappings, versions, overrides, audit history. And we also have to handle the normal workflow that a lawyer has. What happens if they edit a whole sentence that had a placeholder? What happens if they copy and paste some text from somewhere else? What if the summary that the downstream model returned missed some placeholders? What if, like you saw in the example before, there was multiple names for the same person?

And these weren't edge cases. They happened often enough that we had to cater for them. So in our world, the mapping was not just metadata. It was actually the product's memory. The next lesson that we learned was that manual review wasn't just a fallback if the system didn't work.

It was part of the core workflow, core architecture. Users needed to see what what was missed. They needed to mark something as safe. They needed to be able to add custom family law terms. And they needed to decide what's sensitive in their matter. You know. Because once users can override the system, we needed them to be able to create those as rules that they can use later. So the other thing was the system needed to know and remember the lawyer's judgment.

If it didn't, it almost felt like it was arguing with them. And that destroyed trust. So the architecture, like I said, the privacy boundary, the LLM shouldn't be it. It's the application architecture. And what that looks like so you have your raw text which is coming from either audio transcriptions or from some other service.

We have a detection engine that runs patterns and models and rules. And then we as I said, we have this review UI that lawyers can use. And then we end up with a de identified payload. Now that's the point at which we say, right, that can cross your application boundary. And any other applications like SmartNode or whatever we've built can use this as a secure API.

And then we get re identification, which produces a final output using a mapping store that we've saved. So while prototyping, we had to kind of work with lawyers to both in both ways, like early on testing with actual transcript using the UI, but then when they're doing recording and meetings to be for them to be able to see the transcript and see what was de identified and sent.

So this is just a demo of the actual UI. If I can quickly play this. I'm not sure if that's gonna be big enough, but it'll give you a a little bit of an idea. So on the left, you have the raw. On the right, you have tokens and replacements based on the engine and the model. But then there might be instances where something's missed.

So like in this case, think Emirates Airlines was missed, and they can manually say that's an organization and replace that. But then the system also lets you look at the entirety of what kinds of things were replaced and you can look at them and and audit them.

But it saves all the version history and lets you save them as rules and things and add your own glossary of terms and things like that. So there are some key things we stopped doing. We stopped assuming more reduction is always better.

We stopped hiding uncertainty from the user. We stopped treating re identification as just pure string replacement. We stopped trusting one shot transformations without showing an audit trail. And we stopped thinking of this as just like a pure NLP feature. So what I guess I'm wanting you guys to take away is that de identification is a systems engineering problem.

And for us to do it effectively, it touched the UI, the data model, logging, and it was ultimately all about trust. If I had to make a checklist for you guys, if you did work in this kind of niche space, it's designed for reversibility from day one, make mappings first class, assume users will edit everything, do not rely on just the LLM to enforce privacy, and realize that there's gonna be mistakes.

And the question is not whether the user can see the mistake, it's whether they can fix it. So really, the LLM is not your privacy boundary when you do things like this, it's your application architecture. Thank you. You can get in touch with me. On socials, I've also put the slides up. I'm not sure if the link is showing there but it's it's it's there somewhere.

Oh, it doesn't show. Thank you.