LLMs in government service design

Yes. I am Harry. I work for a company called MyLot. And MyLot's a very young startup, so I'm gonna be sharing a lot about some of the products that we create for urban planners or statutory planners. But keep in mind that they are young, they're experimental, but they are in market. So I'm hoping that this talk gives you a bit of a kind of realistic view.

I'm gonna try and follow this vibe of just telling it how it is and this is how this thing works for us in the statutory planning industry. So to make that distinction, I'm not really talking about how we use LLMs to do design or to help us code, even though we do use them that way as well.

It's more how we use them in a product to fulfill that product service offering. But to broaden kind of the scope of that first, I want to contextualize it within the scope of other government services. So when I say government services, I'm use the very broad definition of a service as something that helps us to do something not very helpful. But government services, mean things like applying for a parking permit, getting your driver's license, registering your pet, and all sorts of other things that you will go to local, state, or federal government to to do. And by self-service on this slide, so often when we're designing these services as government, a very common objective is to increase self-service or allow people to do the service themselves.

So that might be you have a service that only exists as a paper form and you're gonna put that on a website. Or you have this, you know, common information need and everyone's calling council or whoever to understand to to serve those information needs. So you'd be putting that information on the website so you get less of those calls.

So yeah. So I'll go into a bit more soon about how statutory planning is very hard to make self-service. But to start with, so a good example of how self-service is done is pretty much just with good content design. I find that in government service design, the outcome is more often than not a solution that's based in information architecture and content design.

That's often where the challenge lies and that's where could be solved. And this is a very suitable solution. It doesn't have to be fancy by any means. So you, for example, in this one on screen, you go to nsw.gov.au because you are pissed off that, I don't know, delivery drivers are always riding down the footpath and you wanna figure out if that's legal or not, you go to a page like this.

And rather than call counsel and complain about that, you can understand, oh, yeah. They actually can't do that or they can do that and and so on. Actually, don't know if they can, but there we go. And there's there are other kind of service examples where content design is more than sufficient. So if you're trying to find out your bin day rather than call council, if they have a well designed page in their website, that can be very much facilitated there.

And to extend that a bit further into a slightly more complicated case, so let's say you might in some of these services just need a very short interaction. So you're trying to figure out what local government area you're in, you've just moved house, and you might just have a very simple tool. Like, I can enter an address and it'll tell me where what council that address is part of.

But it's still a lot of hard work. So you can imagine, although this has ended in something simple or something that feels simple, you can imagine all the kind of GIS stuff that has had to go into this to be able to map the whole state of Victoria or wherever you are to make sure that that is accurately represented.

And then we have this kind of other category of services where things like the ones I listed before, like parking permits, driver's licenses and all that kind of stuff, would fit in, in my eyes. So we've got to gather quite a lot of information from the user in order to able to fulfill that service. So this is an example from City of Sydney that I worked on a while ago now, where we have kind of this concept of a landing page or this is actually from gov UK now.

Or what they call a star page. So something that is introduced to the service and helps people understand that they're eligible. They're then asked a series of hopefully simple questions. And then at the end, either no, you're not eligible or you are eligible, this is the outcome of of that service. And this pattern is really common and really because it's really effective.

So and when I say this pattern, mean things like being able to ask just for one thing at a time, which enables you to order questions so you don't waste people's time. So in this for a parking permit, might start with what's your address? And then that'll not waste someone's time if they're suddenly applying for this parking permit with the wrong council.

And that enables you, when we're only asking one question, to design those questions really well. So you have a whole screen to be able to do that. And when I say design a good question, I mean, for example, a closed question is often easier to answer than an open question. So asking them something that can be framed as like yes or no, not something broader than that.

As well as a bunch of other principles. So it's really important in this pattern to, when you can, be able to have people be able to respond with I don't know. Whenever you can. And there's a bunch of other principles that go into this. So kind of the barriers to self-service are really varied and they can be very difficult to overcome.

So this is a good example from snap send solve, which I'm sure many people have used, where there's this big issue where I see a damaged footpath or some graffiti, but I don't really know who is responsible for cleaning up that graffiti or fixing that footpath. So this is where a private industry provider has come in and said, well, just upload the photo of the footpath and tell us where it is and we'll tell you who is managing that footpath, which can be a very complex question.

It could be that Transport for New South Wales owns just that bit of the road. Maybe no one owns that bit of the road, which I know causes a lot of pain in certain intersections and and so on. But these these guys can just work it out for you. So when the barriers are too great, this this is where the service is no longer self-service.

It results in someone saying, I just need to talk to someone, or they would just do it anyway. So in statutory planning, I'll go into, that has really massive environmental consequences. But for parking permits, that might just mean that someone is going through the pain of moving their car every two hours so they don't get a ticket, or they're parking it down, you know, three blocks where the only one twenty four hour free space is, which is really for visitors, but they're parking there because they couldn't figure out how to get that permit.

But that's kinda how I wanna frame what we're getting into with statutory planning that it's often, you know, a big objective to make a service self-service or do it yourself. But there are some services who really resist that a lot, and statutory planning is one of those. So what is statutory planning? And I I I'll I'll say this like this.

It's not not become self-service because no one's tried. There's really, like, this massive graveyard of of products and trials to get this thing to be better. And it hasn't only been in service. It's either there's been a lot of legislative changes regularly and that's going on right now with the housing crisis and everything. But statutory planning refers to the legal framework and processes that govern how land is used and developed.

It's really kind of like the the unglamorous side of urban planning. So we think of urban planning and we think of at least I do. I think of all the YouTube videos I watch at night just about, you know, preferencing bike paths and stuff. That isn't really what a statutory planner does. The statutory planner is the person at council or the you know, they exist in other orgs as well who is receiving an application.

Like, say, someone said, I need to prune this tree, and they're assessing if they can do that or not against legislation. And that is, yeah, that is distinct from strategic planning, which would be kind of what I said said is kind of the more glamorous, interesting side of things. Statutory planners are not popular. They really you imagine they have to deal with telling people what they can't do all day.

It's not a very fun profession to be in. They face a lot of animosity. And yeah. So interactions with them may start with a question like, can I build I build a pool in my backyard? Is my neighbor allowed to build that pool in their backyard? Because I didn't hear about it. Like, why didn't I get feedback on that?

It's gonna do something to my house. Or I just need to convert my garage into a living space for my elderly grandmother. How can I do that so I don't have to deal with so much bureaucracy? Because it might be urgent. And many of them are quite urgent. So I might be looking to purchase a property. I need to figure out if I can build a house there.

So I need the answer now. So I'm gonna call counsel immediately. I'm not gonna deal with the, you know, huge amount of info and research tasks that I'd find online to help me do that. So there are a few reasons why statutory planning resists self-service quite a bit. And the key one is that the rules of statutory planning are numerous, contextual, and full of jargon or legalese.

So when I say the rules, I am referring to legislation. So a lot of this stuff is stuff you find in Acts and things like that. But they're often just referred to as planning provisions as well, which are not laws per se, but they are given authority through Acts and things like that. And these are really just impossible to navigate for a layperson.

We we often say that planning is an industry where you need a higher help in order to interact with it. You can't do it yourself by any means. So people rely on planners if they can afford to hire a consultant, or they just rely on their builder who may not know these rules at all anyway and they give them the wrong advice.

So when we see solutions in statutory planning in the content design space, we see solutions like this, which, you know, are pretty cool. They are at least graphical in this case. But they're very limited in s scopes, so they might just deal with very typical cases. So you in this one from Burundara Council, you can click on these various things and it'll say, yeah, you can build a shed without a planning permit, but it needs to follow these conditions or you can do this.

But what we often find where these things fall down is someone believes rightly believes that their case is unique and they can't really proceed beyond this. We often say to our customers that it's it's like a you can't even ask your neighbor about how you might build your shed because it's gonna be a completely different rule book to how you did it.

You might have a slope on your land. There might be an easement, like a shared driveway or an electrical box on your land that changes a lot of things, a bunch of other stuff, which is kind of the whole contextual side of this. So on that, yeah, so they're they're very contextual. That it makes solutions very difficult to scale.

So each state's planning system in Australia is completely different. Each council has their own local rules, and those rules change regularly and for good reason. So for example, I've mentioned the housing crisis a couple of times. Governments are constantly seeking to figure out how to make that how to build houses faster. They have to look to planning.

I don't think there are blames in planning, but many many people do. And that causes all these rules to change. So if we were to design a solution, which in government is often referred to as rules as code, where we might seek to codify all this legislation and say what have I got on screen? Like this one which is about a setback rule.

Well, this applies to your site if, for example, your development type is dwelling, you are on a irregular lot or a corner lot and so on. And theoretically, you could do that. You could go in and codify all of these rules, and many products have done that. What we find with those products is they're very short lived and that's where that big graveyard I talked about comes from.

And that's mainly because the rules change all the time. They might only be able to launch in one council area, so it doesn't have much scalability. It's gonna just die when that project team moves on and so on. And finally, the the required data isn't always available. So and what I mean by that is not necessarily why we don't have that mapping layer to be able to build build something like this, But it might be that it's just we can't translate that into an easy to answer question.

So it might be, hey, I'm looking to cut down my tree. The kind of legislation says, well, you can cut down the tree if it's kind of essential to protecting that property from bushfire. But I can't really answer that. I need a professional to be able to answer that for me. So kind of a a certain or comprehensive answer is almost almost impossible.

Another thing that goes on is in planning, people just don't understand what government actually needs them to do. So if we take the parking permit example again, I kinda like this kinda three point model where we say, well, what I wanna do is park my car on the street. What I think government needs me to do is get a parking permit. And what government actually needs me to do is get a residential parking permit. Now that kind of gulf of understanding between the second and third one is not great at all.

It's pretty easy to rectify just with a good content design or a quick question. But in planning, it's a very different case. So in planning, what I wanna do might be I wanna convert my garage into a granny flat because my mother's getting old and I want her to to live there. I might think government well, I I have common attitude to say, I think government needs to do nothing.

It's my property. Another kind of common one is they're only really concerned with the building side of things, not kind of the affectations that surround the site. And then what government actually needs to do is they've gotta get all these different permits, which is a really hard thing to do. So for me, like, this model kind of sums up the planning problem quite well because over on the left, we've got kind of huge breadth and diversity in what people wanna do, which makes the service really difficult to design from a just using content design and methods like that. In the middle, there's kind of huge misunderstanding about what I think government needs me to do.

Planning has kind of poor awareness, so people get very confused when interacting with it. And then over on the right, there is kind of massive complexity and expense in what government actually needs me to do to the point where it's kind of often just seen as ridiculous. So for example, the average planning permit in New South Wales in 2023 took a hundred and two days to process.

That includes things like just, you know, building a shed in your backyard. And I think that's gone up since. I think in Victoria, it's actually closer to two hundred days. So understandably, in that case, people would rather just avoid that whole thing or ignore that rule and just build the shed and not talk to council at all.

It is effectively essential to get professional help to navigate the procedures of getting a planning permit. I need to hire a consultant to navigate this stuff, which is expensive as well. Or it's actually essential. So for example, if I'm in a bushfire protection zone, bushfire management overlay, I will need to get a certified bushfire expert to tell me that you know, this is okay to do and whatnot.

And after all that, the rules may also just simply say you can't do that. In your current form, you're gonna have to change the design or not do it at all or move house or something else. So to to kind of really double down on the complexity here, to make this self-service in its current form, what would I need to do to figure out what government needs from me?

Well, in Victoria, I'd have to go to a website called VicPlan or another website that tells me my property's zone, size, overlays, and other mapping layers that apply to it so that when I say zone, I mean something like a neighborhood residential zone. Overlays could be something like you're inland that's subject to inundation or are prone to bushfire or there's a vegetation protection overlay there.

I'd have to check my title and check that any other restrictions or rights that apply to the property are gonna affect that stuff. It's gonna trump what's going on there. I'd also need to accurately classify what I'm proposing to do. So this goes back to the jargon I was talking about before.

But in Victoria, if I'm trying to do a granny flat, that's actually called a small second dwelling. It's only called that if it's under 60 square meters. Otherwise, it's something else entirely. And I really need to know that specific term in order to find it in these rules and determine what applies to it. And searching through all of that, obviously, like a very tricky thing to do. It's going through not just one legal document, but many, many of them because they're not all in one place.

And this has a ton of consequences, of course. I talked about people are gonna find a way to do something whether they could figure it out or not. So if someone's just gonna build that shed, they're just gonna cut down that tree, which is part of a, you know, essential wildlife corridor, it's gonna do a lot of damage.

And then from the planner side, they're kinda getting inundated with, requests for further information so people submit applications which are not complete and so on. I won't go into this, but this is kind of the content design that you would see to figure out if you need a permit or not in Queensland. So how have LLMs helped us do this?

So I'm gonna speed up a bit. But what our product does is it effectively is a tool that helps people work out what government needs them to do when they're doing something. So it's kind of following a very similar pattern to the one I showed before for parking permits and whatnot. But we have a start page, as Gubby Kay would call it, over on the left.

They go on the council's website. You then answer a series of questions and you get a result at the end. So we start, for example, start with what's your address and then what are you planning to do and then that goes on and on. But what is kind of going on here is LLM, a large language model, is driving a lot of that experience.

So how do we use LLMs in this experience? So it doesn't really look like a chatbot by any means. You kind of it's it's a bit hidden, at least in the UI. So the first thing it allows us to do is allow us to start broadly. I mentioned before that people often feel like their case is unique, often it is unique.

So LLMs allow us to at least do natural language processing a lot faster. It's a lot more it's a lower barrier to entry to get this working. So people can really answer this and articulate what they're doing in any way they want. So they can just say, I'm building a carport. They don't have to know what that specific planning term is.

They can just enter it and the tool will do the rest. The other thing LLMs do, they don't do this actually that well, but I'll go into that, yeah, in a in a bit, but they also allow us to formulate the next best question. A traditional approach when you're building these big forms in government service design is what I think it was Caroline Jarrett coined as a question protocol where you would do a ton of interviews.

Let's say you're trying to digitize a paper form in government And you would be asking these people in interviews why that question's asked or why you need that piece of information, who uses it in your organization, what do they actually use it for. And you do that to distill everything down into a set of questions and you figure it out from there.

But if you did this for planning, you would get a very, very long list. There would be, you know, there are also a very contextual list. It depends what the person's doing. So over here yeah. Given the context of you are in this zone and whatnot, what we use the LLM for is say, hey, given this context, look at these parts of the planning scheme.

This look at this part of the legislation and figure out what info you need to kind of satisfy their query and formulate that as a question, and we present that to the user. The other thing it does is it helps us classify proposals. So let's say someone responded with, I wanna build a shed in my backyard for my gardening equipment.

What LLMs are very good this is something that we found out they're very good at, is that we get in that list of terms, and that's not a short list of terms for the Planning to Massive Domain. I think in New South Wales, land use or development type is shown as I think there's 365 different terms. And, obviously, it doesn't cover everything.

People will fit between these things. So if someone says, I wanna build a shed in my backyard for my garden equipment, the other might say, I think it's a store. It's a rural store. Or maybe it's just not anything. It's just something that's ancillary to the dwelling that exists there. But they're gonna ask a few more questions.

Then the person might review, oh, actually, it's garden equipment that I sell online. So then it's a home based business. We find this is something that happened that works a lot better than someone who is doing this a human that's doing this. It's not it's a task that's better suited to LLMs. Because statutory planners might often kind of work in a very specific area of development. They might be unfamiliar with this stuff.

They don't come across that well that often. There are also some terms that are just very specific. So let's say someone was trying to sell fruits out of their farm shed on the side of the road. A lot of planners might have not come across that before. They might say, oh, that's a store. Like, I'll just categorize it as that.

In reality, it's what we call a primary produce sales, which has very different rules that apply to it. And we do this with LLMs in in AdMiler until what we say is sufficiency is reached. So we keep asking those questions. We have another large language model which is there to just summarize the proposal and get it succinct to control that context.

And we live around it again and again until that sufficiency is reached. Again, if I'm honest, we're not very good at judging when sufficiency is reached. Often it's quite a kind of unsophisticated way. We just say we just ask eight questions and we get there or just wait for the LLM that's figuring out how to classify a proposal. Wait till they're more certain, like they only have one possible answer, and then show them the result.

And then, of course, the kind of the known LLM benefit is we use that to generate the result of our service. So at the end, if someone gets a report, we refer to it as a report like this where, yeah, your proposal requires a planning permit. And a lot goes into this to kind of get it to work and make it scannable.

You can imagine talking to ChatGPT. You don't necessarily see something like this that is, I wouldn't say succinct, but at least kind of formatted in a way that's more designed for the web. So on that, so to kinda wrap this up, like, are we doing because we've used LLMs, what have we had to do to get the service to actually work?

So I'm trying to frame it this way at the end just to kinda distance it from the hype. We're not saying that LLMs have solved statutory planning by any means. It's not gonna solve very much at all really on its own. But it has allowed us to make some headway. So as I said, out of the box, LLMs do not create good content for the web at all.

So that report I showed earlier is actually made up of a bunch of other things. So we have a static heading at the top. The body of it, we prompt this large language model that generates this with a markdown template so it can kind of insert things in a more predictable way that we think is gonna be more scannable for someone to digest.

Another part of it might come from a different large language model, which has a kind of more constrained task and can articulate a very different concept to the person. And then we well, I won't bring this up as its own slide, but we have a very disclaimer heavy experience as well. So the disclaimer's all throughout, they kind of are inserted for various reasons and also just to say an LLM generated this.

On that as well, LLMs are very prone to making stuff up. What we've heard of was hallucination. I wouldn't even say making stuff up. They're just very prone to wanting to help you. So they will generate tokens or or text as much as you can. Obviously, there's like a financial kind of incentive for the AI company for that to happen as well, which maybe that's why it's happening. They have a if that kind of pay by token business model continues. But why this happens generally is because there is either too much context with the LLM or just missing context.

So a good example in planning is it will say, well, if there are any conditions in this part of the legislation, then disregard this other thing. But then there might be nothing in that part of the legislation. So the LLM will look there, they'll see there's nothing there, and then they'll make something up, which is a big hurdle we had to overcome.

So what we do is effectively it's an a lot of people, when I talk to people about our products, they say, oh, how did you train the LLM to do this? We're not really we don't have our own large language model. We use providers just like anyone else does, like OpenAI and so on. But we have orchestrated the different instances of those large language models in a specific way so their context is effectively controlled.

Kind of the whole role of the prompt engineer is an art of controlling context. So in this diagram we have, for example, an LLM that's only responsible for summarizing the proposal and getting it really succinct. We have a bunch of what we call assessors who are only focused on the part of legislation to do with parking or to do with signage or to do with whatever it may be.

And they all work together to kind of compile either next questions to ask or to generate a result. And that's also very handy for other reasons. So that creates what we refer to as a trail of reasoning and that helps us QA this whole process. So if we just kinda generated, here's your big result at the end, it'd be very hard to kinda understand how the LLM got there. So we have a bunch of things which we can kinda backtrack and understand that and we QA that a lot with our own in house planners and planners at council.

Yeah. And because we use LLMs, we review every single result. So it's not a set and forget thing by any means. I think it's actually got much more intensive QA than a typical approach with a more traditional method. This is kind of just a review form we give the counselors we work with. And because we use LMS, they respond to anything, so we had to put in guardrails.

This example isn't that funny. It's just, someone's doing subdivision. We can't deal with subdivision. It's too complicated, so we kinda stop them, but you get all sorts of weird responses to those open ended, questions at the start. I wanted to end on a bit about kind of this argument of chatbots versus graphical user interfaces. So as you may have noticed, like, oh, this doesn't really look like a chatbot. This is kind of common discourse on LinkedIn especially where, like, why aren't people who use large anchor titles designing anything more interesting than a chatbot?

I don't really have an answer to this. I would just say that we've decided not to for a very conscious reason. We think that, especially when we're asking complex questions like, can you measure this tree for me in this big screenshot, we really need to design a UI that's more than just an open question and help and guide people through that.

We've had other variations of our tool which are more chatbot like. This was a very early one where we have kind of this ongoing chat on the left and then the part on the right would kind of update and iterate as people gave more information. That was really just limited by its speed. There's not enough feedback to the user once that information comes in.

And our original prototype was an app on Telegram, which was very much a chatbot. But, yeah, we've ended here. We don't think it's right necessarily. We haven't really sold the idea of how do you do something like this but still have the affordance that someone can respond however they want and give context to the LLM in any way they want, that kind of has been lost in this design.

But we find that we're getting a better kind of success rate and satisfaction rate with something like this. So, yeah, so LLMs don't solve statutory planning. They they as I said before, they've helped us make some headway. I would say just they've enabled new capability that wasn't there before, so things like being able to classify things in very large taxonomies, which are very common in planning, being able to start with an open question and be able to interpret what people are saying there.

And also, kinda most importantly, with that kind of graveyard of products I said before, being able to actually maintain the product because we're just pointing LLM at legislation. We're not kinda codifying all these rules. When the legislation changes, it's just new context for the LLM to interpret. It's not kind of this big undertaking that we've gotta do every, you know, sometimes two weeks.

And that's it. Thanks so much.