AI After an Apocalypse

Good afternoon. I'm Simon. This is AI after an apocalypse. Yeah. It looks looks pretty grim. And the most important word there is an. We're not talking about the apocalypse. That's the one on you're right, I guess. That's that's the end times, and you don't really need to worry about that because you'll be dead.

So that's the the general like vibe of what you go for. We've we've got the two categories of things, the things that we can worry about, and then just like, don't worry about it because it's too bad or it's not our problem. And that's just a recurring theme of how I'm gonna try to address stuff here. So that's what could go wrong is the first question.

That's an excellent question. And if you just think about you're sitting here on your little computer, you're typing, you want to talk to an agent, and then that sends a network request to some kind of cell tower that goes through a bunch of other little servers and things and leaves the country, goes under the ocean, then to a bunch of other data centers, then eventually to an agent and then you just get some kind of error come back.

And this this wasn't even worrying about all the things that were along the way that could go wrong. Was just like a local network error because my phone was offline. But if we just worry about our our little bubble here, then nothing else really matters because the the list of failures that could happen along the way, really.

You've got kind of the model could block you. It could get prohibitively expensive and you can't use it anymore. Maybe you lost your phone and you can't log in. Maybe your government requires you to submit ID to access the website and you don't want to do that because you don't know where it's going. Maybe some other server's down.

Maybe you've lost logins, been locked out, been banned from a country. Maybe you've got some weird sanctions issue going on. Maybe any number of things could happen. But none of that happens in your bubble. In your bubble, you're safe and everything's good. And so why do I want to do any of this? Why do I care? This is important to me ever since I was a little boy.

Here's a picture of me as a child but with a phone. I grew up in country Victoria and it's the the internet is bad enough out there that there's still no phone reception. My parents have satellite internet now, but if you get too far from the house, then it's like this. So you're pretty pretty constricted in where you can actually use the internet and that's that's still an issue, so that just made me get into offline stuff, but then I wanted to work with the internet, so I thought I'll move to a city and it's great here and now I take the train to work and that's me in the middle. There's a little happy kid at the back looking out the window, someone scrolling their reels, having a great time.

Still a terrible time to be a programmer on the trains in Melbourne. Just a lot of the network has dead spots. It's inconsistent and we still just don't have apps that are built for this. A lot of just streaming responses getting cut off. They break, and it's just all kind of terrible. So I I wanted a way to solve this and make this little train bound developer happy.

And, yeah, then I ran into this book. So this was the first step. You can just maybe close the laptop a second, pick up a damn book, learn something. Alright. This book's Linux pocket guide from 2004 as a first edition. I like to think that's Daniel Barrett himself on the cover as the cowboy, but this this Linux cowboy book, it's just like a dictionary of commands that existed in 2004.

So it's all the good ones. You got SSH, you got RSYNC, you've got you don't have git, maybe that's a good thing, I don't know. But it was it was a simpler time back then and Daniel Barrett takes you through it. I just thought it might be good to learn some of the commands that I was just blindly clicking accept on when the LLMs try running this stuff.

So this has been a good little reference to carry around with me and learn a little bit about some of these simpler commands. And that was the kind of underlying theme of doing this stuff was trying to keep it as simple as possible for if you're running it on your computer, just having simpler tools and that's suitable in apocalypse because you're responsible for everything.

You can't have some super complicated, fan dangled cloud managed server infra thing. Like all that infrastructure across the world and data centers and complexity and stuff is amazing and reliable and I'm shocked stuff doesn't go out more often. It's just a miracle anything works ever. How are the lights how are they not just power outages constantly? I don't know, but I don't have to worry about that because I stick with simple stuff like the Linux cowboy.

And because any anytime this stuff breaks, then you're responsible for fixing it or maintaining it, and it's kind of the selection of tools is just stuff that I'm able to change or able to fix if it breaks. Just like, yeah, good examples of having to repair your own stuff. You don't want anything too complex because even if it's great, if everything goes to shit, then that's up to you to make it great again and you don't wanna have to be the guy that people are depending on for their.

Why why would you be coding after an actual apocalypse? I don't know. Anyway, after so I just want want good, friendly, simple tools. And, yeah, that's a lot of the modern development bits kind of erred well that way because everything's just become files. We have just different names for all these things in the typical AI style, they just give different names and stuff but this all just mostly seems like files and that's great because you can you can read them, you can see what's going on, you can poke around inside and that that works well for my my simple workflow.

So that's the the underlying premise is just keep stuff simple and repairable because you're responsible now. And so for the meat of it, how to actually run a local LLM on a laptop or LOL for short. You're getting started how to run. You wanna land on the ball of your foot, and, I think that was a placeholder slide.

Forget it happened. Anyway, so we've got we've got two options. The the easiest, if you just wanted to get started today and run a model on your computer in this very room, here's some LLM GUIs. I wanted to pronounce it like Luigi, but I didn't figure out how to do that Here's three options organized in my order of familiarity from left clockwise.

You got LM Studio on one side, Unslaught Studio at the top and then a one from a Swedish guy who looks kind of like me at the bottom called Odysseus. It just came out a couple of days ago. All kind of similar things. There's like some just UI wrappers around model installers. These are good for so I've just mainly used the top two.

The other one I just looked at and thought it was funny that he's doing stuff now. It's great. But the so these just make it very easy to browse the models. You can see what to install. You can switch it. They have some just good stuff built in. I like them less when they randomly break. I've had an issue with one of them.

I'm not gonna name names. It was Ellen Studio. I just can't open the UI any I it's still I can access it over the CLI, but the like that window that you see there was a screenshot from the website, because I can't open it anymore. There's just the system tray icon. So yeah, so that's that's out. So we wanted simpler tools than this.

This is this is still like the easiest way to get started, and they work fine, and it's pretty much like a lot of this is pick a thing and use it, so you stop changing your mind and trying a new tool every month, because then you can actually just get familiar with stuff and know your tools, because you don't wanna be like reading the handbook when real apocalypse scenario happens again. So for the CLI ones, this one's black, so you know what happens in the terminal.

LamaCPP is my preferred one. They just set up a website, lama.app, for an easy install, you can just load up the website if you wanna try that out. That's like a bare bones is not the word. It does lots of it's it's huge, but it's great and works well for me. I've had a good time with it.

That can do a lot of stuff. Otherwise, if you're into Python with UV, then you can use LLM and LLM MLX, which is not to be confused with MLX LM, totally different. LLM is just like a Python wrapper. So if you wanna write little scripts and apps and things, then you can do that easily from the comfort of your own computer.

And last one I just included for so there was three things. I don't really use that, but they're all fine. Just use whatever you want. I don't care. And so now you've got this and this is you want to choose a harness or whatever you're using as well, so that also ultimately doesn't matter. Py and OpenCode, both pretty great, pretty modifiable.

You can change stuff and just make them work as you'd like. And so now you're much set up and ready to go. This is so this is the thing that'll run the you've got the thing that'll run the model, and then your harness is pointing to that instead of to the cloud because the cloud doesn't exist for us anymore. We're just all inside the bubble.

And before we get into actually loading some models and which ones to load, you've just got the thing that lets you load the models. You still gotta download the big files. And this happened to me a couple of days ago. I just had some random stuff stop working on my computer. I couldn't sync files anymore, just my bunch of random stuff started being weird and it's because your hard drive fills up with all these damn files that you're downloading to test out different models.

So it's a thing to be aware of, especially if you're trying different models and apps and things. It's getting more standard now. They like have kind of gradually coalesced to Hugging Face cache and stuff gets stored in the same folder, but otherwise you yeah. It's worth being aware if you have a paltry hard drive like this one that it can fill up pretty quickly and you won't necessarily know why. And similarly, on the web app storage note in parens there, that was just a note I left for myself, you don't read that, That if you start doing stuff like just spinning up little apps because you can do voice control now from a web browser or something and just having them download a model there, then just be respectful of other people's storage so they're not downloading stuff because that problem also happens for just random apps on my own computer. I just like have some that have little voice models or like a little just local throwaway app, then you have kind of 10 of the same thing and just end up having 400 gig of model files.

And in terms of actually choosing a model, what can you run on a laptop and and why am I interested in laptops anyway? This belonged on the why slide earlier. People are like, oh, you can just buy hardware and have it at home. Why would you buy a laptop when you can buy a dedicated giant black world chip from the leather jacket man?

But I I like the idea of running stuff on laptops because I have one. It's right over there. I carry it with me. I can use it for other stuff and now it can do LLM stuff too. This is LLM fit. There's other things like it. You can just find like online benchmarks of what people have managed to do. People have done some absurd compression that I don't entirely understand of fitting like a bigger model than RAM you have available.

So you can take this as far as you want, but for the most basic thing, you can just look up the list like this and say, here's my computer or graphics card and it'll tell you stuff that you can run. In terms of good models to run, if you want like specific answers, I'm reluctant. No. It's they're they're all fine.

There's like little quins are pretty great. There's there's some listed there, the three fives, like up to 27 b, you can run a 32 gig of RAM laptop. Just any of the m series max are pretty good at it. You can run the the really tiny ones like four b and seven b are ample for a lot of stuff and then Gemma, the Google one I've been using recently because you can feed it images. A big thing that made the local models much more appealing other than them getting better and harnesses and tools and stuff getting better was that you could use images.

And just images and audio for input was important to me because I got tired of typing. That was what started me down this whole thing anyway. And how to change models, which we didn't quite cover yet, but you can get Lama swap is a thing that runs. So you're just running like a local server with one model on it, and most of the apps just like you having one model loaded by default, but you might wanna have like a little one and a bigger one, and this is the the classic adviser loop.

We've a happy little guy at the top, and then we'll try something, and if it works, then that's a little computer dance party. If it doesn't work, then it can retry a couple of times, and if it eventually fails, then you can try the angrier, bigger model. Having multiple of these running on your laptop can be not ideal because you've either got them both loaded in RAM all the time or you got a switch and then there's like changeover and a whole bunch of other complexities, but it's stuff that you can do if you if you want to enrage your computer.

You can otherwise, for that step, you can have the the no step down there where the angry computer is. That could go out to the cloud and then just make an external request, and this still like improves my train journey thing a bit. If I just have the local model running most of the time and if it gets stuck and can't do stuff, then it'll make a call and for the like 30% of the trip that I actually have internet, then that'll succeed.

So if you're just making it less often to call externally, you obviously can't be offline all the time. I had to get the files from somewhere. But you can be offline a lot of the time and just do stuff with your happy little laptop model up there. And if I love the cloud so much, then why don't just do everything there? So you can then, if you wanna just do that further, do the whole, like, separate dev containers thing where your work lives in the cloud, and then you just check-in on that periodically. I've done that a little, but I still just get bill shock of running cloud models generally, so I'm afraid to just let them run on their own.

But and also, this just comes back to the the initial thing of you not owning the data. There's the whole, like, sovereignty, whatever the word is, for just owning your own files and computers and not just maybe the reason the frontier models are better is because we've been feeding them all our secrets and hopes and dreams for years instead of typing that into your little bubble computer.

You could have been training your local model to be getting better and know you better and function better by you've instead just been feeding the next Claude. It's okay though, because we can also have clouds at home, and then we get the same experience of the the dev, because the dev container thing I like a lot because we're doing like little portable disposable things. That's safer, you're not just letting like a model run rampant on your computer, because that's the other risk of smaller models is instead of having kind of the genius god agent cloud model run and do stuff, You just have some little lobotomized idiot.

So use it carefully, but they're generally very capable. I've just found some of the, like, the little reasoning ones seem to get stuck in little loops of self doubt, which is a little sad. They're like, no, not like this. But the cloud at home, this was just me pursuing my dreams of a more offline lifestyle, and I thought, so I can do stuff on my land, because the the issue with running an LLM on your laptop for too long if it's on your lap is it gets really hot and really loud.

On the train, people look at you funny because the fans are spinning up. You feel it more in your fingers than your legs first because you feel the heat through the keyboard, but when you've tried it a few times, it just it gets warm. The slide's unrelated. This is just about connecting computers. So I was then so where this is this is now at my house, and we've got two devices.

We've p one is these are phone one and phone 2. They're just whatever computers. You've two computers that want to talk to each other because I decided I didn't want to run the big model on the laptop that I have sitting on my legs on the couch, so I have a different computer in another room and I want to talk to it, but a lot of this still then depends on the external internet, like just sending data from one computer to the other, still just requires at least like an initial connection.

You can't often just go directly peer to peer for stuff, and so I thought what if instead of a signaling server, we just did the little key exchange thing. They're still on the same network, but you need to tell them how to talk to each other, and so I did this thing. This was a kind of separate unrelated experiment of just making a QR code. This one looked a little dense though and it was hard to scan, so I realized you can do a chain of QR codes and then just send like a series of data like this, and your phone loves it.

So then the other one just picks up a scanner and then you scan this, and you can send like multiple kilobytes of data this way, and there's a drawing of that. I just kept that in because drawing a QR code was challenging. But so the point of this is just so you can connect two computers without any upstream server at all.

Because so if my at my parents' house, if the satellites go down or they anger Elon or something and get cut off, then they have no internet. They just have a LAN and I thought, how can I make the LAN bigger because I wanna be able to go for a walk and then use my computer outside and still talk to the biggest server in the house and then went down a whole rabbit hole of how to make your land bigger?

So we've got the regular WiFi in the house on the left. That's a house viewed from top down because there's no roof, and you connect that to a halo, I think it is. There's these these separate little still WiFi, but slower and longer, and they can go up to like a kilometer. You get a few 100 meters at least with decent line of sight, and then plug the other guy into your laptop or it just has a little local network, and then you can talk from pretty far away. And that was that was about as far as I took it. I briefly dipped into looking into more radio stuff.

The LoRa, I think it's just long range, the first LoRa there, which is like internet over radio or I was looking into either using sound or just sending stuff over regular radio, but radio again in Australia for a foundation license, you gotta be licensed. It was a whole process. I didn't wanna do that, and I just yeah.

This is not strictly tied to LoRa's. I just liked the title. And that was that was about as far as I got. So I got a LLM on the laptop or at my house that I can access from my house up to some distance away, and then you can look into the, like, radio mesh network bit just because it's interesting for what happens if the regular internet gets shut down or you get cut off, which was again relevant for me because I have a Mongolian domain there.

That short domain on the left is a link to talk info if you wanna scan that or it's sci.moon slash The other QR code's a job ad. That's it. Thank you.