Everyone Wants AI Capacity Now. Few Are Actually Running It.

Foremost, I'm so excited to be here. It's my first AI engineer event, and I was by this is the kind of conference I wanna be at. Anyone that has this is their slide at the keynote, this is I'm in the right place. I just like the fact. That's just the way you present. Right? We need to talk about capacity in a roundabout way, about deploying AI, consuming AI, some of the constraints, and where I believe it's going. I guess full full disclaimers is is my thoughts, not so much the company I work at, so please don't fire me.

Firstly, let's start with the obvious. So everyone wants to run AI. Everyone now wants AI. The conversation has shifted from should we use AI to we absolutely need AI. What does that actually mean? I think that's still being worked out. Most organizations are still focused on the model. Can I get a bit of a feel for that?

Is that a yes or a no? Whether we think that's still actually the the reality, still focused on the model, or is it getting beyond that now? You know, I think I think the industry is starting to to get a bit more working up. The reality is be the model is becoming just part of that stack. Other models are catching up.

We saw earlier today the artificial analysis. Did I get that right? That was actually really cool because I validated everything I was gonna come and talk about today or parts of what I was gonna come and talk about how the AI open models are starting to catch up. We're gonna talk a little bit more into that as we go. The main specific knowledge thing is becoming key.

We're talking about that and what that actually means as we go through. But slightly self serving, but also I've got probably different optics to most people in the industry about capacity, the constraints, and what is coming, what challenges the industry is going to be facing pretty soon. Let's talk about the great convergence.

So we've all we all know what open models are. We have open weight models. Yeah? Okay. Cool. So we've all all had that. So wave one, we saw GPT. What was that? 2023. Can't remember. Is that October, November 2023? GPT three came out, everyone, you know, lost their minds and sort of that was the they called the GPT moment.

Right? That was when the wave one, that the frontier models started and took over. And wave two. So Lama, Mistral, they started to bring in some open weight stuff, stuff that we can run-in their own GPUs, run-in their own data centers, bring you know, tweak, tune, actually take control of the inferencing endpoint. Then we had a deep sick moment.

That was at the 2024, yeah, deep sick moment. That was if I was to to lean on I completely forgot that gentleman's name in the farm outfit. I loved it. It was the best presenting of Jeff. That was a power o shit moment, I think, when it comes to inferencing. So open source has really crossed this chasm.

So it used to be and we saw the this again, the the titles that we saw in the or the the sheets that we saw in the, the keynote this morning, how open models were just essentially always just three to nine months behind, and that that gap seems to be getting tighter. There's they've crossed this chasm. Most of those open models now are so good that you don't even really know the difference depending on your on your workload.

And this is where we get to the point of customers don't really care about the models anymore. Yeah. Sure. Sometimes they want the frontier model if it's a frontier service or or whatnot. But I think when it comes down to a random number, 46% of statistics are made up on the spot. Let's just say 80% of the industry don't really care.

They don't really care what if they're running the frontier model versus running a deep sea versus running a mini max or a Kimi, whatever. As long as they're getting the output, it's the framework. It's the harness. It's the the pipeline that's actually critical to this process. I think that leans into this was this morning. So I was hoping that the trend was still actually right.

OpenRoute, we actually put there the top 10 popular models today. Half of those, at least, are open models. So I think that actually talks to that point what the industry is actually, using and proving with their wallets. Again, hope there was no copyright on the the keynote, but this, you know, just validates what I was talking about earlier.

So it shows that the the intelligence index, the models, the open models are really right at almost at that top line. We had mini max three drops two days ago. I'm really interested to see what happens with that. Quite disappointed they changed their licensing model. So we've gotta send a, please, can we host your model email to them, but hopefully, we can do that.

Frontier models know everything. Right? No. What is actually valuable inside a company? So the frontier models know the generic stuff. They know what they could scrape off the Internet. They know what whatever data sort data set they could actually access, they know that and they can you can inference against that. But they don't actually have the intelligence inside the business.

They don't have stuff about the customer interactions, the the user interactions, particular product set, or the particular history of a CRM or or, you know, historical decisions or or whatever that might be. They don't know that or at least they shouldn't. You know, OpenAI, Anthropic, Google, they don't know that. Again, at least they shouldn't. DeepSeek probably does, but, you know, we won't say that out loud.

Training. So that's kinda leads to the next point of going, well, how do we actually create that knowledge, create that intelligence inside a business? And everyone talks about training. Okay? And and again, it was spoken this morning. It was kinda like my session was laid out for me in the keynote. Was great. The Sarah, I think it was no.

The the again, it was the analytics guy saying Australia's not gonna have a model. There's no point in us running and building a model, I don't think anyway. Because what actual point of values or value prop is it gonna bring? So training is so expensive. You need such large scale computers, astronomical compute. I'm not sure people kinda realize what compute is required for that, to do it properly.

But it's extremely expensive. Most organizations can't and won't do it. So then leads into the next type of, domain specific knowledge around tuning. Domain specific data. How can we actually, deliver value in our model? And you go to Hugging Face now. There's millions of models of of tuned, distilled, whatever it might be, of models trained on specific data.

We're playing with that in in our facilities as well. But it is. It's a it's a bit of a bridge. It's a bit of a if you're gonna try and find a big model, you got a big data set, and you actually wanna be able to tune, it's an expensive exercise. You need a lot of capacity, lot of compute.

And sorry. Should put my phone aside. Sales guy. But it's becoming accessible. I'm a a vibe coder. Loved it. So the other night I went home, chatty with tea. Within about three or four hours, I'd done a whole POC using Axolotl to do a fine tuning process end to end. I think fine tuning is actually gonna become very, very popular and the driving force of imprisoning in the future.

Little side note there that actual smart people in our business hate me for doing that because, you know, coming and doing a little POC doesn't actually prove, you know, that the thing can actually work at scale. So we talk about inferencing and training and tuning. How does that all come together? Training and tuning if we can look at it as a metaphor, training and tuning the on ramp, that's how you get going.

But ultimately, the highway is inferencing. That's where the the data goes. That's where the the problem's gonna be down the line. Everything we lead to, especially in the AI world, leads to inferencing, which leads to the next problem or the problem that is starting to be surfacing is a successful AI application becomes an inferencing problem. Where do I run it?

Do I have capacity to run it? Can I load balance that? Can I scale that? What about data sovereignty? Where do I actually run this model? Who's actually owning the stack where this model is running? Do I have privacy? Do I have guarantees? SLAs? Where's my cost? You know, with we heard this morning with Notion, you know, yes, model comes out. It's got three extra performance, but all of a sudden, I'm paying three extra tokens because of the outputs improved.

Right? How can I actually have clarity on what this is going to cost my business? What's my application value there? It kinda leads into the tokenomics point. Now I actually thought very long and hard about having a slide on tokenomics. I backed out of that because I'm a big chicken. But, essentially, if we look at the basics, right, we go more usage, equals more tokens equals more GPU equals more cost.

Sure. There's nuance to that. Sure. But essentially, that's where we are. And we're starting to see that actually in the industry. A few our inference providers, you know, asked to be soon, is people putting cost metrics around or you per user cost around increasing endpoints where you get a, you know, certain amount of tokens, certain amount of API calls, different different context sizes, you know, per plan that you consume because everyone spinning up their or their or whatever it might be, and they wanna have a fixed cost on that.

So that's one way of kind of of addressing that that problem. So oh, my my click texting didn't work. But leads us to the supply crunch. So we have pressure number one, reserve capacity now. These are what businesses are facing. They're gonna we know we need to actually consume.

We know we need to build something out now, so we need to reserve our capacity now. But the issue number two is going well, right now, the market leader in GPU availability is the h 200, but the Blackwell's are shipping now. B 200 is coming online. B 300 is coming online. Different precision points, different potential use cases.

Jensen's up there telling Vera Rubin. Does a great song and dance. I saw him last week at Delta Tech World. He's also on stage. So organizations are going, hey. But if I invest in this now, next year or two years down the track, I'm gonna be having a seven or ten year life cycle on some dead tech.

Then this stuff is not cheap. You go buy software b 200, b 300 servers. Anyone have any idea what that costs? It's gonna pick a b 300 server. You you know, very active audience. Let me start picking on you to make sure you're awake. I was gonna throw a number out there. Million bucks. It's a million dollars.

It's for one b 300 server running eight GPUs, and that's not even to actually operate the thing. That's not to connect it to a network. That's not actually to put any storage around it. There's no parallel file system. There's no object store. That's just $1,000,000 for one server. And how do you scale that? DeepSeek v four flash.

How many deep how many b two hundreds do you think take, to run one instance of not optimized DC v four flash? Anyone? Two days. Okay. Yes. Do you wanna come up here and do that joke? That's exactly 16 GPUs. But again, if you wanna scale that and improve the cache, then you really wanna be serving that at 60 32 GPUs.

And then if you're running that as a production endpoint, you then wanna be able to have life cycle management, so you really need two instances, two replicas. So now we're at 64 GPUs. Let's backtrack on the economics there. 64 GPUs divided by eight is eight. Eight times a million, $8,000,000, and that's not even talking about the networking to get the stuff connected.

That's not even talking about the power to run this thing. What's the power ratio of a v 200? Anyone have any idea how many kilowatts? I'm sorry. How many watts? I just gave the answer away. How many watts per per GPU? A kilowatt. Alright? So if we run 64 GPUs to serve v four flash in my production environment, I've now got 64 kilowatts of power to deliver that that that one, essentially, one model endpoint at a production level scale.

And that's not including the networking, that's not including the management cluster, that's not including my click on the ticket either. So you're starting to see how the the challenge comes around. But these are what businesses are concerned about, and this creates risk. Capacity risk, know, you how much can I actually build, consume, and run, operate, creates procurement risk?

Well, if I buy now, it's it's not gonna arrive now if you go and place an order for if you dig in your pocket somehow get $10,000,000 and go buy some GPUs, they're not gonna arrive for six months. And then you gotta spin them up, test them, light them up, put your put your fingers in your pocket, have your dead mouse in your pocket, I don't know what the superstition is, and hope to your savior that those GPUs all work.

Running a Neo Cloud, GPUs fail at a very high rate. So then you've got a supply and demand risk. So the future is optionality. So kinda got really off track there. We're really raw dogging this. This is great. So the reality one. So models are converging.

Again, I'm I'm trying I'm telling you. I'm dictating new realities here, but do we agree with this? Do we actually agree models converging? They all have their blind spots. No? We don't agree? Not particularly. No?

I mean, the the so they are good at different things at the front here, and those different things widening. So, like, the difference between Gemini's visual intelligence and Claude is wider now than it was a year ago.

Unless you mean on aggregate? I mean on aggregate. I mean on general general purpose of. So

Sure. Like, at your at the level of intelligence for most

most tasks. Tasks. Yes. Specific use cases which a lot of that well, not an AI builder. Don't really know. I build Neo Clouds. So I'm assuming you guys would know. Is, you know, if you're running your OpenCore or your or called code and you're pointing that at an endpoint, 99% of the output is gonna be very similar.

I sort of assume. Maybe I'm wrong. Should I just shut down my laptop and leave now? No. I'm joking. But they have their blind spots, and I feel again, is my my take, not recent data. I feel that the blind spots are converging. And the fast forward two years, I don't think the choice will be whether I go with OpenAI, whether I go with Anthropy, whether I go with Microsoft and announce their models of build last in the last twenty four hours.

I don't know if anyone saw that as well. I think it's gonna be the the model that people are buying. It's the ecosystem around it. It's the harnesses. It's the cost models. It's the value that's beyond the actual model. So I should change it from reality one to Dan's assumption number one. Alright? The reality two maybe I'll just change to Dan's assumption number two, is domain knowledge.

Now this actually stands firm. I actually believe that the future of AI inferencing is around domain specific knowledge. Sure. You're gonna have the focuses and the the the top model endpoints, but bringing actual knowledge inside your business, this is what we're talking to businesses about now, is they actually wanna have their own model built on their own intelligence, on all their staff's interactions. Because when they have staff come in, operate in their business and leave, when those people leave, they lose that knowledge, they lose that intelligence. They're very a lot of people are very particular about their language because they don't want they come to us and they go, they don't want AI to replace people, but they want AI to actually be able to extract the knowledge out of their people.

It gives it gives businesses a point of differentiation and widens their moat against the market. And Dan's assumption number three, capacity matters. So right now, I don't know if anyone's actually tried to go to AWS or Azure and and try and get yourself a h 100 or h 200. Good luck at that.

And if you can, a 50% chance you're gonna get approved for the quota, and then there's a 50% chance that you deploy and it'll work. And then if you do deploy, that'll be probably on The US West Coast somewhere. Unless you've got large scale capacity buying and then they may talk to you, but then they will fleece you.

Capacity matters. That problem is coming. So organizations now are starting to talk, not even GPU capacity, not not CPU capacity, not storage capacity, not capacity like that. They're talking about megawatts. They're talking about how can I secure the power to deliver the capability that I need to run my business? Australia's a little bit behind the world. Again, we only had 28,000,000 people with a certain economic value versus say The US, but that's the trend that's happening there, and we're gonna follow suit. We're already having those conversations now.

You look at all the announcements on the news, AWS invested $20,000,000,000 of infrastructure. Microsoft two weeks ago when Satya was out for the AI to a $26,000,000,000, you know, invested, finger quotes, kind of kind of satirical, in the Australian ecosystem. They're securing their power. That's what they're doing right now is they're securing their power. In The US, those hyperscalers, they're buying nuclear power plants.

They're building nuclear power plants. They're securing their power for their future. I've got one minute and fifty eight. Well, there's some reverb. The here, I'm supposed to actually talk about our platform, but I'm not gonna do that because I don't wanna turn this into a sales pitch. What I do wanna say is something that I'm actually this is actually completely off off kilter and completely raw dogging it, if that's what I say, is I jumped from my industry into to this industry about two years ago, And I see the AI industry in Australia's position in the AI industry being a unique inflection point. We've probably unless you don't actually watch media or news and whatnot, there's a lot of noise about how we're trying to do a gas tax or, you know, good, you know, this movement about trying to get revenue from our our resources.

That was sold. That ship's gone in the nineties. That was sold out a long time ago. Australia is geographically positioned prime to deliver AR factories and AR data centers throughout the world. We've got the speed of glass pretty much similar to The Middle East to Southeast Asia, East Asia, and The US. Geopolitically kind of neutral. We've got great access to water, power, and cooling water and power.

Water and power, plus we got good good fiber to all those regions. Something that's I've also been fortunate enough to have in my travels of working with Reset Data is I'm absolutely in awe at the skills and capability and talent that we have in this country. I think we actually punch above. When it comes to quantum, we're actually in the top two or three quantum builders in in the world.

AI and HPC go hand in hand with that. Now is the time for Australia to actually build their sovereign AI capability. I've got two young kids, two and four. I'm an old dad. Yes. I know. But all I want is my kids to have a future in this country to be able to create and work and not just consume American AI models.

So that was my little political spiel. Hope this wasn't recorded. Is this recorded? It's recorded. I'm in trouble. Look. That's it. Thanks for listening to my little blurb. As I said, we run an AI existing platform as part of our our value prop next to our large scale AI GPU GPU clusters. So if you wanted to consume models, tokens, you get $50 free credit if you scan there.

And if you sign up before the end of tomorrow, you get a thousand dollar a chance of winning a thousand dollars free credit. We just launched Gemma four on there. We're in the midst of trying to get MiniMax three going, but that's probably gonna be a uphill battle. Quencoder's on there. MiniMax two five. Some other things. Thank you very much.

You've been a very quiet and somber audience. Love you, Dan.

That's fantastic. Thanks, Dan.

It's alright. Oh, look. I've consumed 600,000. No. Where am I?

I've ten minutes. Yeah. That's fine. Alright.

Thanks, man. Nice to meet you. Sir? Appreciate that. Told you it was gonna be rough.

No. It's good.

I was I was also it was recorded. Like, how how how loose can I go? We had a doctor thing or from what I I don't know. They they do. So I like to Oh, woah.

The, which chords come in.