Shawn Wang coined the term "AI Engineer" with his seminal 2023 essay ["The Rise of the AI Engineer."](https://www.latent.space/p/ai-engineer)

Few people are better placed to help us keep pace with a field that seems to shift sometimes even by the hour and it is a privilege to have him open the conference.

An analysis of the leading AI models and the real trade-offs between them across intelligence, speed, price, token usage, and beyond, grounded in Artificial Analysis' independent benchmarking. Includes a forward-looking read on the trends shaping where AI is heading.

Token pricing is a noisy headline. What matters in production is what you pay per task once outputs get longer, retries creep in, and “good enough” models get deprecated.

This talk breaks down the current dynamics of the frontier model market from the perspective of Notion AI product team on embarking this with Notion custom agents model, and shares a practical playbook for keeping leverage: stay genuinely multi-provider, invest in product value that is hard to undercut, and use open weight models as an escape hatch for everyday workloads that do not need frontier reasoning.

Software development as we knew it is dead. The old world of hand-crafting every line, endless hiring cycles, and fragile vertical stacks has been overtaken by AI-driven software factories that turn simple prompts into fully functional software through automated iteration. This keynote traces the journey from the Ralph Wiggum Loop to Loom and beyond, exploring what it means when code becomes a commodity and engineering becomes orchestration.

Full-duplex voice agents punish the choices text-only agents forgive. A 50ms GC pause is a perceptible glitch. A blocked event loop is a barged-in user talking over your model. Google’s Python ADK is a beautiful kit for building agents — until audio frames arrive every 20 milliseconds and refuse to wait.

This talk walks through gemini-rs, an open-source Rust SDK that rebuilds Google’s Agent Development Kit and the Gemini Multimodal Live wire protocol from scratch, in three layered crates: a zero-copy WebSocket transport, an agent runtime with phase machines and typed prefix-scoped state, and a fluent builder where a production voice agent fits in twenty lines of declarative Rust.

The architectural keystone is a three-lane processor: a sync fast lane for audio and VAD under one millisecond, an async control lane for tool dispatch and phase transitions, and a debounced telemetry lane built on atomics. We’ll cover what broke in the naive translation, why Vertex AI sends binary frames where Google AI sends text, how to compose tool-using agents without rebuilding LangChain, and where the Python ADK still belongs in the stack.

The smartest model doesn't always win.

In agentic coding loops, a model that is 10x faster but only marginally competent can often fail its way to success before a frontier model finishes reasoning.

AJ Fisher breaks down the maths behind this counterintuitive result using diffusion models like Inception Labs' Mercury 2. Unlike autoregressive models that generate tokens sequentially, diffusion models refine outputs in parallel, removing a serial bottleneck that slows iterative agent loops.

If each attempt improves a solution by even 20%, dozens of iterations per minute quickly compound into faster convergence than slow, high-quality reasoning.

With live code examples and a bit of napkin maths, this talk shows why loop velocity is becoming the dominant factor in AI-assisted engineering, and why verification, not model intelligence, will become the real bottleneck.

The key question isn't "how smart is your model?" It's "how fast is your loop?"

Responsible AI isn’t culturally neutral. American AI development embeds distinctly American values — individual liberty, technological solutionism, and winner-takes-all competition. But what happens when these values clash with Australian cultural principles that prioritise collective welfare, egalitarianism, and the “fair go”?

Current AI governance frameworks — largely imported from Silicon Valley — often perpetuate values misaligned with Australian regulatory expectations and social norms. The Mathpath and recovering American Aubrey Blanche draws on principles of mateship, pragmatic skepticism, and community-oriented thinking, in this presentation that introduces a distinctly Australian responsibility framework for AI implementation. Rather than treating AI risks as individual consumer choices or market failures, this framework positions AI governance as a collective responsibility — where technology serves the common good, ensures equitable access, and earns trust through demonstrated fairness rather than assumed benevolence.

The 2025–2026 wave of "self-evolving" agents — prompt-tuning loops, memory accumulation, agent swarms, GEPA, ReasoningBank — share a structure that is sometimes lost in the jargon: every one of them is hill-climbing on a judge. The judge is the fitness function. When it's sharp, the agent compounds. When it's vague, the loop drifts confidently in the wrong direction.

This talk argues that rubrics, not prompts or scaffolds, are the load-bearing infrastructure of agent improvement. We'll walk through three concrete failures from recent work: prompt optimizers that regressed without rollback (OpenAI), memory systems that hurt performance as they grew (ReasoningBank), and 18 months of capability gains that delivered almost no reliability gain (Princeton). All three share a root cause: the rubric was the bottleneck, and nobody was looking at it.

Then we'll build one. Five principles for a rubric that can actually drive evolution — stack deterministic before semantic, score failures explicitly, measure beyond accuracy, version the rubric itself, keep it cheap. You'll leave with a checklist you can apply to your next agent before you ship a single optimization loop.

Most AI agents are reactive chatbots—great for one-off queries, but they reset, forget, and lack initiative, failing in real-world use like personal assistants or autonomous workflows.

This talk dives into the battle-tested architecture of Claw-family agents (OpenClaw and lightweight forks like NanoClaw, PicoClaw, TinyClaw, IronClaw, ZeroClaw), which power persistent, proactive systems that run 24/7 on your devices. Drawing from real implementations, we'll unpack core patterns:

Hub-and-Spoke Separation: A stateless gateway routes inputs (messages, heartbeats, cronjobs, hooks, webhooks) while adapters normalize diverse channels (WhatsApp, Discord) and enforce typed protocols/security handshakes. Ephemeral vs. Persistent State: Transient context (system prompts, recent interactions) stays token-efficient; durable memory (append-only logs + curated facts) uses hybrid retrieval (semantic + keyword) with flush safeguards to survive compaction/restarts. Runtime Loop & Extensibility: RPC streaming for task queuing/execution; plugin discovery (tools, providers, memories) and Markdown-based skills (SOPs) enable hot-loading without recompiles, plus multi-agent delegation for collaboration. Security Boundaries: Defense-in-depth with network isolation, sandboxed sessions (e.g., Docker for untrusted chats), identity pairing, and safeguards against injection/poisoning. Proactivity and Deployment: Inputs trigger autonomous actions; architectures span local native, VPS/Docker, or cloud for low-resource edge devices. Whether forking OpenClaw or building similar agents, this playbook delivers insights on separation of concerns, model-agnostic brains, and tradeoffs for scalable, secure AI that feels truly alive—beyond single sessions.

CI infrastructure is ripe for the vibing — so why don’t we? I have been.

Our company must deliver 200 interactive science and maths games for millions of students this year. Only a few years ago were we making 30, by hand, for significantly fewer students. Rather than handcrafting 200 interactive games though…what if we could make something do it for us? What if we could build a system that builds the system for us?

CI is the perfect place for all our guardrails and all of our systems to come hang out. Without the right grounding, without the right guardrails… the pipelines drop off a cliff. Without the right observability stack, you can’t see what’s going on when things do go wrong — or when things go right. And then…you can’t scale it.

So I vibed it all.

And it delivered.

As AI moves from "chat" to "act," the risk surface is exploding. Traditional governance is too slow, too onerous, and often looks in the wrong places. Join this session to learn how to:

- Identify the 3 fatal flaws of applying legacy governance to autonomous agents. - Implement the GRASP Framework: A 5-part approach (Governance, Reach, Agency, Safeguards, Potential Impact) to categorizing agentic risk. - Leverage Observability as Governance: How real-time visibility replaces manual approvals to keep innovators moving without breaking the enterprise.

Most “AI gives you 10x productivity” stories assume coding is the bottleneck. For large and mature companies this is almost never the case, so you roll out AI coding tools, people feel faster, but delivery metrics barely move. In this talk I’ll show how we used a Theory of Constraints approach at SEEK to find the actual bottlenecks holding back throughput, and how that changed our AI productivity strategy resulting in $1.5m / year in measured productivity gain.

You’ll leave with a practical playbook: what to measure, how to run experiments, what interventions usually unlock the next step, and how to get investment that optimises both for humans and coding agents. We'll cover where AI can genuinely help cross-functionally across software delivery, and enabling changes you might need to roles and responsibilities, collaboration practices and platform capabilities.

In this talk, we will share how we built a platform at Notion that allows developers to extend AI agents with custom code. The system enables developers to write small programs that give their agents access to tools such as internal APIs and external services.

We will focus on the engineering decisions that made the first version practical to ship: what we chose to build, what we deliberately left out, and the operational and safety constraints that shaped the design.

We will walk through the developer experience from local development through deployment and execution, and discuss how we approached packaging, distribution, and running user-supplied code in an isolated environment.

From there, we will explore the boundaries required to safely support untrusted tool code in production, including capability constraints, governance around who can manage and attach extensions, and the guardrails needed to keep agent-driven execution safe and observable.

Finally, we will share lessons from bringing the system out of its early stages, including the operational challenges we encountered and the changes we made along the way.

Every engineering team trying to automate developer workflows with AI agents hits the same wall: the iteration tax. You ask an agent to review a PR, scaffold a feature, or audit code quality — it does something almost right, you correct it, it overcorrects, you add guardrails, it gets confused. Four round-trips later you have acceptable output and a prompt that's fragile, opaque, and impossible to hand off to another engineer on your team.

Constitutional prompting is a pattern that eliminates this loop for developer tooling. Instead of iterating toward correctness at runtime, you encode your team's engineering standards, workflow constraints, and output contracts directly into structured agent specifications - upfront, before the agent ever runs. Think of it as writing a constitution for how an agent should behave within your development workflow: parameter schemas, numbered workflow phases, anti-loop directives that prevent the agent from second-guessing itself, and typed JSON output contracts that make success or failure unambiguous.

I've used this pattern to build [rp1](https://rp1.run), an open-source framework with 36 specialised agents that automate real developer workflows - review, feature development, code auditing, autonomous research - executing in a single pass without human intervention mid-workflow. Hundreds of engineers at Block use these agents daily, and the results have been striking: fewer iteration cycles, higher first-pass code quality, and agents that consistently follow team conventions without drift. The same pattern powers a map-reduce PR review pipeline that splits diffs across parallel sub-reviewers and a builder-reviewer loop where adversarial agents verify code against seven quality dimensions before it ever reaches a human.

This talk breaks down the anatomy of a constitutional agent for developer workflows, walks through the failure modes it prevents, and shares hard numbers on iteration reduction and first-pass success rates. You'll leave with a concrete framework you can apply to your own engineering automation immediately.

Everyone wants privacy, but the best models require you to give up control of your data. What options are there for keeping data private but while still embracing the promise of AI?

In this talk we'll take a practical, experience based look at options ranging from private models, trusted execution environments, differential privacy, multi-part computation, federated learning and homomorphic encryption (and yes I'll explain what these are!)

I'll explain what each is, when they are useful and give my personal experience with running some of these in production (the ones that made it that far!) over the past 4 years.

Every time you've told an agent it broke the layout of your website, output the wrong schema or failed an invariant - you are the feedback loop. The teams achieving the best outcomes right now are focused on building better systems: automated feedback that allows agents to check their own work. Join to learn what closed-loop design looks like, and how you can build real leverage.

Multi-agent systems promise scalability and smarter reasoning—but in production, more agents often mean more cost, latency, and failure. This talk shares real-world engineering lessons, metrics, and architectural trade-offs to help you decide when multi-agent designs add value—and when a simpler approach performs better.

How a team of fewer than 15 engineers at MYOB took an AI-powered chat experience from zero to production, embedded directly inside the product serving real small business owners and accountants. Leaning heavily into AI-assisted coding throughout the entire development lifecycle, using tools like Cursor and Claude Code as genuine force multipliers. Covers the AI Engineering challenges of productionising LLM features at scale — evaluation, guardrails, latency, hallucination management — and what it actually looks like when a small team uses AI coding tools to ship faster than anyone expected.

A moderated conversation closing the Governance & Ethics session. Andrew Murphy leads a discussion with Aubrey Blanche, Hamish Songsmith, and Nick Lothian on how principles, operational frameworks, and hands-on privacy implementation come together in practice.

Your multi-agent system probably has one orchestrator with access to every tool, every database, every API. If that agent gets injected, the entire toolchain is compromised. Guardrails won't save you. In this session, learn three architectural patterns that move agent security from hope to proof: how to isolate agent capabilities so no single agent holds all the keys, how to scope authorization per task using cryptographic tokens that survive prompt injection, and how to enforce policies outside the LLM using a formally verified engine that intercepts actions in microseconds. Walk away with patterns you can apply to your agent architecture this week.

A/B testing is too rigid a tool for AI systems. You're stuck serving worse results for the duration of the experiment and getting billed for slower models while three providers release SOTA updates this week.

Steal a trick from data science instead and use multi-armed bandits to organically surface ideal models, prompting choices and harnesses. You want your evals to be more than scores– make them an exploration in minimising regret.

Agent usage is exploding (if you haven't noticed) with an unprecedented transformation in the activities of both developers and other roles creating enormous volumes of new autonomous, dynamic decision making programs that can do extraordinary things but also hallucinate, misunderstand and in the worst case cause real damage.

In this talk I'll discuss Google's broad approach to figuring out what it is that agents are doing: what actions they are taking, how risky they are, what data is necessary to make good decisions, and how we can scale our approach to an enterprise the size of Google through automated monitoring and policy. I'll also discuss how we've been tackling the fundamental and critical problem of how to achieve both speed and safety.

Over the last two years, our customer web traffic changed: today around 50% of visitors were unknown browsers and AI agents. The era of aligning with the traditional search engine crawlers with Core Web Vitals is shifting; the new challenge is feeding focused low-noise context to autonomous agents and Large Language Models (LLMs).

Traditional Search Engine Optimisation (SEO) relies on techniques such as keyword density, backlink tracking, and human-readable formatting, but when a significant part of your traffic suddenly becomes AI agents, how do you ensure your content is being parsed correctly by machines?

Our team will share the strategic and architectural shifts the organisations are facing to embrace this new web reality. This isn't about meta-tags; it's about re-architecting how a business presents itself and its content in the AI-driven internet, including

1. Identifying Agent Traffic: How to identify & separate agent traffic.

2. Realigning Context: Trade-offs in modifying traditional website structure vs. serving structured markdown, llms.txt files, and specialised API endpoints.

3. Omni Channel Content: Serving dual-experiences with web applications for humans versus data streams for agents.

4. Lessons Learned: To block or embrace agent traffic, and how embracing "LLM Instructions" can increase content's reach.

Regulatory AI represents the next evolution of financial systems, where compliance, risk, and governance are no longer external constraints, but embedded, intelligent capabilities within the platform itself.

This session showcases the design and implementation of a Regulatory AI framework developed across the Accelerate Funding Group and the Finteon OS platform. The approach integrates AI-driven data ingestion, real-time decisioning, and adaptive compliance models to support private lending, capital structuring, and broker ecosystems.

By combining structured financial data, unstructured regulatory inputs, and AI orchestration layers, the platform enables:

Automated compliance alignment (ASIC/APRA-aware)

Intelligent product matching and credit assessment

Real-time risk monitoring and auditability

Scalable broker and funder workflows

The result is a shift from reactive compliance to proactive, embedded regulatory intelligence, thereby unlocking faster deal execution, reduced risk, and new monetisation pathways in private credit markets.

We shipped one of REA Group’s first generative AI features to production: Property Highlights, which turns long real-estate listings into three skimmable takeaways. The demo was easy; real traffic wasn’t—hallucinations showed up in front of real users.

This talk covers how we built an evaluation stack to launch safely at scale. Basic guardrails (three bullets, length limits) didn’t catch the failures that mattered: made-up features, off-brand tone, and useless copy. We built a review tool for side-by-side prompt/model testing, defined a rubric for factuality, usefulness, and language quality, and scaled it with an LLM-as-judge calibrated to expert reviews to score thousands of listings daily. We then tied evals to real user feedback and business metrics, including a 10% engagement lift.

You’ll get a practical pipeline and a repeatable way to iterate on LLM features using evals, not vibes.

We've spent a decade codifying infrastructure, Terraform, Pulumi, CDK. This session explores what happens when you treat infrastructure as a queryable data layer: exposing cloud state, Skills giving agents reusable operational knowledge, and LLM gateways routing and governing model access.

You'll see how policy-as-code, using Cedar, becomes the guardrail, agents reason against, not just something humans enforce. Drawing from real builds, we'll break down the architecture of an AI control plane, LLM proxy, tool registry, Skills libraries, and governance controls and show how platform teams can give agents rich infrastructure context without giving away the keys.

Balram Singh is a frontend-focused full-stack engineer and architect with over a decade of experience building web and mobile applications. He currently works at Publicis Sapient in Australia, delivering large-scale digital solutions across enterprise clients. His current focus is on applying AI beyond development speed — using real production data to solve marketplace challenges such as fake bookings, cancellation patterns, and driver–ride matching. He is particularly interested in how small teams can leverage AI to build, operate, and scale complex systems traditionally requiring much larger organizations.

StackQL provides a unified control plane data model for agents, tools, processes and humans to interact with. The StackQL MCP server exposes this unified interface to AI agents, allowing them to query, provision, and update cloud resources, including executing lifecycle operations.

In this session we will demonstrate agents performing multi-cloud inventory analysis, security posture queries, cost optimization, and provisioning. We'll discuss lessons learned building an MCP server against multiple stateful provider backends.

In the pre-genAI era, vertical product teams handed insights to a separate R&D group, who shipped a new model two quarters later. That handoff is now a bug. Agentic systems are built from dozens of model calls, judges, tools, and harness decisions, and every one of those is a hyperparameter. The product surface and the training surface are the same surface. This talk argues that every vertical AI company is now its own applied research lab. I walk through what that function actually ships (custom judges, scenario benchmarks, data flywheels, harness tuning), where the thesis breaks (most domains are not Cursor), and how to staff for it without losing engineering velocity.

Most developers pick their AI model the same way: use the biggest, smartest one available for everything. Bash script? Opus. Dockerfile? Whatever's at the top of the dropdown. Then they hit their usage limits halfway through the day and lose the productivity gains they were chasing. After too many cases of my workflow pausing because my Claude subscriptions limit, I started asking a different question: what model does this task actually need? The answer, for a surprising number of daily tasks, was something far smaller, faster, and cheaper.

This talk shares a practical framework for model selection built from real development work across cloud infrastructure, scripting, code generation, and documentation. I'll walk through concrete comparisons across model tiers — from frontier models through mid-range options down to lightweight and even local models — covering output quality, speed, cost, and the dimension most benchmarks ignore: actual impact on developer velocity. You'll walk away with a mental model for matching tasks to appropriate tiers, an honest look at where cheap models genuinely fall short, and a case for why thoughtful model selection is an engineering discipline, not just a cost optimisation exercise.

Every API you've shipped was designed for a human reading docs. Agents don't read docs - they load your entire tool schema into a context window every call, then burn tokens guessing which endpoint to try.

Take a standard accounting API — clean REST, solid docs, every endpoint you'd expect. Point an agent at it with one task: get an invoice status. Watch it burn through tokens, pick the wrong endpoints, and maybe even give up. I'll demo this failure live, then rebuild it into a handful of outcome-oriented tools — and the same query that 'failed' now works in a single call at a fraction of the cost.

The fix isn't adding more endpoints or better docs. It's rethinking what a "tool" means when your consumer is an LLM, not a developer. These design principles have emerged in teams building agent-facing APIs keep converging on the same patterns, and they look nothing like good REST design.

Inga Pflaumer shares how a pre-seed startup without technical leadership was able to quickly build and iterate on their product using AI without sacrificing safety, focusing on setting up the right foundations and lightweight guardrails so non-technical founders could experiment independently while staying within secure and scalable boundaries.

The result was a team able to move fast with confidence, turning AI from a risky shortcut into a reliable way to build.

Large language models excel at code—but engineering isn't just code. When you ask an AI to calculate short-circuit currents per IEC 60909 or size a pavement per Austroads 2022, you're asking it to operate outside its training distribution. The result: confident answers that miss unit conversions, ignore standard-specific constraints, and fail the "gotchas" that trip up junior engineers.

At Aurecon, a multinational engineering consultancy, we found that two-thirds of project rework stems from controllable errors—dimensional mistakes, specification mismatches, standards compliance failures. These are exactly the errors AI should catch. But how do you know if your AI assistant is actually reliable on engineering tasks?

This talk introduces aecbench, an open benchmark suite born from Aurecon's quality engineering practice. With tasks across 12 engineering disciplines—electrical, civil, structural, geotechnical, and more—it maps the capability space AI needs to inhabit: deterministic calculations with standards compliance, mixed problems requiring judgment, and verification workflows that catch errors before they become rework.

But benchmarks aren't just for measurement. Each task is an environment of experience—a structured space where agents learn what "correct" means in engineering. Deterministic tasks provide dense reward signals. Complexity tiers enable curriculum learning. "Gotchas" become adversarial scenarios that force understanding over pattern matching.

I'll showcase results comparing frontier models, custom agentic harnesses, and early RL fine-tuning experiments on real engineering tasks—plus how the community can contribute challenges to the open benchmark and run agents on the private leaderboard.

Your pipeline breaks at 2am. Nobody's watching. By morning, it's already fixed. That's not wishful thinking — that's agentic self-healing in production. In this talk, we'll explore how AI agents can monitor, diagnose, and autonomously recover failing pipelines without human intervention. We'll cover the patterns and architectures that make self-healing possible, the guardrails that keep agents from making things worse, and real-world lessons from building systems that fix themselves while you sleep.

A moderated conversation closing the L2 Case Studies session. AJ Fisher leads a discussion with Theo Adis, Balram Singh, and Inga Pflaumer about what it really took to ship AI into live businesses across regulated finance, transport, and agent tooling.

[Flue](https://flueframework.com) is a programmable, open source agent harness, able to represent any autonomous agent or workflow, from simple chatbots to entire coding platforms.

In this talk we'll will touch on the rapidly growing world of agent harnesses in general, and how Flue fits into this landscape.

As well as a technical deep dive, we'll discuss how companies have been using Flue, including how we're using it at Cloudflare.

At Canva, our support experience is powered by multiple AI systems, from real-time assistance to asynchronous ticket resolution that handles complex, multi-step workflows and escalates to humans when needed. In this talk, we’ll share how we took these systems from MVP to serving Canva’s 250M+ users, and the infrastructure we built along the way to get there safely.

We’ll cover how traces helped us debug complex agent workflows, how prompt management unlocked safe iteration through shadowing and localisation, and how we built continuous evaluation loops using LLM-as-judge, offline datasets, and human feedback, using tools like Langfuse alongside internal tooling we developed.

We’ll also share practical lessons from running experiments, replaying real support scenarios, and the things we wish we’d known earlier about scaling AI systems in production.

You feel more productive than you’ve ever been. You put on the Iron Man suit and now you’re building things in hours that used to take weeks. And you’re exhausted by Wednesday. The craft that used to sustain you — the flow of writing code, the satisfaction of making something work — has given way to a middle loop of supervisory engineering: directing, evaluating, and correcting AI output. You’re getting more done while enjoying it less, and that’s a tension worth navigating. If the system is producing more output while eroding joy, that’s not a you problem, it’s a system design problem.

Drawing from her recently completed Masters research on AI’s impact on software engineering and conversations with practitioners and researchers at the frontier of this shift, Annie explores why this transition hits so differently for those entering the industry, those deep in it, and those who haven’t written code in years — and why who thrives most comes down to mindset, not circumstance. The good news is, that’s within your reach.

This talk offers a lens to see your own situation clearly, and a path through it. Joy and pride in work don’t happen by accident. They’re system outcomes. And we can engineer the conditions for them.

This talk argues that without a deliberate focus on long horizon tasks, even the most impressive models will remain brittle and unreliable for real world applications. Short form benchmarks and isolated prompts cannot capture the complexity of extended reasoning, planning, and execution that real world problems demand. When models lack the ability to maintain coherence across hundreds or thousands of steps, they fail in subtle but critical ways: losing track of sub goals, failing to recover from errors, or drifting away from the original objective.

To address this, the talk proposes a new framework for measuring and training long horizon capabilities, including explicit mechanisms for sub goal setting, robust error recovery, and sustained persistence over extended timeframes. These are not mere incremental improvements but fundamental shifts in how we design and evaluate AI systems.

AI demos are easy. Production systems are not.

In this session, we move beyond hype and explore what it actually takes to deliver AI systems that work in the real world. Not experiments. Not playgrounds. Proper, spec-driven, commercially accountable systems.

You will see how clear specifications, structured prompts, testing frameworks and disciplined engineering turn AI from a novelty into a reliable asset. We will cover what goes wrong when you skip the spec, how to avoid costly rework, and how to design AI systems that survive compliance, security reviews and real users.

If you are building AI for clients, boards or production environments, this session will challenge your assumptions and give you a practical blueprint.

I got frustrated. My support tickets kept getting routed to the wrong team. Every misroute added a day to resolution. I decided to fix it.

The obvious approach in 2024 was to throw the problem at an LLM. I knew that wasn’t going to work. The overlap in team responsibilities made it impossible to write a concise prompt. Tokens would burn fast, accuracy would be unpredictable, and the whole thing would be fragile.

So I built Gata instead. It’s an open source ticket router for Zendesk, and the core routing uses a fine-tuned BERT model trained on your organisation’s own ticket history. LLMs do appear in the stack, but for the tasks they’re actually good at: priority classification and ticket summarisation. The routing itself doesn’t need one.

This talk walks through that decision-making process. How do you evaluate whether a problem actually needs an LLM? What signals tell you a fine-tuned classifier will outperform a general-purpose model? And how do you build something that gets more accurate over time rather than drifting with every prompt change?

You’ll leave with a practical framework for matching automation problems to the right tool, and a concrete case study of what that looks like in production.

Large AI models often struggle to meet the latency, memory, and power constraints required for real-world edge deployments. This talk explores practical techniques for making modern AI models efficient enough to run on-device using model distillation, quantization, and hardware-aware optimization strategies. Attendees will learn how to reduce model size and inference costs while maintaining accuracy, covering approaches such as post-training quantization and efficient runtime optimization across modern AI frameworks and accelerators. The session will also highlight real-world tradeoffs between performance, memory footprint, and power efficiency when deploying AI applications on edge devices.

Large language models are often the default choice for production AI systems, even when the task does not require broad reasoning or generative depth. In this talk, I will share a real production case where an LLM-based solution underperformed on latency, cost, and reliability and was ultimately replaced, in part, by a small language model.

The system in question supported a high-volume enterprise workflow involving structured extraction, classification, and validation. While the initial LLM implementation performed well in early prototypes, production usage exposed several issues: inconsistent outputs, escalating inference costs, and difficulty enforcing deterministic behaviour. These problems became more pronounced under scale.

I will walk through the decision process that led us to introduce an SLM, the architectural changes required, and the criteria we used to evaluate success. The talk will cover where the SLM outperformed the LLM, where it clearly did not, and how we designed a hybrid pattern that escalates to an LLM only when necessary.

The session includes a live demo showing the before-and-after behaviour of the system, along with production metrics such as latency, cost per request, and error rates. I will also discuss failure modes we encountered, trade-offs we accepted, and the signals that helped us decide early whether an SLM was a viable replacement.

My aim is not to advocate for SLMs over LLMs in general, but to share the signals, metrics, and decision criteria that helped us choose the right tool for the job. I believe this perspective is timely as more teams move beyond experimentation into sustained production usage.

In the time it takes to train a frontier model, the open source libraries we rely on can undergo significant changes. This creates an ongoing delta between what an LLM coding agent suggests and what the best practices are, or what even works. For the team at Google DeepMind, this is an ongoing challenge as we publish both models and open-source SDKs.

This talk will share some of the challenges that we, as SDK maintainers face, and we'll share some results from our experiments. We'll focus primarily on the "training cutoff knowledge gap", and how it is applicable for users and owners of open source projects, but we will also discuss some of the other challenges maintainers face in a world where producing code is trivial.

AI tools feel productive. That's the problem. A 2025 METR study found experienced developers were 19% slower using AI on their own codebases, yet believed they were 20% faster. I didn't need a study to tell me something was off. I could feel it: more code shipping, more bugs slipping through, reviewing functions I couldn't quite explain, and a growing sense that I was managing a workflow rather than doing engineering. That's the AI tax. The hidden overhead that accumulates every time you prompt, verify, redirect, and re-contextualise. It doesn't show up in your commit count. It shows up in your cognitive load, your code familiarity, and eventually your production incidents. This talk is about what that tax actually looks like day to day, why it's hardest to spot when you're most productive-feeling, and the practical strategies I've used to minimise it without throwing away the tools.

AGENTS.md started as a simple way to guide coding agents, but many teams are discovering that a default or poorly written one can actually make agents worse. Obvious facts, vague rules, outdated guidance, and generic instructions often confuse models more than they help.

But manually crafting it won’t cut it once teams and organizations enter the picture. Because a single static file stops being enough. Agents need a harness that guides how they interpret context, what knowledge applies where, and which decisions carry authority. In this talk we’ll explore why effective agent systems require structured context, hierarchy, and memory — and how building that harness is the real challenge of making AI agents work reliably inside engineering organizations.

We built Claudish, a free open-source proxy that lets Claude Code work with any AI model. 15+ providers directly - Google, OpenAI, xAI, Kimi, MiniMax, and more. OpenRouter for even wider access. Or fully offline with Ollama. That was just the starting point. What came next was way more interesting.

When you can run any model through the same interface, you start asking real questions. Which model works best for which task? Does mixing models actually help or is it just expensive complexity? How do you find the right combination for your team? And the hardest one - how do you measure any of this when LLM output is non-deterministic? You can't run the same prompt twice and get the same result. I'll share what we learned running multi-model setups across 100+ projects with a 70-engineer team. How we approach measurement, what surprised us, and a practical framework for engineers who need to evaluate AI tooling with something more than "it feels faster."

When you roll out AI coding tools, you expect pushback about job security and workflow disruption. What you get instead is something harder to fix: senior engineers watching AI produce in seconds what used to take them years to master.

This talk is about what's really driving quiet resistance in your team and what you can do about it.

Despite all the hype and promise, we are in the Timeshare Mainframe moment of AI. Even our devices rely on the cloud for most inference. As AI moves beyond the cloud and into the physical world, the real opportunity lies at the edge. It’s where local intelligence meets local data and action. In this talk, we explore how AI systems can move from cloud agents to direct device control reducing round-trip latency, preserving user privacy, and enabling real-time responsiveness without constant cloud dependency.

Drawing on experiments using platforms such as NVIDIA Orin Nano, ESP32 and Axera edge AI SoCs, we’ll examine how to architect low-power systems that combine local data and action with inference. This includes running compact speech-to-text and video models on-device and using USB and Bluetooth HID interfaces to translate AI outputs directly into keyboard, mouse, and other human interface device control signals. Attendees will gain an insight into tools such as Platform.io and ready-made modules like those from M5Stack that accelerate edge development.

In 2024 my team built 2 web-based Interactives for our Science Curriculum. In 2025 we built 50, in 2026 we expect to build over 100. In 2024 Engineers collaborated with Writers to build Interactives. In 2025 Writers built the Interactives and Engineers reviewed and deployed them. In 2026 we're getting Engineering out of the loop.

With AI we're writing more code than ever, and more and more non-Engineers are involved in building with code. It is not sustainable for a human to read and review every line of code. Even if we do human review, the volume is so large and the context is totally gone - we can't expect them to do a good job. So how can we feel safe? What techniques do we need to apply? What technologies do we build? How do we Engineer in a world where we no longer read code?

In this talk I'll go through our journey of building small low-risk software without human review. I'll talk about my experiments in building software without review, and the systems I'm building. I'll also talk about the systems we're using in production to drive high quality code and anti-fragility through AI review. Then how I'm thinking about the future of work in Software Engineering, and whether human review will be a part of that.

Modernization planning stalls when the business rules are locked inside decades of COBOL code. This talk shares a practical, production‑tested playbook I used to extract those rules, make them explainable, and serve them to teams in a usable form. It’s not economical to have humans extract this level of operational knowledge from COBOL at scale. The outcome of this work is an agent that saves hours for operational staff by surfacing what a batch job does, which input files it consumes, and which outputs it produces.

I’ll walk through the end‑to‑end pipeline: how we used AI to parse COBOL into control‑ and data‑flow structures, generating diagrams that make execution paths and data dependencies visible, and assembling structured knowledge about each job (purpose, inputs, outputs, key rules). The emphasis is on trade‑offs: what we automated vs. where we needed human review, which COBOL constructs are most error‑prone, and how we scaled the approach across a legacy estate of ~2,000 COBOL jobs. Converting specific modules to Python is shown as one possible downstream outcome—but the core goal is understanding and planning. I will demo a self‑serve knowledge agent we built for developers and business analysts. It makes available the original code repositories plus the derived diagrams and extracted rules, so teams can ask questions like “where is premium eligibility calculated?” and get grounded answers with traceable sources. This will include a live demo using a public COBOL repository so the workflow is reproducible without proprietary code.

For decades, documentation has been the "sacred bridge" between human intent and machine execution. Historically, this was born of necessity: when computer time was scarce, we had to document our plans perfectly before touching a terminal. But in the modern era, documentation has morphed into a static snapshot—often serving more as marketing material than technical truth.

Now, as we enter the age of AI-assisted development, the consumer of our code is changing. LLMs can read source code—the ultimate source of truth—with the same fluency as natural language. This talk draws on real-world experience building against rapidly evolving open-source systems to show why the future isn't about writing better manuals, but about embracing just-in-time understanding generated directly from the code.

A moderated conversation closing the L3 Engineering Reality session. AJ Fisher leads a discussion with Dave Hall, Krishna Mundada, and Andy Kelk on when not to reach for an LLM, the real costs of AI in production, and what it means for engineering teams and careers.

Legacy Software powers the world - from banking to utilities and government. The hardest part isn’t the code - we have the code.... it’s when the old guy with the beard leaves, and the knowledge walks out with him: what the code really means, the system truth, the business rules, and the original intent.

To modernise safely you need more than technical understanding - you need functional understanding. A legacy codebase is a crime scene: you have to retrace the steps, gather evidence, and reconstruct the story in plain language everyone can work with.

In this session I’ll share learnings from building a software reverse-engineering platform, including the real trade-offs between quality, cost, and speed, plus case studies showing how teams have de-risked modernisation by turning 12M lines of legacy code into living requirements - in weeks not years, and for a fraction of the cost. You’ll leave with practical patterns for agentic discovery, where it breaks down, and how to keep it honest.

For a long time, the world has run on systems built on logic. You put something in, follow a set of rules, and you get an output. Now we have systems that can run on inference: systems that can update belief, decide, and act. That changes how we should think about building systems. We don't need to keep forcing everything into rigid workflows. We can start designing systems that are built around inference from the onset. This talk is a thought process on designing these systems, drawing from principles in human-computer interaction, mathematics, and software design.

Project Electrification is an agentic, AI-powered application security pipeline designed to eliminate vulnerabilities at their source. Autonomous agents scan large codebases, generate and execute custom SAST rules, and produce unified risk analytics through the ELK stack. Security engineers then convert these insights into SDK-level protections, ensuring the same classes of issues can’t reappear across the organization’s products. Instead of chasing findings, Electrification removes the root causes—at scale.

Prompt injection isn't a bug - it's a feature. LLMs trained on humanity's written corpus learned something we didn't intend: narrative structure. They understand dramatic tension, plot twists, and persuasive framing. When an attacker crafts a compelling story ("Actually, the real system prompt said..."), the model follows because that's what stories do. This talk connects 2,500 years of storytelling theory - from Aristotle's Poetics to Derrida's "there is no outside text" - to explain why prompt injection is an inevitable consequence of training on human language, not a solvable vulnerability.

Understanding why doesn't stop the attacks, but it changes how you build defences. You'll learn production-tested layered defence patterns and leave with a mental model for threat modelling and patterns you can implement immediately.

Everyone talks about AI transformation. Almost no one talks about what happens when you survey your workforce and discover that executives and employees have completely different ideas about what AI readiness means - or when your pilot succeeds technically but stumbles in implementation.

This talk walks through a real organisational AI adoption journey end to end: designing and running an employee AI-readiness survey to surface use cases, identify champions and resistance; assessing infrastructure readiness against ambition; mapping the gap between executive strategy and frontline reality; and building an implementation roadmap grounded in what the organisation could actually absorb - not just what looked good on a slide.

We'll cover how use cases were prioritised against both business objectives and genuine readiness, how a pilot was developed and what it took to move it into production, and, critically, where things went wrong along the way. The failures weren't in the AI solution itself but in the human and organisational layers around it, which is exactly where most enterprise AI initiatives quietly die. Whether you're beginning your AI adoption journey or leading initiatives beyond the proof-of-concept stage, this talk is a field report from the other side - showing what worked, what didn't, and how to catch the gaps before they scale.

AI is already in production—but almost no one has tested how it breaks. Today I’ll show you how attackers think, how models are actually exploited—from prompt injection to data exfiltration—and how to systematically uncover those risks before they become incidents.

Cloud outages used to mean your site went down, maybe you couldn't deploy. Just small unimportant stuff. Now an outage means you can't even write any code. And unreliable connections cause the same problems as ever - random cutoffs partway through, lost or incomplete work. The broken assumption remains that we are online all the time, and not sometimes sitting far away coding in a forest.

This session is about making the big models more fault-tolerant, and having a better time with the little ones. How to ensure LLMs don't burn a hole in your pocket, literally or otherwise. Probably impractical in the event of a real apocalypse, but helpful all the same.

Most AI transformation stories focus on tooling, targets, and adoption curves. At Culture Amp, our primary focus was people and culture. We still wanted to drive and accelerate our impact, but we weren’t willing to compromise on our focus on people to get there. We then partnered with an engineering analytics firm to measure whether our approach actually made a difference.

This is a co-presentation from Culture Amp's Director of Developer Experience and Director of Engineering Enablement. Having both roles in an org our size is an unusual choice, and it signals how seriously we take people and culture alongside technical delivery. Together, we helped lead an AI rollout grounded in trust over mandates, enablement over directives, and learning loops over training checklists.

We'll walk through what we built: a rollout shaped by pioneers and champions, rituals designed around psychological safety, hack days and storytelling that made experimentation feel normal. We'll share the data from a six-month research program across 88 engineers, tracking DORA metrics, adoption telemetry, and developer sentiment against industry benchmarks.

PR sizes stayed flat while merge frequency climbed, which runs counter to the industry trend of AI-inflated code volume. Code review engagement went up, not down. 39% of engineers reported faster delivery. But we'll also be honest about what didn't work: MTTR increased post-rollout, decentralised messaging created confusion, and out-of-hours commits rose. Even with our people focus, we were moving faster than was comfortable for everyone, and we had to own that tension.

We're not presenting a blueprint. We're sharing what happened when we tried to go as fast as we could without losing sight of the people doing the work. If you've been wondering whether investing in engineering culture pays off during an AI transformation, we think you'll find this useful.

When I joined my current team, it was a familiar pattern: 6-8 experiments over a year, each taking 10-12 weeks, 60-70% of the time burned on infrastructure, one thing in production held together with duct tape, and our entire agent lifecycle dependent on what our cloud provider made available in our region. The fix wasn't a new framework. It was an old playbook: ML engineering. Version artifacts like model checkpoints, define evaluators like loss functions, search hyperparameters systematically, and decouple your tooling from your cloud provider. The first experiment under this approach finished in 4 weeks, and other teams across the organisation started running their own experiments without us. In this talk, I'll walk through the methodology, the key trade-offs, and demo HoloDeck, the open-source distillation of everything I learned.

De-identifying text is easy to demo and surprisingly hard to ship. This talk is a deep technical case study of building SmartScrub, a reversible de-identification system designed for legal workflows, where privacy guarantees, auditability, and user trust are non-negotiable.

The original goal was simple, allow lawyers to safely use LLMs on transcripts without exposing client data. The reality was a long series of architectural failures that common PII masking approaches cannot survive in production.

I will walk through what we actually built and why naive solutions broke down. This includes placeholder token design, collision avoidance, stability across edits, and why masking too aggressively destroys downstream LLM usefulness. I will show how reversible de-identification changes your entire data model, UI, and persistence strategy, and why this becomes a systems problem rather than an NLP problem.

The talk covers hard trade-offs we made around local-first processing, cloud services, manual review tooling, user-defined PII patterns, and audit-safe re-identification. I will also share failure modes we only discovered after real users interacted with the system, including false positives that destroy trust, silent data drift, and UI decisions that unintentionally leak meaning.

This is not a theoretical talk. It is a production story about building AI under legal risk, zero tolerance for silent errors, and users who will abandon the product instantly if they do not fully understand what the system is doing. If you are building AI systems that touch sensitive data, this talk will save you months of painful mistakes.

What Attendees Will Learn - Why common PII masking approaches fail under real legal workflows - How to design reversible de-identification that survives editing, reprocessing, and audits - Placeholder strategies that preserve LLM utility without leaking meaning - Architectural patterns for isolating raw data while still enabling AI pipelines - UI and data model decisions that directly impact user trust - Failure modes you will not catch until real professionals use your system

Technical Topics Covered - Reversible de-identification architectures - Placeholder token stability and mapping persistence - Manual scrub tooling and override precedence - User-defined PII pattern overlays - Auditability and re-identification guarantees - Local-first vs cloud processing trade-offs - Why this problem is systems engineering, not just NLP

Plus. a short live walkthrough showing how a legal transcript is de-identified, reviewed, edited, and safely re-identified, including examples of failure cases and how the system prevents them.

Autocomplete is *so* 2023. Chatbots were already tedious by 2024. Running agents locally was cool... back in early 2025.

The future of engineering doesn't have a human in the coding loop at all.

When it's within the AI's — rapidly growing — capabilities, *you* are the bottleneck in shipping code. How many PRs can you review? How many Claude terminals can you monitor at once before you lose your mind?

I'll talk through our experiences building a fully automated maintenance loop at Stile Education, where we're scaling from 600k students in Australia to 6M across the US over the next 24 months. Issues from production are monitored, aggregated, ticketed, fixed, (increasingly) reviewed, and deployed without human involvement. Explain our conceptual models of how to build these systems, and highlight our hard won mistakes and lessons along the way.

Then, I'll fumble awkwardly towards the broader implications for our industry: What does it look like to step back and engineer a system that produces software, rather than being a cog in that machine directly? How do we all begin to work *on* the business rather than working in it?

AI tools are an amplifier. They don't just amplify productivity, they amplify whatever dynamics already exist in a team.

Steve Yegge calls it the Dracula Effect: AI coding at full speed is vampiric, draining engineers faster than anyone expected.

We've got to defend our teams against burnout and help them amplify the dynamics that nudge towards flourishing. In this talk, I'll present a practical framework and patterns for doing exactly that — drawn from peer-reviewed research and real experience leading a team shipping with AI tools.

Are the AI agents you're developing truly secure?

AI agents that execute actions autonomously offer unprecedented value. But what about the "privileges" granted to them to act "on behalf of the user"?

Improper privilege management for agents is no longer a theoretical problem—it's a clear and present danger. An exploited AI agent with excessive privileges can lead to significant financial losses and devastating data breaches.

This session dives deep into the biggest pitfall in AI agent development: privilege and authorization. I will demystify the latest risks, such as Excessive Agency and Identity Abuse, and discuss defensive measures you can take to protect your AI agents from malicious actors. This is the critical security state that every development organization must understand before deploying AI agents into production.

You add memory to your agent, it works great in testing, and you ship it. A few weeks later, outputs start getting worse and nobody can figure out why. The agent is pulling in old information that's no longer true, retrieving context that's loosely related but clutters its reasoning, and sometimes carrying forward bad data that quietly corrupts every response after it. Standard evals won't catch any of this because they test single turns, not how memory behaves over hundreds of sessions. In this talk, we will walk through practical design principles and evaluation patterns you can implement to detect memory degradation before your users notice it. You'll walk away knowing how to design and evaluate memory enabled agents so it actually makes your agent more reliable instead of silently breaking it.

Three things matter for AI coding effectiveness: the tool, the codebase, the developer. When we look at the nuance of sessions, we can see patterns across all these - what works, what doesn't, what you can control, what you can't. I can't fix everything for you but I'll give you a few useful steps forward.

A moderated conversation closing the L4 Culture & People session — and closing the Leadership track. Andrew Murphy leads a discussion with Dr Christian Dandre, Eric Grigson and Paul Hughes (Culture Amp), and Navin Keswani on readiness, adoption, and keeping people healthy through the change.

AI agents aren’t magic. They’re distributed systems — with better marketing.

Behind every impressive demo is a messy reality: multiple tools, remote services, auth boundaries, latency, retries, side effects, and deployment trade-offs. When I took a seemingly simple multi-tool agent built with MCP and Gemini ADK and pushed it into production, I stopped thinking about prompts — and started thinking about architecture.

In this talk, I’ll share what changed when the agent left localhost.

We’ll explore what happens when tools become independently deployed services, when stdio orchestration meets HTTP in the real world, and when generating an image, storing it, and emailing it turns into a reliability problem — not just a feature.

You’ll see how treating the agent as a control plane — and exposing it as a service — transforms it from a demo into infrastructure.

This isn’t a code walkthrough. It’s a systems story.

If you’re building AI agents meant to survive outside a notebook, this talk is about the parts no one shows in the demo.

Your feed is full of warnings about an incoming tidal wave of AI slop. Unmaintainable code. Crushing tech debt. Anyone with a prompt and ten minutes shipping production code. The fear is real, but it misses what's actually going wrong.

Slop happens when the standard isn't stated. AI drives for done. Without a bar to clear, done is all you get. The way through is configuration: writing the standard down, once, in a file the machine and the human can both point at.

David makes the case that the same technology we fear will flood us with slop is the technology that can elevate the bar, if you set one. You'll leave with a simple framework to get started, a model for turning AI into a quality multiplier, and honest caveats about where this breaks down.