Towards Long-Horizon Tasks

Okay. Thanks, Sean. That's the best advertisement I've ever heard of ZAI and GLM. So as you can see I put our model name and our company name here, so it's not relevant right? The ZAI and GLM, why you are not called g.ai, but like G dot AI belongs to Google, so as Google is one of the sponsors, we'll not use G.

AI as our company name. So first today we're talking about long horizon tasks, we're not talking about GLM. So if you want to talk about like open source, IMIT license or Chinese labs, you can reach out to me after the session. But today we are share our understanding of long horizon task as a generic terms.

So first something about our company, actually we bought this domain zdot.ai, so that's the reason why we changed our company name to this one, it cost like more than a million of dollars, like super expensive. And the founder of this lab, he gave me the domain, so I become the head of ZDoA actually.

So now I'm in charge of this chatbot agents, model release, research exploration. So also I cover a lot of research because currently a lot of people are bluffing their new ideas, try to differentiate like who's who's real, who's not real. So that's one of my job. And also I connect the researchers and the marketing team, so inside ZLEA we don't have marketing team.

It's fantastic because we believe, yeah believe that the builder needs to market himself and model. And also we have good organizations like artificial analysis, so we use their bar charts and no need to talk about our models. But frankly speaking Zeta AI is one of the first global companies to explore large language model.

So we start this large language model exploration back in 2020. And in 2021, we have our first paper called GLM. So GLM actually stands for general language model. Yeah at the same time with GPT-two. And as you can see I've labeled two people here, first is Jilin Yang who is the founder of Moonshot AI.

So if you have heard of Kimi, Kimi two point five, Kimi two point six, Zhi Lin is the founder of Moonshot of Kimi and he is one of the author of GOM, but he is the student of the founder of ZDoll AI, so actually it's very interesting style like open and anthropic. So Julian, the the founder of Kimi, wrote the paper of GRM that led the company and created Kimi.

Yeah. And he's students of the founder of ZDogg And also about the model, that's the last I think the last page of GOM. So we are very good at coding energetic task. So GOM 5.1 the model released in March is on par with OPUS 4.6 in coding and we use Clock Code as the harness.

So if you will say or ask which harness is the best coding agent for using GLM, so I still recommend using Clockcode. So Clockcode is a harness for almost all Chinese labs or Chinese model like Kimi K 2.6 or Deepsea v four because I think we need to focus on one thing, the model itself.

So we use other harness as the main coding agent for us. Okay so that's all for GOM and now we are moving to long horizon task. So how many of you have heard of long horizon tasks or long running tasks? Yes. All the labs here are talking about this kind of stuff like Geon 5.1 long horizon, Cloud Opus 4.7 long running task, and also Kimi long horizon.

All happened in April. So what actually it is, we will spend like near twenty minutes covering this topic. First, just to start from some simple stuff, long horizons capabilities are very useful. Because before long horizon task, you can only do the things yourself, but with the development of Open Claw together with Opus 4.5, Opus 4.6, your agents can use agents.

So you can go to sleep and let let your agents finish all the task for you. That's the first thing. And the second is inside OpenCLR or other autonomous agents sometimes you get interruptions because you need to grasp new news, new information and sometimes it went wrong you need to give new instructions.

So long horizon task or long horizon capabilities will make sure that your agents do can finish the goal. You when you set a goal and you you get interrupted, your agents and your model can survive that interruptions. And it also changed our way to see our meetups and hackathons and workshops. So as a model lab, we no longer host three hour hackathon. Our hackathon become forty eight hours a week or so because we want to utilize the night to let all the participants explore this model. For example, seeing our first forty eight hour hackathon, seven people spend their night, let their agents operate on their own, and they they finally win because their agents can do task during night. So I need to assign all the tasks to it before I go to sleep.

So that's a an enemy showing that maybe in the in the next era like people assign tasks daily and all the agents finish all the tasks during night. So what are long horizon tasks? We have we haven't explained it. So there are different methodologies or different types of long horizon task.

So first one is subjective goal. So with subjective goal if I want to build a website, there's no judgment. So I I can say it's beautiful, it's ugly, it meets my requirements, it doesn't meet my requirements. So there's no single metric to optimize. And the second type is objective goal. So if I want to optimize a CUDA kernel, or I want to speed up something inference or KV cache.

So there's some explicit numeric objective, such as speed and latency that can make the model best optimized for. So with subject to goal, here's a prompt example prompt and we also have a video talking about this. For example, we let the model build a beautiful website or a web based Linux replica and we require it to have 50 fully functional apps. But it doesn't matter like which 50 apps they are, so I need to judge it by myself.

And inside ZDollar AI, we are doing a lot of auto eval development. So for this type of thing, it's hard to use numbers to assess the process, but we began to use agents, vision models or other capabilities try to explore all the things. So when people are talking about computer use, you rarely see like people really use computer use to finish task.

But inside evaluation process, there are already lots of people using computer use try to click into the every app and check a status to see whether the full requirements has been fulfilled. So that's the first example of a subjective goal. And even there's a goal, so different models can behave differently.

If you are using an OPUS 4.8 or a GPT-five point five, may achieve that goal. But if you are using smaller models or local models sometimes you cannot help. So what can you do if you you actually want to do a long horizon task by using a model that doesn't have long horizon capabilities?

There are some tips for you. So the core ideas is to use checklist, to use verification in the every step, to let the model reflect and think back and continuously put all the things back into the memory. So if the model cannot handle it, you need to use harness. The reason why we need harness is that model cannot integrate all the things, all the knowledge into itself.

So you need to automatically incorporate all the checklist, all the evaluation process, the verifications into the process. And also there's a organization called METER, it has its own interpretation of long horizon tasks. So their understanding is that the task for humans is highly correlated with the capabilities of long horizon task.

So the time needed to complete the task for humans with appropriate expertise will decide whether that problem is a long horizon task or not. So as you can see there's a like dataset called math, it's quite simple. Maybe currently all the models can handle it 100%, but there are also very tough problems like sweep bench, live code bench, etc. So these these benchmarks or these tasks may cause people about one hour or two hours to finish the task.

So here are some examples. For example, find the script it cost people three second and when you want to optimize a CUDA kernel it cost you eight hours. Yeah but the reason why post it is not because it's very useless, is because it has some flaws.

It might not be ideal for task with objective goals. Here's some reason because first time is not a good measurement for AI behavior. Because as a human like we can't handle it in one hour, two hour, but with AI the inference speed may matter a lot. For example, there's a new like architecture incorporate all the model weights inside the chip, and the inference speed can be more than 16,000 TPS per user and also you have CIRBAs, Grok, these super fast chips. So using these chips and inference, the time doesn't matter. Yeah you can use a single model, the model can finish the task within ten seconds or cost you like more than two hours. Speed doesn't matter that much, maybe tokens or maybe runs turns means more for agents.

And another thing is the task in meter are too specific. For example, so the task is let the model optimize a CUDA kernel by 30 x. So why not two x? Why not like 100 x? So that thing highly depends on the outcome. You want to optimize 2x maybe an engineer only spend ten seconds to figure out a way.

But if you want to optimize 100x, so maybe cost them a year or ten years or they cannot finish it. So for task, actually we think it in a broader way, so it's not let it optimize in a single like digits or numbers, but to think it as a broader topic.

For example, optimize a CUDA kernel as much as possible, whether your model can make it. So that's very important. So my understanding for task with objective goals, I think it in this way, depth means more than time. So your model, your agents needs to find meaningful improvements, needs to continuously improve itself. Here is one of our best like showcase for this idea. So we also let the model optimize a vector database.

From the beginning, it only increase it to like 6.5 ks QPS. But after 600 runs increase from 6.5 ks to 21.5 ks. There are all different strategies to optimize it. And if you understand what's going on here, so actually these strategies are completely different. So cut over or even even they use some like model training skills to optimize for a kernel or a vector database. For example, quantized routing and pruning.

So use different ways to handle different things. And an important stuff you can see here is that there are many failures. All the crosses there means failures. So to achieve a goal, you can fail actually, and you can fail 100 times. But the long horizon capability actually means that even if you continuously fail 100 times, you can figure out a way to learn from these failures, to reflect, to think about new strategies, or you use searches, use all your knowledge to solve the the problem in the end.

So here is a comparison for a CUDA kernel optimizations. So we compare Cloud Opus 4.6, GEM 5.1, Opus 4.5 and GEM five. So what long actually means is that it can goes deeper. For previous models like Opus 4.5 and GLM five, it actually stops there. So when it goes above 800 runs, it can no longer improve that much. So the progress stops.

But for OPUS 4.6, GLM 5.1 and current new OPUS, it can completely improve itself and surpass the last version of itself. So that's the key idea of long horizon. So long doesn't mean time or run, it mean whether you can exceed yourself, exceed the last version of yourself to reach the limit.

And as a model lab, we find it very hard to train such a model, because there are some misconceptions. First is that people think that a large context window makes things quite easy. But GRN 5.1 is actually a 200 ks model, But there are a lot of 1,000,000 context window model that cannot solve these questions, because currently we have harness.

To solve a problem we don't have to make that context that long, because our harness can make us remember the goal, can make us reflect. So the core idea is that whether your model can reflect and like can think like a human, can find all the flaws, can replan when things goes wrong. So all the things need to be made in a very solid way, not just pure context window.

And the second misconception is that people think that if I write a prompt and let the model plan before it act, it can solve all the questions. But frankly speaking nobody like ever achieve, right? So the model doesn't follow your instruction and they often forget all the things you let it know.

And when we want to optimize for something, it actually stopped in the very early phase and told you that I have fixed it, I have optimized it. It will stop quick quite early. So that's the second misconception. And the third misconception is the longer the stronger. So even if AI can optimize for 800 rounds, 1,000 rounds, it doesn't mean that model is more powerful.

Because today we care more about token efficiency. Why not like finish in three rounds, like instead of 800 runs. Right? So to improve the model, maybe in the future we are not letting model to optimize for so many runs, we just need to let it improve maybe in five runs or 10 runs, that's enough. So we're we're trying to make it more efficient.

So how we train the model? I think these two are very important. First to find as many realistic task as possible and to get a robust evaluations. So here's an example is called Frontier Suite. Yeah everyone knows suite, but here's a new dataset called Frontier Suite.

We have something like that. It's super interesting and I think the categories of all the all the questions are quite meaningful. So there are three type of questions or tasks. So first one is implementation. They say that they can actually put all the things into the implementation because if you want to build something is actually an implementation.

For example creating website and a database something like that. And the second type is research. So research is very important because currently people are working on AI for science. So if you want to analyze the research and use long horizon capabilities to solve research problems, you'd better look at these type of questions, how they define the task and how they evaluate.

And the third one is performance, actually stands for optimization. So there are a lot of optimizations there, people can learn from it, and with these problems you can foresee the future. So training model I think it's not that hard because you can have human labeled data, you can use synthetic data, but what's most important is whether you can find meaningful task, whether you can define the task people need and try to find the right answer for these tasks. So that's all for today's sharing and feel free to connect and talk about more research progress or like other type of services. I'm also covering services.

So here's my axe and linking and we are about to have new models coming soon, so stay tuned. That will be I think the next model like will impress the whole open source industry, so it'll be fantastic. And even if I said told you that that context window might not that be important, we will improve a lot in context size.

So still a good way to improve the long horizon capabilities. So stay tuned. Thanks all. Thanks for listening. Yeah.