the previous few weeks, we have now seen the discharge of highly effective LLMs corresponding to Qwen 3 MoE, Kimi K2, and Grok 4. We’ll proceed seeing such fast enhancements within the foreseeable future, and to match the LLMs towards one another, we require benchmarks. On this article, I focus on the newly launched ARC AGI 3 benchmark and why frontier LLMs wrestle to finish any duties on the benchmark.
Motivation
My motivation for writing this text is to remain on prime of the newest developments in LLM know-how. Solely within the final couple of weeks have we seen the Kimi K2 mannequin (finest open-source mannequin when launched), Qwen 3 235B-A22B (presently finest open-source mannequin), Grok 4, and so forth. There may be a lot taking place within the LLM area, and one solution to sustain is to trace the benchmarks.
I feel the ARC AGI benchmark is especially fascinating, primarily as a result of I need to see if LLMs can match human-level intelligence. ARC AGI puzzles are made in order that people are capable of full them, however LLMs will wrestle.
You can even learn my article on Using Context Engineering to Considerably Improve LLM Efficiency and take a look at my web site, which accommodates all my info and articles.
Desk of Contents
Introduction to ARC AGI
ARC AGI is basically a puzzle sport of sample matching.
- ARC AGI 1: You’re given a sequence of input-output pairs, and have to finish the sample
- ARC AGI 2: Just like the primary benchmark, performing sample matching on enter and output examples
- ARC AGI 3: Right here you might be enjoying a sport, the place it’s important to transfer your block into the purpose space, however some required steps in between
I feel it’s cool to check out these puzzle video games and full them myself. Then, you may see LLMs initially wrestle with the benchmarks, after which improve their efficiency with higher fashions. OpenAI, for instance, scored:
- 7.8% with o1 mini
- 75% with o3-low
- 88% with o3-high
As you can too see within the picture beneath:
Taking part in the ARC AGI benchmark
You can even strive the ARC AGI benchmarks your self or construct an AI to carry out the duties. Go to the ARC AGI 3 web site and begin enjoying the sport.
The entire level of the video games is that you haven’t any directions, and it’s important to work out the foundations your self. I get pleasure from this idea, because it represents determining a wholly new downside, with none assist. This highlights your skill to study new environments, adapt to them, and remedy issues.
You’ll be able to see a recording of me enjoying ARC AGI 3 right here, encountering the issues for the primary time. I used to be sadly unable to embed the hyperlink within the article. Nonetheless, it was tremendous fascinating to check out the benchmark and picture the problem an LLM has to undergo to unravel it. I first observe the setting, and what occurs once I carry out the totally different actions. An motion on this case is urgent one of many related buttons. Some actions do nothing, whereas others have an effect on the setting. I then proceed to uncover the purpose of the puzzle (for instance, get the item to the purpose space) and attempt to obtain this purpose.
Why frontier fashions obtain 0%
This text states that when frontier fashions had been examined on the ARC AGI 3 preview, they achieved 0%. This may sound disappointing to some folks, contemplating you had been in all probability capable of efficiently full quite a lot of the duties your self, comparatively shortly.
As I beforehand mentioned, a number of OpenAI fashions have had success with the sooner ARC AGI benchmarks, with their finest mannequin reaching 88% on the primary model. Nonetheless, initially, fashions achieved 0%, or within the low single-digit percentages.
I’ve just a few theories for why frontier fashions weren’t capable of carry out duties on ARC AGI 3:
Context size
When engaged on ARC AGI 3, you don’t get any details about the sport. The mannequin thus has to check out a wide range of actions, see the output of these actions (for instance, nothing occurs, or a block strikes, and so forth). The mannequin then has to guage the actions it took, together with the output, and take into account its subsequent strikes.
I consider the motion area on ARC AGI 3 may be very massive, and it’s thus tough for the fashions to each experiment sufficient to seek out the right motion and keep away from repeating unsuccessful actions. The fashions basically have an issue with their context size and using the complete size of it.
I just lately learn an fascinating article from Manus about how they develop their brokers and handle their reminiscence. You should use strategies corresponding to summarizing earlier context or utilizing a file system to retailer essential context. I consider this can be key to growing efficiency on the ARC AGI 3 benchmark.
Coaching dataset
One other main purpose frontier fashions are unable to finish ARC AGI 3 duties efficiently is that the duties are very totally different from their coaching dataset. LLMs will virtually all the time carry out approach higher on a activity if such a activity (or an analogous one) is included within the coaching dataset. On this occasion, I consider LLMs have little coaching information on working with video games, for instance. Moreover, an essential level right here can be the agentic coaching information for the LLMs.
With agentic coaching information, I imply information the place the LLM is using instruments and performing actions. I consider we’re seeing a fast improve in LLMs used as brokers, and thus, the proportional quantity of coaching information for agentic conduct is quickly growing. Nonetheless, it is perhaps that present frontier fashions nonetheless will not be nearly as good at performing such actions, although it can doubtless improve quickly within the coming months.
Some folks will spotlight how this proves LLMs wouldn’t have actual intelligence: The entire level of intelligence (and the ARC AGI benchmark) is to have the ability to perceive duties with none clues, solely by analyzing the setting. To some extent, I agree with this level, and I hope to see fashions carry out higher on ARC AGI due to elevated mannequin intelligence, and never due to benchmark chasing, an idea I discover later on this article.
Benchmark efficiency sooner or later
Sooner or later, I consider we are going to see huge enhancements in mannequin efficiency on ARC AGI 3. Principally as a result of I feel you may create AI brokers which are fine-tuned for agentic efficiency, and that optimally make the most of their reminiscence. I consider comparatively low cost enhancements can be utilized to vastly enhance efficiency, although I additionally count on dearer enhancements (for instance, the discharge of GPT-5) will carry out effectively on this benchmark.
Benchmark chasing
I feel it’s essential to go away a piece about benchmark chasing. Benchmark chasing is the idea of LLM suppliers chasing optimum scores on benchmarks, reasonably than merely creating the very best or most clever LLMs. This can be a downside as a result of the correlation between benchmark efficiency and LLM intelligence shouldn’t be 100%.
Within the reinforcement studying world, benchmark chasing can be known as reward hacking. A situation the place the agent figures out a solution to hack the setting they’re in to attain a reward, with out correctly performing a activity.
The rationale LLM suppliers do that is that each time a brand new mannequin is launched, folks normally take a look at two issues:
- Benchmark efficiency
- Vibe
Benchmark efficiency is normally measured on identified benchmarks, corresponding to SWE-bench and ARC AGI. Vibe testing can be a approach LLMs are sometimes measured by the general public (I’m not saying it’s a great way of testing the mannequin, I’m merely saying it occurs in apply). The issue with this, nevertheless, is that I consider it’s fairly easy to impress folks with the vibe of a mannequin, as a result of vibe checking tries some very small share of the motion area for the LLM. You might solely be asking it sure questions which can be found on the internet, or asking it to program an software which the mannequin has already seen 1000 cases of in its coaching information.
Thus, what you must do is to have a benchmark by yourself, for instance, an in-house dataset that has not been leaked to the web. Then you may benchmark which LLM works finest to your use case and prioritize utilizing this LLM.
Conclusion
On this article, I’ve mentioned LLM benchmarks and why they’re essential for evaluating LLMs. I’ve launched you to the newly launched ARC AGI 3 benchmark. This benchmark is tremendous fascinating contemplating people are simply capable of full a number of the duties, whereas frontier fashions rating 0%. This thus represents a activity the place human intelligence nonetheless outperforms LLMs.
As we advance, I consider we are going to see fast enhancements in LLM efficiency on ARC AGI 3, although I hope this won’t be the results of benchmark chasing, however reasonably the intelligence enchancment of LLMs.
