For ARC-AGI-3, a powerful model is more important than a sophisticated agent architecture

The researchers took their previous ARC-AGI-3 agent apart piece by piece. Previously, it had incorporated three ideas—an executable world model, scheduled simplification, and exact replay verification—and it was unclear which one was driving the results. The answer turned out to be counterintuitive: none of them worked as expected.

They assembled four nested agents based on Codex: a text-based baseline; an executable world model without verification; the same variant with simplification; and a full version—with a fixed interface, simplification, and exact replay of observations. We ran everything on GPT-5.4 and GPT-5.5 at “high” and “xhigh” reasoning effort levels.

The most consistent result has nothing to do with architecture. Every agent variant improves when switching to a more powerful model and when increasing reasoning effort. The difference between architectures within a single model-effort setup turned out to be smaller than expected. Simply put: use a more powerful model and give it more tokens to think with—this yields better results than restructuring the pipeline.

The requirement for a persistent executable deliverable isn’t always helpful. The text-only variant—the simplest one, without an executable world—outperformed the flexible-interface executable variant in both GPT-5.5 setups. Simplification helps in three out of four configurations, but not in the weakest one. Full verification took first place everywhere, but at the cost of significantly greater resources.

In an exploratory follow-up on GPT-5.6-Sol, the verification variant solved all public games at both effort levels, achieved ~99% RHAE, and used less than half the actions of the human baseline. It sounds like a victory—but the authors make a fair disclaimer: the model was released after these games were created, and its held-out performance has not been tested. This is saturation of the public dataset, not proof of generalization ability.

A practical takeaway for those building agents: don’t overcomplicate the architecture just for the sake of it. Components like an executable world model can even be harmful in certain configurations. Verification is the only component that consistently outperforms, but you pay for it with resources. The main lever is the model and the depth of reasoning, not a clever orchestrator on top of it.

Source: cs.AI updates on arXiv.org