AI agents are finding new ideas, but so far only in game worlds

AI agents can now be tested on more than writing code to a specification. In AutoWorldModel-Bench they improve a world model themselves, without being told in advance which direction to search in — and that is far closer to research work than an ordinary coding benchmark.

Across 64 sessions, Codex-5.4 and Claude Opus 4.6 improved the starting model in 63 of them. In 91% of the successful sessions the winning change was not hyperparameter tuning but a full-blown research idea: a new objective, a state representation, a rollout procedure or a change of architecture.

The test design is fairly careful. The agent works with one starting world model and a limited compute budget, runs it in eight game environments and receives structured game state: entities and their parameters have already been extracted from the environment and are passed in a common tensor format. That allows experiments to be repeated quickly — a single run takes minutes — and separates modelling the dynamics from recognising the picture.

And that is where the main caveat sits. The benchmark shows agents are capable of finding non-trivial improvements inside a given research sandbox. But it does not yet prove they are equally good at formulating problems, working with noisy real data, or carrying ideas across fundamentally different domains.

The limitations are substantial: the study ran on just 64 sessions, the suite consists of eight game environments, and the state representation is unified and extracted from the game in advance. Besides, the direction of improvement really is open, yet the task still starts from a ready model and a fixed compute budget. So this is an important step from "build to spec" towards autonomous search, but not yet a test of a full-fledged AI researcher.

Which barrier matters more for such agents next: working with real perception, or setting a scientific problem on their own? Source: cs.AI updates on arXiv.org