Agent-based architectures for ARC-AGI-1 without fine-tuning: 67% at $0.62 per task

Sh0ny

9 июля 2026

1 min read

In short

The researchers proposed two agent-based pipelines based on DeepSeek V3.2 without ARC-specific fine-tuning, improving the result from 15.50% to 67.25% pass@2. The key idea is to separate pattern search and program synthesis with reflective orchestration.

Progress on the ARC-AGI-1 benchmark has recently followed two paths: either massive test-time compute on top of front-end models (evolutionary search, exhaustive sampling, long chain-of-thought), or specialized fine-tuning of small models on ARC data. The authors of this paper explore a third approach—an open-weight model operating in a non-reflective mode (DeepSeek V3.2) with a strict budget and without any ARC-specific fine-tuning.

Approach

The main idea is to get the most out of the agent’s architecture rather than from training. Two schemes are proposed:

Explorer-Definer Pipeline—a two-stage pipeline that separates pattern discovery from the synthesis of executable transformations. First, the agent explores the task structure; then, it generates a transformation program.
Reflective Orchestrator — an extension of the pipeline that autonomously initiates a re-exploration of new transformations when previous hypotheses fail to pass validation on training examples.

Results

On the public ARC-AGI-1 dataset of 400 tasks:

Explorer-Definer Pipeline: 57.50% pass@2 at $0.25 per task
Reflective Orchestrator: 67.25% pass@2 at $0.62 per task

Both architectures raise the one-shot baseline of 15.50% by approximately 52 percentage points without benchmark-specific training or heavy test-time compute.

Diagnostics and Analysis

The Orchestrator verifies the falsifiable diagnostic prediction generated by the pipeline. Analysis of unbiased pass@k shows that the pipeline is limited at the generation stage, not the selection stage: accuracy on training pairs covers about 95% of the candidate ceiling. This means that significant improvement requires broader generation, not better ranking.

The orchestrator implements this prediction through adaptive re-evaluation and confirms it: the gain in unbiased pass@1 was +9.81 pp, which matches the gain achieved through selection-mediated pass@2.

Additional ablation analysis revealed that the think tool is a significant component of the pipeline: removing it reduces pass@2 by 5.75 pp.

Source: cs.AI updates on arXiv.org

новости ai агенты llm

Liked this write-up? Get one like it in your inbox every week

Comments

(0)

Approach

The main idea is to get the most out of the agent’s architecture rather than from training. Two schemes are proposed:

Explorer-Definer Pipeline—a two-stage pipeline that separates pattern discovery from the synthesis of executable transformations. First, the agent explores the task structure; then, it generates a transformation program.

Reflective Orchestrator — an extension of the pipeline that autonomously initiates a re-exploration of new transformations when previous hypotheses fail to pass validation on training examples.

Results

On the public ARC-AGI-1 dataset of 400 tasks:

Explorer-Definer Pipeline: 57.50% pass@2 at $0.25 per task

Reflective Orchestrator: 67.25% pass@2 at $0.62 per task

Both architectures raise the one-shot baseline of 15.50% by approximately 52 percentage points without benchmark-specific training or heavy test-time compute.

Diagnostics and Analysis

Additional ablation analysis revealed that the think tool is a significant component of the pipeline: removing it reduces pass@2 by 5.75 pp.