RL with rewards for responses teaches the model to guess rather than think

A long chain-of-thought improves performance on complex tasks, but it comes at a cost: the accumulation of redundancy, context overflow, and error anchoring. The authors of ThinkReset point out a problem that is often overlooked—and one that directly contradicts the popular approach to training reasoning models.

When the context window is nearly exhausted and the task is still unsolved, outcome-reward RL creates a perverse incentive. The model receives a reward for the final answer, not for the process—and instead of continuing careful reasoning, it takes a guess. The longer the reasoning, the higher the chance that the window will run out before an answer is reached, and the greater the pressure to “fire” prematurely.

The authors argue that the root bottleneck is neither trajectory compression nor control during inference. The problem lies in the absence of a reusable intermediate interface that can replace the discarded history and support further decision-making. Essentially, a mechanism is needed that preserves the essence of the work done while freeing up context for continuation.

ThinkReset implements this idea in the text space through two operations: interface writeback (writing an intermediate result) and reset (resetting the context while preserving the interface). Training directly optimizes the success of continuation after a reset—that is, the model learns not just to solve problems, but to solve them within a limited window with the ability to “reset.”

Across several long-horizon reasoning benchmarks, the approach consistently improves the success rate for fixed context windows. This is an important signal: the architecture of the reward function in RL for reasoning models is not a minor detail, but a fundamental choice that can work against you.

A practical takeaway for those building agents with long-horizon reasoning: if a model regularly runs out of context and starts guessing, the problem may not be the window size, but rather that it was trained to optimize the final answer at any cost. A reusable interface is not an optimization, but a paradigm shift in working with limited context.

Source: cs.AI updates on arXiv.org