A closed-source LLM interpreter that doesn't require new API requests

A closed LLM cannot be directly investigated: the user only has access to the prompt, the response, and the API. A paper on arXiv proposes a workaround—training a separate interpreter that can then evaluate the contribution of prompt suggestions to a selected result without making new requests to the original model.

The scheme consists of two levels. First, an Energy-Based Model (EBM) acts as a surrogate and attempts to reproduce the conceptual consistency between the queries and responses of the target LLM. Then, a lightweight interpreter network is trained based on this “energy surface.”

The output is not an explanation along the lines of “the model thought this way,” but rather an assessment of the influence of individual prompt sentences on a specific target response. This is an important distinction: the method analyzes the model’s observed behavior rather than revealing its actual internal computations.

The authors do not rely solely on the local analysis of a single query. The interpreter is trained on a variety of inputs to capture more general patterns of generation and reduce the dependence of the explanation on a specific example.

The practical value here is clear: if the surrogate mimics the target LLM well enough, a researcher or developer will be able to analyze prompts repeatedly without constantly having to call upon an expensive or limited API. This is particularly useful for large-scale auditing and identifying the prompts that most strongly influence a given response.

However, the quality of the surrogate itself becomes a critical limitation. A good match between the EBM’s behavior and that of the target model does not guarantee that the identified relationships are causal or hold outside the training examples. The presented description lacks metrics, the names of the tested LLMs, datasets, and examples of errors, so it is too early to assess the approach’s readiness for real-world auditing.

The main result of this work is not a “transparent black-box model,” but rather a more cost-effective behavioral analysis layer on top of it. Such a layer may be useful, but its explanations must be verified just as rigorously as the LLM’s own responses: the interpreter, too, remains a model, not a window into the true internal logic.

Source: cs.AI updates on arXiv.org