AI decisions need explaining by what comes next, not only by what came before

The main problem with explanations for reinforcement learning is that they often answer only one question: why did the agent choose this action now? The SPOT method proposes looking further — building several possible continuations and seeing how the policy's behaviour changes.

To do that SPOT takes access to the policy itself and to a simulator of the environment. It runs through actions, models the resulting states and recursively assembles a finite tree. What comes out is not a nice story invented after the decision was made but an empirical map of the agent's preferences and of how they might develop several steps ahead.

That is useful for practical checking. You can compare alternative trajectories, notice delayed consequences and see that the agent prefers something not because of a single feature at the current step but for the sake of behaviour in the future. In an experiment with traffic-light control in SUMO-RL this approach shows what single-moment attribution methods may miss.

The authors also give formal guarantees: with a sufficient number of samples SPOT asymptotically recovers the policy's most likely action when that action is unique, and it describes the cases of divergence under a high-entropy policy.

But that is also the main trade-off. SPOT does not turn a neural network into a fully intelligible system: the tree is bounded by a finite horizon, is built by sampled simulation and depends on having a correct simulator. It is more a tool for checking scenarios and hunting for strange consequences than a universal translator of the agent's internal reasoning. If the environment is modelled poorly, a confident tree may describe not the real future but the simulation's errors.

For which tasks would you trust an explanation in the form of a scenario tree, and where would you demand a check in the real environment? Source: cs.AI updates on arXiv.org