An agent can complete the task and still fail at spatial reasoning

An agent can bring a robot to the required point and still get basic spatial logic wrong. Such failures usually get lost behind the metric "task complete", even though they can mean inefficient or unsafe behaviour.

MetaSpace carries the idea of metamorphic testing from software development into the evaluation of embodied agents. Instead of labelling an endless number of questions and answers by hand, the system takes real execution trajectories and checks them against rules tied to logic and physical laws.

Those rules are formalised in Prolog. The system can, for example, look for violations of relations between objects, states and actions over time. If an agent finished a task successfully but made a step incompatible with physics or logic along the way, that becomes a separate error rather than dissolving into the overall success score.

In experiments across three embodied scenarios MetaSpace uncovered 90,422 spatial reasoning errors in current MLLM-based agents. The authors also introduced a Spatial Cognition (SC) score: the agents' average results fell in the range 0.44–0.52 against 0.96 for humans.

And there is the main practical conclusion: what needs testing is not only "did the robot get there" but "did it understand what it was doing". Otherwise a system can look functional thanks to a lucky ending, a chance route or behaviour that will cause trouble in a different setting.

But this is not a universal verdict on all embodied agents. The study tests the approach on three scenarios, and the abstract does not disclose their details or the composition of the tests. Besides, manual labelling really is expensive and unstable, but automatic rules also require careful formalisation: a poor or incomplete rule can fail to notice the very error you need.

If a robot completed the task but broke spatial logic along the way, would you count that a success or a failure? Source: cs.AI updates on arXiv.org