What decided the LLM agents' success here was the control layer, not the dialogue

The work's main conclusion is unwelcome for fans of "let's just add another LLM": in a multi-agent dialogue the quality of individual turns does not guarantee a useful result. If the agents have opposing goals and no shared coordination mechanism, the conversation can end in the visitor's capitulation or in the site's behaviour freezing — with neither side reaching its goal.

The authors tested this in a simulated financial service. One agent was to lead the visitor towards contact with an adviser; the other maintained realistic resistance. Above them worked an Experience Orchestrator that did not so much "talk better" as manage the trajectory of the dialogue:

• a Contextual Bandit chose content variants based on web analytics;
• a PID controller held behaviour within set bounds through dynamic constraints on the schema;
• a POMDP model updated the probabilistic estimate of the visitor's intent.

Across 60,000 simulations this loop took the share of high-intent visitor enquiries to 78.1% against 46.1% under naive LLM control. It is particularly telling that variant selection through the Contextual Bandit explained 97% of the differences between outcomes. In other words, what proved decisive was not a chance set of starting conditions or the model's "empathy" but the policy that chooses the next move.

There is an important caveat. For visitors already inclined to make contact, naive empathetic LLM responses were enough. The real value of the control layer showed up in difficult scenarios where no natural readiness to convert existed. That resembles an engineering rule: orchestration is not always needed, only where the system does not hold the goal by itself.

But practical deployment is still far off. All the results come from an LLM-with-LLM simulation; the PID controller was not tested against the unpredictability of real people, and the authors call a trial on live traffic the next critical step. So the figures cannot be taken directly as proof of higher sales or of effectiveness on a website.

The question now is practical: would you trust such a control loop with real customer dialogues before it has been tested on live people, or would you first keep it a tool for simulations only? Source: cs.AI updates on arXiv.org