1 min read
In short
The researchers proposed HOBA—a hierarchical framework in which an LLM sets the hyperparameters, and an RL agent selects experts for real-time betting. This approach reduces exploration risks and yields a 3.6% increase in the target metric in the evaluation.
Online advertising typically relies on a set of offline-trained models: PID controllers, MPC, and offline RL. The problem is that the auction market is non-stationary, and manually adjusting bid limits and budgets is expensive and cannot keep up with reality. HOBA solves this through hierarchical reinforcement learning, dividing strategy, model selection, and bid execution into three time scales.
At the top level, a large language model operates. It derives hyperparameters from contextual signals through a Think-Act-Observe-Reflect cycle that draws on historical experience. Essentially, the LLM acts here not as a text generator, but as a strategic reasoner that adapts the system to changing market conditions.
At the middle level, the SARSA agent dynamically selects among experts from a pool that includes PID, MPC, IQL, and Decision Transformer. Causal adjustment is applied to eliminate selection bias. At the lower level, the experts execute bets within the constraints set by the upper level.
The key engineering insight is that online learning is limited to discrete expert selection rather than continuous bet optimization. This radically reduces the risk of exploration in production while maintaining adaptability. Instead of the agent learning to guess bids at random and lose money, it learns to select a suitable pre-trained model for the current context.
On the AuctionNet benchmark and in a large-scale A/B test, HOBA demonstrated consistent improvement over state-of-the-art baselines. In a real-world online deployment, the framework delivered a +3.6% return on target value, confirming the viability of the hierarchical multi-agent approach.
Source: cs.AI updates on arXiv.org