Financial AI agents cannot be evaluated based on a single prompt

The main challenge in evaluating financial AI agents isn’t a lack of test tasks. The problem is that a good result is often determined by unspoken professional standards that aren’t explicitly stated in the prompt itself.

This is the foundation of FinProBench—a benchmark for financial tasks—and the Role-Grounded Rubric Construction (RGRC) method. Instead of deriving evaluation criteria from the task description or the model’s response, the authors first collect real-world deliverables from professionals in the same role, extract competencies from them, synthesize a rubric, and then validate it.

This isn’t just a more labor-intensive way to write a checklist. Rubrics developed at the role level can be transferred between tasks within that role. According to the authors’ estimates, this reduces the estimated cost of creating criteria for a single task by a factor of 6.7 compared to writing each rubric from scratch.

Where the Power of the Prompt Ends

The authors divided 57 professions into two groups: 30 roles with well-represented general conventions in the training data and 27 specialized roles where such conventions are less pronounced.

For the first group, the gap almost disappears: the prompt-only approach achieved 89.2% compared to 90.7% for RGRC. In other words, in familiar domains, the model can already roughly infer expected standards from the task context.

In specialized roles, the picture changes dramatically: 78.0% for prompt-only versus 99.1% for rubrics grounded in professional deliverables. This is an important limitation of the trendy idea that “it’s enough to write a good prompt.” If the required standard is not part of the model’s stable prior, instructions cannot replace samples of actual work.

What Exactly Does FinProBench Test?

The dataset includes 1,723 curated deliverables from 57 professions, 8 financial subsectors, and 161 types of documents or work outputs. The first published part of the evaluation includes 20 full-scale tasks for 20 roles across 7 sub-sectors.

In this sample, human-generated deliverables ranked first on average—73.7 points compared to 70.3, 70.2, and 69.6 for the four systems. However, it is important not to overinterpret this result: the 95% confidence intervals for all systems overlapped, and the systems themselves demonstrated different strengths.

Therefore, FinProBench does not prove that models are already useless in finance. It shows something else: the average score without a role-based category may mask a failure precisely where the task requires narrow professional intuition.

For agent developers, the practical approach is quite straightforward. If the task pertains to a common role, prompt-based evaluation can be an acceptable, low-cost approximation. If, however, the agent must operate in a narrow domain, real-world examples of results and criteria derived from the practice of that role are needed first. Otherwise, the agent might be optimized to produce a “nice-sounding” answer that a professional would immediately recognize as incorrect.

It’s also worth keeping the study’s limitations in mind: so far, only an initial evaluation set of 20 tasks has been published, and the conclusions are based on a specific procedure and a set of professional materials. Another important question is how robust RGRC’s advantage is outside the financial sector and when the roles themselves change.

Source: cs.AI updates on arXiv.org