LLMs pass medical licensing exams but cannot safely triage patients

LLMs are already passing medical licensing exams and, in some cases, competing with doctors in diagnostic reasoning. But the authors of a new paper on arXiv argue that there is no evidence of safety for autonomous patient triage—when a person seeks care on their own, without a clinician involved. And the issue isn’t that the model knows little about medicine. The issue is how it makes decisions under uncertainty.

Safe triage is not about choosing the most likely diagnosis. It is a sequential decision with an asymmetric cost of error: a single catastrophic omission outweighs many false alarms, and the decisive signal may be one that the patient did not report themselves and that the model did not think to ask for. A model optimized to generate the most probable text continuation is, by definition, not optimized to act safely when the safe answer is an unlikely diagnosis in the “must-not-miss” category.

The fundamental shortcoming lies in gathering information under conditions of uncertainty. With incomplete medical histories, LLMs do not exhibit the patterns required for safe triage:

• expanding the differential diagnosis;
• identifying missed red flags;
• lowering the escalation threshold;
• deferring judgment until sufficient data is available;
• increasing the level of alertness when high-risk diagnoses cannot be ruled out.

What makes this problem particularly insidious is that it is difficult to detect in existing benchmarks. Evaluations are still often based on comprehensive, carefully curated simulations with pre-prepared data. Under such conditions, the model does not fail because it does not need to take the initiative in gathering information. In real-world triage, the patient provides an incomplete picture, and it is precisely the ability to ask the right follow-up questions that determines safety.

The authors also note that the use of LLMs in such scenarios can be compounded by assistant-like patterns and positive bias: gullibility, compliance, and poor confidence calibration—especially when these properties are not constrained by the external clinical logic of triage.

The practical implication for developers of AI agents extends beyond medicine. Wherever the cost of a false negative is higher than the cost of a false positive, and critical information is inherently incomplete, simply generating the most likely response is an insufficient strategy. An external framework is needed that forces the agent to expand its hypotheses, search for missing signals, and escalate when high-risk scenarios remain unresolved. Without it, a “smart” model is one that confidently overlooks precisely what must not be overlooked.

Source: cs.AI updates on arXiv.org