Smarter Doesn't Mean Fairer: LLMs Perpetuate Stereotypes in Medicine

In medicine, it’s not just incorrect diagnoses by AI that are dangerous. An equally troubling scenario is when a model confidently describes a patient based on racial or gender stereotypes, and the doctor accepts that description as neutral.

Researchers at Flinders University tested two reasoning LLMs: o3-mini and DeepSeek-R1. They generated 36,000 clinical descriptions of fictional patients and compared the distribution of race and gender with the expected representation of different groups for common diseases.

The result is troubling: the ability to reason better did not, in and of itself, lead to greater fairness. For o3-mini, the researchers found significant bias in 78% of cases regarding race and in 56% regarding gender. For DeepSeek-R1, the figures were 89% and 67%, respectively. By comparison, GPT-4 showed 67% for both metrics in a previous analysis.

A bias toward Black populations was particularly noticeable when describing conditions such as sarcoidosis, systemic lupus erythematosus, preeclampsia, and essential hypertension. The models also frequently overestimated the proportion of the “majority” gender. In other words, AI doesn’t just make random mistakes—it can turn statistical associations from training data into a patient profile.

A practical takeaway for those implementing LLMs in medicine: a reasoning benchmark does not answer the question of how safe and representative the model is. Separate checks of demographic distributions and ongoing monitoring are needed, especially if the model’s output is integrated into clinical workflows.

However, the study has important limitations. Only two reasoning LLMs were tested, and the analysis focused specifically on generated clinical vignettes rather than actual physician decisions or treatment outcomes. The authors also do not claim that the cause is necessarily due to conscious “bias” in the model: it may select prototypical rather than representative cases due to patterns in the training data. This is still a problem, but the mechanism and the actual harm require further study.

If an AI generates a medical case with specific details about a patient’s race and gender, would you trust it to choose those characteristics without verifying them separately? Source: Hacker News - Newest: ""AI" "LLM""