The wording of the prompt influences the LLM's cultural responses more than the model itself

If you ask an LLM to “respond as a German resident,” “imagine that you are German,” or “predict how Germans would respond,” you get three different sets of answers. This isn’t just a cosmetic difference: the framing of the prompt turns out to be a key factor determining how closely the model’s responses align with the actual distributions of human values.

The researchers tested GPT-5.4, Claude Sonnet 4.6, Gemini 2.5 Flash, and Qwen3-235B on 101 questions from the World Values Survey across 13 language-country combinations, yielding 21,008 lines of responses. Four formats were compared: a baseline without country-specific prompts, personalization (user-country), role-play (persona-country), and third-person forecasting.

The main practical conclusion: third-person forecasting—“how people from country X would likely answer”—provides the strongest and most targeted cultural alignment in three out of four host models. Personalization and role-play perform weaker and are less consistent. If you’re building an agent that needs to reflect the values of a specific audience, the intuitive approach—“you are a user from country Y”—turns out to be less effective than the more detached “predict how people from country Y would respond.”

This is counterintuitive: it seems that role-play should immerse the model more deeply in the context. But third-person framing apparently activates patterns in the model that were learned from survey data and sociological texts, rather than from stereotypical characters. Personalization, on the other hand, pushes the model toward average “polite” behavior, which blurs cultural specifics.

Standardization focuses on prominent value axes—religiosity, gender roles, and material work-related values. However, institutional trust and issues of democracy remain challenging for all models and formats. This limitation is important: if you use an LLM to simulate public sentiment on politically sensitive topics, the results will be least reliable precisely where the stakes are highest.

A practical implication for agent developers: don’t treat prompt framing as a minor detail. The choice between “you are—,” “imagine that you are—,” and “how would people from— respond” affects the measured alignment just as much as the choice of model. Test all three and measure which one aligns most closely with your audience’s actual data; don’t rely on intuition.

Source: cs.AI updates on arXiv.org