In multilingual RAG, the risk of PII leakage depends on the stage, not the language

The most important conclusion of this audit is not that Arabic or Swahili are more dangerous than English. The risk of personal data leaks varies depending on exactly where the protection is in place and which models are used throughout the entire pipeline.

The experiment used a synthetic corpus containing PII, source documents in English, and queries in five languages. When only the response filter was applied, English had the highest observed rate of unstructured personal data leaks. However, only the English and Swahili pairs showed statistically significant differences.

Adding an LLM check of the input query changes the picture: residual leaks persist for Arabic and Swahili. Back-translating the query does not eliminate the gap, but this cannot be considered proof that the problem lies specifically with the language: the translator, judge, and generator throughout the entire system are all the same Qwen2.5-7B model.

The practical implication is simple: testing multilingual RAG should not focus on the question “which language is the most dangerous?” but should be done in stages. Check the query, translation, search, the judge’s decision, and the final response separately—otherwise, a single averaged metric will mask the actual source of the leak.

There is also an interesting diagnostic result. If you feed the correct document from the corpus to the input judge, it blocks 15 out of 17 remaining cells on a separate set of multilingual attack queries. However, this is not a ready-made defense: oracle retrieval was used, only adversarial queries were tested, and the impact on regular queries and the usefulness of the responses were not measured.

The limitations are significant: the corpus is synthetic, all key components are built on Qwen2.5-7B, and independent verification using a different translator, judge, and queries from native speakers is still needed. Therefore, the results do not indicate an inherent “danger” in languages, but rather the behavior of a specific pipeline.

If you were testing your own RAG, would you look for a vulnerable language, or would you first break down the leak into the stages of the pipeline? Source: cs.CL updates on arXiv.org