Why Explainable AI Doesn't Yet Grant the Right to an Explanation

The right to receive an explanation for an automated decision is already enshrined in European regulations. However, a systematic review of the research reveals an unfortunate reality: there is still no reliable bridge between this right and actual Explainable AI.

The authors started with 2,643 publications and, after screening, examined 57 full-text papers. Only 19 of these truly combine legal and technical perspectives. This does not mean that the rest are useless: the problem is that legal scholars and developers often address different questions while using the same term—“explanation.”

An explanation is not simply feature importance

In the context of Art. 15(1)(h) of the GDPR, Art. 86 of the AI Act, and related provisions, it is not sufficient to show which features most strongly influenced the model’s prediction. It is necessary to understand who the explanation is intended for and what legal purpose it serves: to inform, to provide an opportunity to challenge a decision, or to ensure some other form of accountability.

The review highlights three recurring errors:

• Authors often incorrectly identify the legal basis under the GDPR;
• Only a few take into account the CJEU’s ruling in the Dun & Bradstreet case—likely due to the timing of the studies’ publication;
• the form of the explanation, which depends on the recipient, is conflated with the content, which is determined by the legal purpose.

The latter error is particularly significant in practice. An explanation for a person who was denied a loan, an explanation for a regulator, and an explanation for an internal auditor may all relate to the same model decision, but they do not have to look the same. A universal framework highlighting a few key factors does not automatically solve this problem.

The authors refer to this as the distinction between the addressee and the purpose of the explanation—the Addressee/Purpose Framework—and propose a four-phase plan for operationalization. They also formulate six open research questions. The shift in focus itself is important: the discussion should not be limited to “how understandable the model is,” but should also address whether a specific explanation is suitable for the procedure mandated by law.

For teams building AI systems in regulated fields, this implies a practical constraint. XAI cannot be added at the end of a project as a visual layer on top of a finished model. First, one must identify the recipient of the explanation, their rights, and the legal purpose, and only then select the method, format, and level of detail.

Otherwise, a company may end up with a technically sound report that does not help a person understand the decision and does not provide them with a practical way to challenge it. In that case, the “right to an explanation” exists only on paper, and there is still no enforceable compliance mechanism in place.

Source: cs.AI updates on arXiv.org