AI explainability requirements are not satisfied by a SHAP plot or a feature importance chart. Regulators asking for explainability want a plain-language account of why a system produced a specific outcome, written for the person affected by it or the official reviewing it, not for the engineer who built it. Confusing the two is the single most common gap AICA sees when governance teams prepare for audit.
This distinction sits at the center of every serious AI regulation now in force or in draft, and it is the one most technical teams get wrong first.
What Do AI Explainability Requirements Actually Mean?
Explainability requirements ask an organization to show, in terms a non-technical reader can follow, why an AI system reached a particular decision about a particular person or case. That is different from interpretability, which asks whether a technical expert can trace how a model's internal mechanics produced an output.
A model can be highly interpretable to a data scientist and completely unexplainable to the person it just declined for a loan. Regulators care about the second condition. That is the gap this article maps.
Most compliance failures on AI explainability requirements happen not because a team lacks the technical tooling, but because they hand regulators an interpretability artifact and call it an explanation.
Technical Interpretability: What It Is and What It Is Not
Technical interpretability is the set of methods that let a practitioner inspect how a model weighs inputs to arrive at a prediction. SHAP (Shapley Additive Explanations) and LIME (Local Interpretable Model-agnostic Explanations) are the two most cited approaches, alongside simpler tools like feature importance rankings, partial dependence plots, and attention visualization for neural networks.
These methods answer a narrow, technical question: which input variables moved the model's output, and by how much. SHAP assigns each feature a contribution value derived from cooperative game theory. LIME approximates a complex model locally with a simpler, interpretable one around a specific prediction. Both are genuinely useful for model debugging, bias detection during development, and internal validation.
Neither method was designed to produce a sentence a consumer or a regulator can act on. A SHAP output showing that "debt-to-income ratio contributed -0.34 to the score" is accurate and useless to someone asking why their loan was denied. It requires a second translation step that most technical teams skip, because nothing in the model development pipeline forces them to take it.
Interpretability tools also operate on the model, not on the decision as experienced by the individual. They tell you what the model did internally. They do not tell you whether the underlying logic is defensible, whether it used a proxy variable that correlates with a protected characteristic, or what the affected person can do about the outcome.
Regulator-Grade Explainability: What Officials and Affected People Actually Need
Regulator-grade explainability is a plain-language account, tied to a specific decision, that a layperson or a reviewing authority can understand without technical training. It answers four questions: what was decided, what factors drove that decision, whether a human reviewed or can review it, and what recourse exists.
The EU AI Act is the clearest statutory anchor for this standard. It imposes transparency obligations on providers and deployers of certain AI systems, including a requirement that affected individuals be given a clear and meaningful explanation of the role of the AI system in a decision-making procedure and the main elements of the decision taken, for high-risk systems under Article 86. That obligation cannot be satisfied by a technical interpretability report handed across without translation. The Act's transparency provisions (Article 13, on instructions for use, and the broader Chapter III high-risk requirements) are written for deployers and, downstream, for the people affected by deployment decisions, not for the modeling team.
Regulator-grade explainability typically includes:
- A decision summary in ordinary language, free of model jargon.
- The main factors that influenced the outcome, described in terms the subject recognizes from their own application or record.
- A statement of whether the decision was fully automated or human-reviewed, and by whom.
- A path to contest or request human review, where the regulation requires one.
- Documentation the organization can produce on demand showing how the explanation was generated and validated.
This is a governance and communication output, built using interpretability methods as raw material but shaped by legal, policy, and product judgment. Building it requires people who understand both the model and the obligation, which is precisely the skill set AICA's governance credentials are built to certify.
Comparison: Technical Interpretability vs. Regulator-Grade Explainability
| Dimension | Technical Interpretability | Regulator-Grade Explainability |
|---|---|---|
| Primary audience | Data scientists, ML engineers, internal model validators | Affected individuals, regulators, auditors, boards |
| Typical format | SHAP/LIME plots, feature importance tables, attention maps | Plain-language decision notice, policy documentation, audit narrative |
| Underlying question | How did the model compute this output internally? | Why was this decision made about this person, and what can they do about it? |
| Governing standard | Model development best practice, internal MLOps policy | Statutory obligation (e.g., EU AI Act transparency articles), sectoral regulation |
| Timing | During model development, tuning, and validation | At or after the point of decision, on request or by default |
| Who produces it | ML engineering, data science | Compliance, AI governance, legal, product, with technical input |
| What it proves | The model behaves as designed and is technically sound | The organization meets its transparency and accountability obligations |
| Failure mode if missing | Undetected bias, poor model performance, debugging difficulty | Regulatory non-compliance, individual harm, loss of contestability |
Why Do Regulators Draw This Line?
Regulators separate these two categories because they are solving different problems. A financial regulator auditing a credit model wants assurance the model is sound, which draws on interpretability evidence. The same regulator, acting on a consumer complaint, wants to know whether the individual received a fair, understandable account of the decision, which draws on explainability evidence. Both matter. Neither substitutes for the other.
The EU AI Act structures its high-risk obligations around this same split. Providers must maintain technical documentation sufficient to demonstrate the system's design and function (Annex IV), a task that leans heavily on interpretability tooling and rigorous internal documentation. Deployers, separately, face obligations to inform and, where applicable, explain outcomes to natural persons affected by the system. Treating Annex IV documentation as a substitute for the deployer-facing explanation obligation is a category error that shows up repeatedly in early compliance attempts.
NIST's AI Risk Management Framework reinforces the same divide without using EU statutory language. Its "Govern" and "Map" functions call for documentation that different stakeholders, including those outside the technical team, can use to understand system behavior and limitations. ISO/IEC 42001, the AI management system standard, similarly requires organizations to define how AI-related information is communicated to interested parties, a governance function distinct from model validation.
What Should an Explainability Program Actually Produce?
An organization that can answer "how do you comply with AI explainability requirements" only by describing its SHAP pipeline has not built an explainability program. It has built an interpretability practice and stopped one step short.
A functioning program produces three concrete things. First, a mapping of which systems and use cases trigger a statutory or sectoral explanation obligation, since not every model needs regulator-grade explanations and treating all of them as if they do wastes effort better spent elsewhere. Second, a template and workflow for converting technical model outputs into the plain-language notice a decision-affected person receives, reviewed by someone who did not build the model. Third, an audit trail showing when explanations were generated, what they said, and how the organization validated that the explanation matched the actual decision logic, not a simplified fiction of it.
This is fundamentally a governance design problem, not a modeling problem. It requires someone who can read the EU AI Act's transparency articles, translate them into an operating procedure, and hold engineering accountable for feeding that procedure with accurate technical input. That role rarely exists by accident inside a data science team. It has to be built deliberately, staffed, and audited.
How Should a Board Evaluate Its Own Explainability Readiness?
A board or executive team can test readiness with a small number of direct questions, asked of the team responsible for AI governance rather than the modeling team alone. Can the organization produce, within a defined turnaround, a plain-language explanation for a specific automated decision affecting a named individual, using documentation on file, not built ad hoc after the request arrives.
Is there a documented mapping of which systems fall under high-risk transparency obligations, and has that mapping been reviewed since the EU AI Act's relevant provisions took effect. Does the explanation template distinguish between what the model computed and what the organization is legally required to disclose, given that not every interpretability output belongs in a public-facing explanation. And is there a named owner for this function who sits outside the engineering team that built the model, since self-certification of your own explanation is a weak control regulators increasingly discount.
Boards that cannot answer these questions with evidence, not intention, are carrying undocumented AI regulatory exposure regardless of how sophisticated their underlying models are.
Key Takeaways
- Technical interpretability (SHAP, LIME, feature importance) explains how a model computes an output internally. Regulator-grade explainability explains, in plain language, why a specific decision was made about a specific person, and what they can do about it.
- The EU AI Act's transparency obligations, including the requirement for a clear and meaningful explanation of high-risk decisions, are written for the affected individual and the regulator, not the modeling team, and cannot be satisfied with raw interpretability output.
- NIST's AI RMF and ISO/IEC 42001 reinforce the same split by requiring documented communication practices for non-technical stakeholders, separate from technical model validation.
- A functioning explainability program requires a system-level mapping of which use cases trigger disclosure obligations, a plain-language explanation workflow independent of the modeling team, and an auditable trail proving explanations match actual decision logic.
- Boards should be able to produce evidence of explainability readiness on request, not describe it in general terms, and should assign ownership of this function outside engineering.
Organizations building this capability from the ground up, rather than retrofitting it after an audit finding, typically start by training the people who own AI governance rather than the people who own the models: AICA's CCAIGO (Certified Chief AI Governance Officer) credential covers AI governance frameworks and operating models, global AI regulation and standards including the EU AI Act, NIST AI RMF, and ISO/IEC 42001, AI risk management and assurance, responsible AI policy design and enforcement, AI audit readiness and documentation, and board and regulator engagement.