Explainable AI (XAI)

Artificial intelligence in which the results of the solution can be understood by humans.

How it works

XAI encompasses a range of techniques. LIME (Local Interpretable Model-agnostic Explanations) approximates a complex model's behaviour near a specific prediction with a simpler interpretable model. SHAP (SHapley Additive exPlanations) assigns each feature a contribution score based on game theory. Attention visualisation in Transformers shows which tokens the model focused on. Mechanistic interpretability goes further, attempting to reverse-engineer which internal circuits implement specific behaviours.

Why it matters

Regulators in the EU (GDPR), finance (Basel III), and healthcare increasingly require that automated decisions that affect individuals can be explained. Beyond compliance, explainability helps engineers debug models — a model that reaches the right conclusion for the wrong reason will fail unpredictably in deployment. XAI is also fundamental to AI safety research: understanding what a model is actually computing is necessary to verify it is aligned with human intentions.