XAI in Production: SHAP, LIME, Attention and When to Use Each

Explainability Is Not a Report: It Is a System Layer

When a regulator asks "why did the system make this decision", they do not expect a 40-page PDF generated three months later. They expect a traceable, consistent, and reproducible answer, available at decision time or immediately after. This means explainability must be integrated into the inference pipeline, not added as an ad-hoc script.

The three families of methods that dominate XAI in production are SHAP (Shapley Additive Explanations), LIME (Local Interpretable Model-agnostic Explanations), and native attention mechanisms of transformer architectures. Each has specific trade-offs in fidelity, computational cost, and context of use.

SHAP: When Precision Matters More Than Speed

SHAP computes the marginal contribution of each feature to the prediction, based on Shapley values from cooperative game theory. The theoretical guarantee is strong: contributions sum to the predicted value, are consistent, and respect symmetry. For tree-based models (XGBoost, LightGBM, CatBoost), TreeSHAP offers exact computation in polynomial time. For deep learning models, KernelSHAP is model-agnostic but computationally expensive.

In production, SHAP is pre-computed in batch for global explanations (which features dominate model decisions) and computed on-demand for local explanations (why this specific prediction has this value). KernelSHAP cost can be mitigated with sampling and caching explanations for similar inputs.

LIME: Fast Explanations for Debug and UX

LIME generates local perturbations of the input, observes how the prediction changes, and fits an interpretable model (typically linear regression or decision tree) that approximates the original model behavior in the neighborhood of the point of interest. It is fast, intuitive, and produces explanations that non-technical users can understand.

The main limitation is instability: two LIME runs on the same input can produce different explanations if the perturbation sampling changes. In production, this is mitigated by fixing the random seed and increasing the number of perturbations, but it introduces a trade-off with latency.

Attention: Native Explainability in Transformers

Attention mechanisms in transformer architectures produce weight matrices that indicate how much the model "attends" to each token (or patch, or temporal step) when generating output. This information is a free byproduct of the forward pass and requires no additional computation.

However, there is active debate about whether attention weights truly "explain" the model decision. Recent studies show that attention can be manipulated without changing the prediction (attention is not explanation). In production, we recommend using attention as a quick saliency heuristic, complemented with SHAP or LIME for formal audits.

Production Integration Patterns

Pattern 1: Synchronous Explainability (Real-Time)

For decisions that require immediate explanation (credit approval, fraud detection with user notification), explainability computation runs as part of the inference pipeline. LIME with reduced perturbations (500-1000) or attention weights are used if the model is a transformer. Added latency is 50-200ms.

Pattern 2: Asynchronous Explainability (Batch/Audit)

For periodic audits or bias analysis, full SHAP is run in batch over a representative sample of recent decisions. Results are stored versioned and joined with the decision log. This pattern allows deep analysis without impacting service latency.

Explainability Architecture on Google Cloud

Vertex AI Explainable AI (xAI) offers integrated feature attributions for tabular and image models. For custom models and advanced explainability, the following architecture separates explanation compute from the inference pipeline: