AI Incident Response Playbook

Traditional software incidents have clear symptoms: the service returns errors, latency spikes, or the health check fails. AI incidents are fundamentally different because the system can be operationally healthy -- serving responses within latency SLAs with zero errors -- while producing outputs that are subtly wrong. A recommendation model that starts surfacing irrelevant content, a credit scoring model whose accuracy has drifted below acceptable thresholds, or an LLM that begins hallucinating facts that pass superficial plausibility checks. All of these are incidents, but none of them trigger traditional monitoring alerts.

This asymmetry means AI incident response requires a fundamentally different detection philosophy. Instead of monitoring system health (is it up?), you must monitor system correctness (is it right?). And because correctness is harder to measure than availability, AI incident detection requires purpose-built monitoring layers, sample-based quality evaluation, and feedback loops from downstream consumers of model outputs.

AI incidents require a severity classification system calibrated to AI-specific failure modes. Traditional SEV-1 through SEV-4 definitions based on user impact and revenue loss still apply, but must be extended with dimensions for model quality degradation, data integrity compromise, and compliance violation severity.