Build an AI incident response plan: incident types unique to AI systems, detection, containment including kill switches, investigation with traces, and remediation.
An AI incident response plan covers failures that conventional incident processes handle poorly. An AI system that is fully available, returning responses quickly, and producing wrong outputs will not trigger any standard alert, and can run for weeks before anyone notices.
The plan has to account for failures that look like normal operation.
Incident types specific to AI systems
Systematic wrong output. The system produces incorrect results for a category of inputs, often after a model or prompt change. Detection depends on evaluation and sampling rather than availability monitoring.
Data leakage. Information appearing in an output where it should not, including cross-tenant leakage through retrieval or memory. This is frequently reported by a customer rather than detected internally.
Unauthorized action. An agent performing an action outside its permitted set, whether through a permission gap or prompt injection.
Cost incident. Runaway loops or an unexpected usage pattern producing spend far beyond budget, sometimes within hours.
Harmful or inappropriate output reaching users, with reputational and in some cases regulatory consequence.
Provider incident, where the upstream model is degraded, deprecated, or unavailable.
Detection
Availability monitoring is insufficient. Add quality monitoring: run a small evaluation set against production continuously and alert on accuracy falling below threshold.
Sample production outputs for human review at a rate proportionate to consequence. This is the control that catches systematic errors evaluation sets miss.
Alert on behavioral anomalies: unusual tool sequences, spikes in escalation or authorization denial, cost per task exceeding expected range, and unexpected egress destinations.
Make user reports easy and route them fast. For many AI incidents, a user is the first detector, and a slow reporting path extends the exposure window.
Containment
Build a kill switch before you need one, and test it. Every AI system should have a documented way to disable it quickly, ideally without a deployment, and someone on call who is authorized to use it.
Define degraded modes as an alternative to full shutdown: disable automated actions while keeping recommendations, fall back to a deterministic path, or route everything to human handling. Partial containment is often the right response and needs to be designed in advance.
For agent systems, containment includes stopping in-flight tasks and preventing queued ones from starting, which is easy to overlook.
Investigation
You can only investigate what you logged. Traces need to capture the inputs, retrieved context, model version, prompt version, tool calls, outputs, and approvals, with enough retention to cover the detection lag.
The first investigative questions are what changed and when. Model version, prompt, tool definitions, retrieval corpus, and upstream data are the usual candidates, and provider-side model updates you did not initiate belong on that list.
Determine scope: which requests were affected, over what period, and which users or records. This drives notification obligations and remediation effort, and it is impossible without adequate logging.
Reproduce the failure with a test case, and add it to the evaluation set so the fix is verified and protected against regression.
Remediation and notification
Fix forward or roll back, with the choice depending on whether you understand the cause. Rolling back to a known-good prompt and model version is often the fastest containment even when the root cause is unclear.
Correct downstream effects. Wrong outputs may have produced wrong records, communications, or decisions that need reversing.
Assess notification obligations for data incidents, regulated decisions, and contractual commitments. Involve legal and privacy functions early rather than after the technical fix.
Run a review that examines detection lag as well as cause. The most common finding in AI incidents is that the failure ran far longer than it should have because nothing was watching output quality.
Frequently asked questions
Who should own AI incidents?
The existing incident function, extended with AI-specific detection and containment, plus the accountable system owner. A separate parallel process fragments response.
How long should traces be retained?
Longer than your realistic detection lag. Quality incidents surface over weeks, so short retention leaves you unable to determine scope.
Should users be told an AI system failed?
Where it affected them, yes, and being specific about what was wrong preserves trust better than a general apology.
What is the most common gap?
Absence of a tested kill switch, discovered during the first incident when it is needed immediately.
Explore our software and web capabilities or contact Voquarn Code to discuss operating AI systems safely.
Written by
Moueen Togarvi
Founder & CEO at Voquarn Code, focused on product engineering, search growth, and practical AI systems.