# Incident response — when the AI gets it wrong in production

AI systems fail differently from ordinary software. They don't throw an
exception and stop — they return a confident, well-formatted, *wrong* answer and
carry on. So the first job of incident response isn't fixing the bug; it's
**noticing at all**. Everything below assumes the failure is silent until you go
looking.

This is the operational-risk discipline (APRA CPS 230 calls it managing a
disruption to a *critical operation*) applied to a probabilistic component.

## The four moves: detect → contain → correct → learn

**1. Detect.** You cannot respond to what you can't see. The detection surface is
built *before* the incident:

- A monitored quality signal (error rate vs a graded baseline — see
  `verify-with-data.md`) with a threshold that trips an alert.
- The audit trail (`audit-trail.md`) so any flagged output can be reconstructed.
- A reporting path for humans downstream — the borrower, the banker, the analyst
  who saw the wrong number — that reaches someone who can act.

**2. Contain.** Stop the blast radius growing while you investigate:

- **Fail closed.** Route the affected path to the human/manual fallback, or
  disable the feature. A wrong mortgage figure shown to no one beats a wrong
  figure shown to ten thousand borrowers.
- Know your kill switch *in advance* — which flag, who can flip it, and that it
  works. Test it like a fire drill, not on the day.
- Quantify exposure from the audit trail: how many outputs, which customers,
  over what window.

**3. Correct.** Fix the cause, not just the symptom:

- Reproduce the failing input. Add it to the test/baseline set so it can never
  regress silently again.
- Decide whether it was the model, the prompt, the data, or a missing guard —
  the fix differs for each. A bad number from a float where you needed integer
  cents is a code fix; a fabricated citation is a verification-gate fix.
- Remediate the affected outputs and customers — re-run, re-issue, notify.

**4. Learn.** A blameless post-incident review, written down:

- Timeline, root cause, blast radius, what detection *should* have caught it
  sooner, and the control you're adding so this class of failure is caught
  automatically next time.
- Feed it back into `lessons-log.md`. An incident that doesn't change a control
  will recur.

## Severity, so the response matches the stakes

| Severity | Looks like | Response |
|----------|-----------|----------|
| **SEV-1** | Wrong output of record reached customers (money, credit decisions) | Page on-call, contain immediately, notify risk/compliance, customer remediation |
| **SEV-2** | Wrong output caught internally before customer impact | Contain, fix, post-incident review |
| **SEV-3** | Quality drift trending toward a threshold | Investigate within the sprint, tune the gate |

## The one-line test

> When this system returns a confident wrong answer at 2am, what notices, what
> stops it spreading, and who gets paged? If you can't name all three, you have a
> model in production and no incident response — only hope.