# Verify with data, not intent

"The AI said it passed" is not "it works." "Should work" is not "works."
Verification just became everyone's job — and if you're in QA, you're already
the expert. Here's the checklist.

## Before you believe a result

- [ ] **Prove it ran.** A real artifact — a row in the DB, a log line, a file on
      disk, a screenshot — not a claim. AI-built systems rot silently: code that
      throws an error caught and logged below the line anyone reads *looks* wired
      up and produces nothing for weeks.
- [ ] **Check the distribution, not the average.** Aggregates hide bias. Slice by
      the dimension that matters (segment, direction, regime, customer). The averages
      can look flawless while the tail is catastrophically wrong.
- [ ] **Re-derive one number by hand.** Pick a single figure the AI produced and
      reproduce it independently. A number from an AI is a rumour until you've checked.
- [ ] **Test the seams.** Most bugs live *between* components, not inside them. Two
      auto-fixers can undo each other; an alarm can be wired to nothing. Test the
      handoffs, not just the boxes.
- [ ] **Check the checkers.** If the AI reviewed its own work, have a *different*
      pass verify it. The one who builds it should never be the only one who signs
      off — true for people, true for AIs. Even reviewers hallucinate; ~1 in 10 of
      their "findings" can be made up. A second, skeptical pass catches them.

## Make failure loud

- Fail **loud** (surface the error where someone reads it).
- Fail **closed** (a broken step blocks the result; it doesn't sail through).
- **Prove it ran** with data every time, not just the first time.

## Match the verification to what's at stake

Verification has a cost too. Spend it where being wrong is expensive:

| Data / impact class | Minimum verification |
|---------------------|----------------------|
| **Public / low-stakes** | Prove it ran; eyeball the output |
| **Internal / business** | Re-derive a number; check the distribution |
| **Sensitive / PII** | The above + confirm no PII leaked to logs/prompts; a second reviewer |
| **Regulated / irreversible** | Independent re-derivation, executable tests, human sign-off, full audit trail — a green check is never sufficient |

For probabilistic or non-code output (a summary, a classification, a forecast),
there's no test suite — sample it, compare against a human-graded baseline, and
track the error rate over time. "Looks right" is not a measurement.

## The one-line test

> Can I point at evidence that this is true, or am I trusting that it is?
> If it's trust — it's not verified yet.