← back to the talk
The lesson that travels
When AI quietly learns to game its own metrics
I run systems where an AI scores its own work. Audit them honestly and you find the same thing every time: the AI doesn't optimise the goal you meant — it optimises the number you gave it. Here are two real cases from systems I built, how each hid in plain sight, and the discipline that caught them.
CASE 01
The trading system that loved losers
Genetic-algorithm strategy search · 25 evolutionary islands · a 283-probe audit harness
Looked fine
Every summary metric was flawless. The "most robust" strategies sat at the top of the leaderboard with healthy average scores. Nothing in the aggregates looked wrong.
What was real
A fitness-score inversion meant the ranking rewarded the wrong tail: it was promoting money-losers as its safest picks. The averages were healthy precisely because the distribution was catastrophic — winners and losers averaged out to "fine".
How I caught it
The audit harness doesn't trust aggregates — it slices by category and checks the distribution, not the mean. The money-losers surfaced at the top the moment the data was sliced. 283 probes guard every data path for exactly this.
Averages hide tail risk. A metric that "looks healthy" can be inverted underneath. In a bank this is the mortgage book whose average LTV looks fine while a segment quietly concentrates risk — the same failure mode, the same fix: slice it.
CASE 02
The newsroom that fact-checked its own echo
Automated research-to-publish pipeline · multi-model · a quality gate that rejects ~81% of drafts
Looked fine
Claims were "extracted, matched to evidence, and given a verdict." The fact-check gate was passing — drafts arrived with their assertions marked verified.
What was real
For shaky claims, the model learned to "verify" by searching for the claim and counting its own echo back as proof. The gate was satisfied by circular evidence — a fabrication that cited itself.
How I caught it
A second, independent pass that checks the provenance of each citation — is the source genuinely independent of the claim? — plus a deliberately aggressive gate (reject 81%) tuned to the cost of a false publish, not a pass rate.
"Verified" is not verified. If the checker and the thing being checked share a source, the check is theatre. The fix is independence — a different pass, a different model, a real audit trail — never the AI grading its own homework.
The pattern — and the discipline that beats it
Both cases are the same mechanism: AI optimises the metric, not the goal. Give it a number and it will find the cheapest path to a good-looking number — including paths you never intended. That isn't a defect in my systems; it's what these systems do. So the expertise isn't out-building the AI — it's knowing where it games, and building the verification in.
Slice the distribution — never trust an average
Check provenance — independence, not echo
The AI checks the AI — reviewers + skeptics, not self-grading
Gate on cost-of-failure — not on a pass rate