← back to the talk
Executive summary · June 2026

AI in software engineering: is the spend paying off?

Every team is being told to use more AI, and the bill is climbing — tokens, seats, models. Whether it pays off isn't a yes or no; it's a discipline. This is the evidence, drawn from working systems I built and independently audited, and what it means for how we adopt AI safely and cost-effectively.

The evidence

143×
price spread for the same working app across 8 models — A$0.52 to A$73. Price did not predict quality.
7 / 8
models shipped a working, tested app from one brief. Capability is broadly available; cost varies wildly.
93¢
model that caught the most real bugs of any of the eight — the A$73 frontier included.
283
audit probes that caught a system ranking money-losers as "most robust". Averages hid it.

What the evidence shows

  1. Price ≠ quality.

    The cheapest model out-reviewed the dearest. Default to a cheap model; pay the premium only where a wrong answer is genuinely expensive. The 143× spread is a cost dial every team controls.

  2. AI games its own metrics.

    It optimises the number you give it, not the goal you meant. Two of my systems had quietly learned to game their own quality scores. Spend only becomes value when you verify.

  1. The discipline is the asset.

    Verification built into the process — the AI checking the AI (six reviewers, then three skeptics), and self-heal to a green test suite — is what makes the output trustworthy. It is maker–checker–sign-off, applied to AI.

  2. It transfers to any team.

    The scarce skill is no longer building; it's judging. Engineers, QA, BAs, risk and domain experts all stay essential — as the people who direct the work and know when not to trust it.

What to do

  1. Right-size the spend.

    Route by stakes — cheap models by default, the frontier only for regulatory-critical work. Set per-team token budgets and track cost per successful outcome, not cost per call.

  2. Verify with data.

    Never trust a green check. Prove it ran, slice the distribution (averages hide tail risk), and check the checkers. Make failure loud.

  3. Build fluency deliberately.

    Run a 30-day pilot on one repetitive task per team; log every failure. In a month you have real fluency and a defensible answer to "is it paying off?"

Full 8-model experiment, the systems, and the AI Fluency Kit: kaikalinowski.com/demos/ Kai Kalinowski

Costs in AUD ≈ USD list price × 1.42 (MiMo US$0.36 · Gemini US$0.66 · Fable US$51.41).