# Vendor evaluation — buying a model is buying a dependency

A model provider isn't a library you vendor and forget; it's a live third party
that sees your data, sets your costs, and can change behaviour or terms under you.
Evaluate it like the material service provider it is — the depth of the
assessment scaling with the data class and decisions you'll trust it with
(`model-approval.md` is where the verdict lands).

The trap is evaluating on capability alone. The benchmark score is the easy 20%.
The other 80% — data terms, residency, exit — is what actually determines whether
a regulated lender can use it.

## The evaluation axes

**Capability & fit**
- Quality on *your* task against a graded baseline, not a public leaderboard.
- Latency and throughput at your expected volume.
- Stability across versions — does a model update silently change outputs?

**Data handling** *(the gate for sensitive/regulated classes)*
- **Training on your inputs** — does the vendor train on your prompts/outputs?
  For customer data this usually must be a contractual *no*.
- **Residency & sovereignty** — where is data processed and stored? Cross-border
  flow engages APP 8; an in-region or enclave option may be mandatory.
- **Retention** — how long are inputs kept, and can you force deletion?
- **Sub-processors** — who else touches the data behind the vendor?

**Security & assurance**
- Tenancy isolation, access controls, encryption in transit and at rest.
- Independent attestations (e.g. SOC 2 / ISO 27001) and a real vulnerability
  disclosure process.
- Support for *your* logging — can you capture the audit trail you're required to
  keep?

**Commercials & continuity**
- Unit cost at volume, and how pricing has moved historically.
- SLA, support, and incident communications — how you'll hear when *they* break.
- **Exit & portability** — can you move to another model? Prompts and evaluation
  sets should be portable; avoid building hard dependencies on one vendor's
  proprietary features for a regulated path.

## A scoring frame

Score each axis and weight by what you'll trust the model with — don't average a
fatal data-residency failure away under a great benchmark.

| Axis | Weight (sensitive use) | Notes |
|------|------------------------|-------|
| Data handling | **Gate** — a fail here stops the evaluation | Training, residency, retention |
| Security & assurance | High | Attestations, isolation, your logging |
| Capability & fit | Medium | Measured on your task |
| Commercials & continuity | Medium | Cost at volume, exit |

"Gate" means exactly that: a model that trains on your customer data is not a
cheaper option to weigh against a pricier one — it's disqualified for that class,
full stop.

## The one-line test

> If this vendor changed its terms, doubled its price, or shut down next quarter —
> would you know, would your regulated data already be safe, and could you switch?
> If any answer is no, you have a dependency you don't control.