A client wants an uptime and accuracy SLA. What is the sound way to set it?
Why B is the answer
Base it on measured eval results, with defined handling for failures
Commit to what you have measured, and agree what happens below that level. Absolute promises fail, and refusing all commitments ends the talk.
In detail
- Commit to what you have measured.
- Agree what happens below that level.
- Agree how accuracy will be measured in production.
Use case
An illustrative scenario, not a real incident.
A client asks for an accuracy SLA. The architect offers a figure based on measured evals and defines what happens below it, such as review by a person.
See it in different ways
sequenceDiagram participant S as Sponsor participant A as Architect S->>A: A broad ask A->>S: Problem, baseline and target A->>A: Build and measure A-->>S: Result, limits and options S->>A: Named decision
flowchart LR S0["Measure on evals"] --> S1["Agree the target"] --> S2["Define the handling for failures"] --> S3["Monitor"]
Key point: Commit to what you have measured.
Why the other options are wrong
- A. Copy the SLA of a similar product published by a large competitor in the same market
- Another product's SLA says nothing about yours.
- C. Promise perfect accuracy, and resolve any problems as they come up
- Perfect accuracy is an absolute promise that will fail.
- D. Refuse to give any commitment, because AI output cannot be predicted
- Refusing all commitments ends the discussion.