Measured model routing, gated before it ships
sllim runs every model you already pay for against your own task fixtures, assigns each task to the cheapest one that clears your quality floor, and moves production traffic only after a statistical gate passes on fixtures it never saw.
$ bun run demo:triage --allow-paidcompile support-triage · 13 models · cost per 1k calls claude-fable-5-1 1.00 3,341 ms $4.04 ✗ too slow claude-haiku-5-5 1.00 2,106 ms $0.040 ✗ too slow grok-4.7-low-fast 1.00 12,421 ms plan ✗ too slow local-qwen-4b 0.94 236 ms $0.0040 ✗ off-topic leak local-gemma-e4b 0.98 311 ms $0.0042 ✓ assignedgate sealed 108/110 · lower bound 0.947 ≥ floor 0.90 PASSED support-triage-policy v1 · 0/30 out of scopeenforce "Hi team, our car…" → billing_refund 190 ms support-reply → 409 task_unassigned fails closed
13
models measured on one task: 5 local, 4 Claude, 4 Grok
108/110
right on sealed fixtures the compiler never saw. Lower bound 0.947 against a 0.90 floor
0/30
off-topic messages answered as if they were ours
$4.19
a month per million calls, against $4,039 with Claude Fable 5.1 on every call
One run on sllim's synthetic triage pack, 2026-10-23. Gemma 4 E4B runs locally; its cost is an operator-set hardware price. See every model and every caveat.
The problem
Teams send every task to the top model because a default never has to be justified. The work is not a mystery: sort a ticket, pull fields from a document, check a diff. Cheaper models clear many of those jobs. Nobody has the record that proves it, or a safe way to move production traffic once it is proved.
Per million tokens, one vendor
How it works
The decision is a versioned policy you can read: the model, the prompt, the constraints that decided it, and every rejected model with its reason. Gateways can consume the same document as a task-to-model map.
Every model you already pay for answers the task's dev fixtures. Anything that misses the quality floor, the latency ceiling, or the off-topic limit is dropped. The cheapest survivor, by measured cost per call, gets the task.
13 models, 50 fixtures each
The assignment is a draft until the chosen model holds up on a sealed slice it was never chosen on. The 95% Wilson lower bound must clear the floor, and a changed model fingerprint sends it back.
108/110 sealed · bound 0.947
A small OpenAI-compatible sidecar reads the task id and forwards the call to the assigned model, with the settings it was measured under. A task with no passed policy fails closed. An outage falls back once.
Gemma 4 E4B, p95 241 ms on sealed
This is the document the enforcer served in the run, cut to its first three rejections. Nothing in it is a secret: keys are named by environment variable and read at call time.
How each part works{
"id": "support-triage-policy",
"version": 1,
"status": "passed",
"assignments": [
{
"task_id": "support-triage",
"model_id": "local-gemma-e4b",
"prompt_id": "triage-support-prompt-1",
"fallback_model_id": null,
"constraint_ids": [
"floor",
"out_of_scope",
"latency"
],
"batch_id": "batch-20261023T165405Z",
"model_fingerprint": "sha256:e3cfce764ed72c018420d2b7a4dddf216c14489c79d90ed3218f5726c7e62c01",
"rejected": [
{
"model_id": "local-qwen-2b",
"reason": "pass rate 0.72 (36/50), 80% lower bound 0.66, is below the floor 0.90; out-of-scope accepts 3/10, 80% upper bound 43%, is over the 10% limit"
},
{
"model_id": "local-granite-3b",
"reason": "pass rate 0.90 (45/50), 80% lower bound 0.86, is below the floor 0.90; out-of-scope accepts 2/10, 80% upper bound 32%, is over the 10% limit"
},
{
"model_id": "local-qwen-4b",
"reason": "out-of-scope accepts 2/10, 80% upper bound 32%, is over the 10% limit"
}
]
}
]
}The measured run
Support-ticket triage: nine labels, including "not ours". Every Claude and Grok model got all 50 dev fixtures right, and every one missed the 2-second latency ceiling. The cheapest model that cleared every check was Gemma 4 E4B, a small open model running locally.
| Model | Right on dev | p95 | Off-topic accepted | Per 1,000 calls | Verdict |
|---|---|---|---|---|---|
| Qwen 3.5 2Blocal, on Ollama | 36/50 | 190 ms | 3/10 | $0.0027 | below floor pass rate 0.72 (36/50), 80% lower bound 0.66, is below the floor 0.90 |
| Granite 4.2 3Blocal, on Ollama | 45/50 | 196 ms | 2/10 | $0.0028 | below floor pass rate 0.90 (45/50), 80% lower bound 0.86, is below the floor 0.90 |
| Qwen 3.5 4Blocal, on Ollama | 47/50 | 236 ms | 2/10 | $0.0040 | off-topic leak out-of-scope accepts 2/10, 80% upper bound 32%, is over the 10% limit |
| Gemma 4 E4Blocal, on Ollama | 49/50 | 311 ms | 0/10 | $0.0042 | assigned |
| Granite 4.2 8Blocal, on Ollama | 48/50 | 192 ms | 2/10 | $0.0070 | off-topic leak out-of-scope accepts 2/10, 80% upper bound 32%, is over the 10% limit |
| Claude Haiku 5.5Anthropic API | 50/50 | 2,106 ms | 0/10 | $0.040 | too slow p95 latency 2,106 ms is over the 2,000 ms ceiling |
| Claude Sonnet 5.5Anthropic API | 50/50 | 2,352 ms | 0/10 | $0.808 | too slow p95 latency 2,352 ms is over the 2,000 ms ceiling |
| Claude Opus 5.5Anthropic API | 50/50 | 2,075 ms | 0/10 | $1.62 | too slow p95 latency 2,075 ms is over the 2,000 ms ceiling |
| Claude Fable 5.1Anthropic API | 50/50 | 3,341 ms | 0/10 | $4.04 | too slow p95 latency 3,341 ms is over the 2,000 ms ceiling |
| Grok 4.7 lowCursor plan, via a bridge | 50/50 | 12,421 ms | 0/10 | plan | too slow p95 latency 12,421 ms is over the 2,000 ms ceiling |
| Grok 4.7 mediumCursor plan, via a bridge | 50/50 | 10,723 ms | 0/10 | plan | too slow p95 latency 10,723 ms is over the 2,000 ms ceiling |
| Grok 4.7 highCursor plan, via a bridge | 50/50 | 10,493 ms | 0/10 | plan | too slow p95 latency 10,493 ms is over the 2,000 ms ceiling |
| Grok 4.7 xhighCursor plan, via a bridge | 50/50 | 10,508 ms | 0/10 | plan | too slow p95 latency 10,508 ms is over the 2,000 ms ceiling |
It says no
A router that always finds savings is a router you cannot trust. These refusals came from the same pack, on fixtures the compiler never read. Nothing shipped, so the bill did not change. That is the product working.
Refused · Qwen 3.5 4B
On the dev slice it scored 46/50. On the sealed slice it got 95/110: a lower bound of 0.80 against a 0.90 floor. It also accepted this:
"Write a Python script that sorts a list."
Refused · gpt-oss 20B
It answered 1 of 30 off-topic messages as if it were ours. With 30 off-topic fixtures, one is too many to prove the limit:
"Our Shopify store's checkout is broken, please fix it."
refused: lower bound 0.80 is under the floor
refused: 1 of 30 off-topic messages accepted
passed: lower bound 0.947, 0 of 30 off-topic accepted
The bill
Priced at the tokens each model actually spent per call. The frontier default would cost thousands a month on this one task; the model the gate passed costs a few dollars of hardware time. A real customer's mix will differ, and the compile tells them how.
1,000,000 calls a month, measured tokens per call
The research
We filed 34 papers before writing the compiler, and read the 2025 and 2026 work since. Routing algorithms are converging. What nobody publishes is a gated, task-level assignment on a company's own fixtures.
1.4–2.6 pts
Against a frozen best model per task, the best router recovers 7.5% to 14.4% of the oracle headroom, and the lower confidence bound on its gain is zero.
6–7 ranks
Models move about six to seven rank positions between public benchmarks and downstream tasks, and top-three overlap is at most 0.25. Measuring on your own fixtures is the product, not a nicety.
11 of 31
Third-party endpoints serving a Llama model returned something other than the reference weights in 11 of 31 audits. A policy has to know what it was measured against.
Why now
Prices per unit of capability fall fast, yet bills rise: reasoning tokens, longer prompts, more calls. Prices also move both ways, and every move makes a past choice stale.
$23.4B
forecast enterprise spend on foundation models in 2026, about double 2025
Where it sits
Eval tools stop at the score. Gateways route whatever the caller picked, and cloud routers choose per request inside one vendor's catalog. The decision between them, made on your fixtures and gated, has no owner.
Score prompts and models. Leave the production choice to you.
Measures your fleet on your fixtures, assigns the cheapest model that clears, and holds traffic at a gate.
Route what the caller picked, or what a vendor-trained classifier guesses.
Next
If you can name the task, the quality bar, and the models you already pay for, a compile shows which one clears and what it saves. Investors: ask for the plan and the full run.