The problem
Many apps and agents call many models. Without one gateway, every app holds its own provider keys, picks models by habit, and finds out about cost overruns at the end of the month.
The goal was an OpenRouter-style gateway: one place that holds the keys, routes each call to the right model for its cost and difficulty, and refuses to spend money the caller does not have.
Architecture
Every call arrives with a scoped app key and passes a budget check first. A tier router picks the model, the call runs in one of two lanes, and every turn writes telemetry on the way back.
Key decisions
Two lanes that never fall back into each other
An agent lane runs tools and warm sessions for long work. A fast lane handles cheap classification, research grounding, and media. A failure in one lane never silently retries in the other, because the two have different capabilities and different costs.
Tiered routing with a clear order
The router resolves a model in a fixed order: an explicit tier, then the caller's role, then a keyword rule, then a default, then a fallback. A cheap classifier picks the tier for unlabelled calls, so expensive models only see work that needs them.
Sticky model choice
An auto-routing alias is resolved on the first turn and pinned to the conversation. A later turn switches models only when a classifier decides it is new work, so a conversation does not change voice halfway through and prompt caches stay warm.
Budgets enforced before spend
A call over budget gets a 402 before any model runs. Budgets belong to a credit group, not to a key, so rotating a key does not reset spending. Provider keys never leave the gateway.
A prepaid credit ledger
Money comes in through one idempotent entry point that writes a keyed row before the balance moves. Webhooks are verified against the raw body, and credits come from the platform's own checkout record, never from the provider's payload. A reconciler sweeps checkouts whose webhook never arrived.
Telemetry on every turn
Each turn records tokens, cache hits, the route decision, and duration. Prompts are fingerprinted and stored once per hash, so the same system prompt across a million calls costs one row. Cached input runs at a fraction of the normal price, which makes cache hit rate a cost metric, not a curiosity.
Results
- One place holds provider keys. Apps hold scoped keys with budgets.
- Cheap work goes to cheap models by default, and expensive models are a choice.
- Overspending fails at the gate, not on the invoice.
- Unit cost is known per app, per tenant, and per conversation.
Tradeoffs
A pinned model can stay on a model that is no longer the best fit, until a turn counts as new work. The classifier adds a small cost and latency to unlabelled calls. Keeping the two lanes apart means a lane outage is visible to the caller instead of hidden, which is the safer failure.