Your token spend stopped being proportional to your usage the day you deployed your first agent. A single user action now fans out into dozens of model calls, tool calls and retries, so consumption depends on how a loop terminates rather than on how many people are working.
This playbook shows you how to implement the security, observability, and control you need, without asking a single application team to change code.
Common Chellenges
The runaway agent
A retry loop does not terminate. By morning it has made tens of thousands of calls, and nobody finds out until the invoice.
The unattributable invoice
Finance asks which teams accounted for last month. The bill is one number for the whole organisation.
The compliance audit
Security asks for every prompt sent in the last thirty days, who sent it, and what came back.
What you build
- Traffic through one gateway.
Provider keys move out of application config into a Secret. Applications keep speaking the API they already speak.
- Verified callers.
Tokens from the identity provider you already run, plus guardrails that block or redact sensitive data.
- A price on every request.
Attributed to a person, a team or a service, on a dashboard, weeks before the invoice.
- Ceilings that enforce themselves.
Budgets in dollars or tokens. Audit mode first, then on. Blocked requests never reach the provider.
- Applications decoupled from providers.
Stable internal model names, automatic failover, credentials for callers with no identity.
What you can answer afterwards
- Who sent this prompt, and when
- What did this team spend this week
- Can we stop that job before morning
- Can we move to a cheaper model without twelve teams shipping code
Common questions
Q. How do I see which team is spending the most on LLM API calls?
Route the traffic through a gateway that prices each request against a rate catalog you control, and attribute it to the caller's verified identity. Spend then breaks down by person, team, model or credential.
Q. Can I stop a runaway agent job before it runs up a bill?
Yes, with a budget enforced at the gateway in dollars or tokens. Requests over the limit are rejected before the provider is called, so the blocked requests cost nothing.
Q. How do I change model or provider without changing application code?
Applications call a stable internal model name, and you map that name to a provider at the gateway. Repointing it is a configuration change.
Q. What should an LLM audit log contain?
The caller's verified identity, the model requested and the model that answered, token counts, cost, and the outcome, including requests that were refused.
