In this article
Problem: Without limits, a single user or tenant can consume your entire LLM budget in minutes through long prompts, loops, or abuse.
Solution: Implement a token budget system that tracks consumption at multiple levels (per-request, per-user/day, per-tenant/month) and enforces hard or soft limits before making LLM calls.
Implementation:
- Pre-count tokens using tiktoken or the provider's tokeniser before sending
- Check against budget tiers: request limit (e.g., 4K input), daily user limit (e.g., 100K), monthly tenant limit
- Soft limit → warn user, degrade to cheaper model; hard limit → reject with clear message
- Track consumption in Redis or database; expose usage dashboard
Trade-Offs:
- ✔ Pro: Predictable costs; prevents budget blowouts
- ✔ Pro: Fair resource allocation across tenants
- ✖ Con: Users hit limits on legitimate complex queries
- ✖ Con: Token counting adds latency (typically <5ms, negligible)
When To Use: Any multi-tenant or user-facing LLM application, especially SaaS products. When to avoid: Internal tools with trusted users and flexible budgets.
Related Articles
the ai gateway pattern