Problem
Without limits, a single user or tenant can consume your entire LLM budget in minutes through long prompts, loops, or abuse.
Solution
Implement a token budget system that tracks consumption at multiple levels (per-request, per-user/day, per-tenant/month) and enforces hard or soft limits before making LLM calls.
Implementation
- Pre-count tokens using tiktoken or the provider's tokeniser before sending
- Check against budget tiers: request limit (e.g., 4K input), daily user limit (e.g., 100K), monthly tenant limit
- Soft limit → warn user, degrade to cheaper model; hard limit → reject with clear message
- Track consumption in Redis or database; expose usage dashboard
Trade-offs
- Pro: Predictable costs; prevents budget blowouts
- Pro: Fair resource allocation across tenants
- Con: Users hit limits on legitimate complex queries
- Con: Token counting adds work to every request (a local tokeniser is cheap; a provider token-counting API adds a network call)
When to use
Any multi-tenant or user-facing LLM application, especially SaaS products.
When to avoid
Internal tools with trusted users and flexible budgets.