API rate limiting protects infrastructure from abuse and enables fair usage across customers. Limit dimensions: requests per minute/hour/day, concurrent connections, payload size. Subject dimensions: IP (anonymous), API key (identified), user account (authenticated). Tiered quotas (free/pro/enterprise) align price with usage. Cost-based rate limiting charges heavier operations more: a search query might cost 10 units vs 1 for a lookup. Throttling vs hard limits: throttle degrades gracefully, hard limits reject immediately.
Assigning different costs to different API operations.
A 1000 RPM limit means something different for GET /users (cheap) vs POST /ai/complete (expensive). Cost-based limiting ensures a customer who runs 100 AI requests can't do the same damage as one who runs 100k simple reads.
Throttling (queueing excess requests) risks memory buildup and unpredictable latency spikes. Hard limits (429 immediately) are faster and clearer. Well-behaved clients implement exponential backoff on 429 anyway.
Clients need to know their unit consumption per request to plan usage. Include X-Cost-Units-Used: 10 and X-Units-Remaining: 590 in every response. Surprises at quota exhaustion cause support tickets.
Sign in to share your feedback and join the discussion.