You must limit how much an endpoint is called. What fits?
Why A is the answer
AI Gateway rate limiting
AI Gateway enforces rate limits on endpoints, which a request to callers cannot do.
In detail
- AI Gateway applies rate limits to endpoints.
- It enforces them, which a request to callers cannot do.
- Databricks now calls AI Gateway Unity Gateway.
- Set limits by user or group where you can.
Use case
An illustrative scenario, not a real incident.
One team's script floods a shared endpoint. A rate limit in AI Gateway caps its calls, so others stay unaffected.
See it in different ways
flowchart LR A["Requests"] --> B["AI Gateway"] B --> C["Rate limits"] B --> D["Inference tables"] B --> E["Usage tables"] D --> F["Agent Monitoring"]
sequenceDiagram participant U as Caller participant G as AI Gateway participant E as Endpoint participant T as Inference tables U->>G: Request G->>G: Apply rate limits G->>E: Forward E-->>U: Response E->>T: Record the request and response
sequenceDiagram participant P0 as Caller participant P1 as AI Gateway participant P2 as Endpoint P0->>P1: Request P1->>P2: Within the limit P1-->>P0: Over the limit, refused
Key point: AI Gateway enforces rate limits.
Why the other options are wrong
- B. A prompt that asks callers to slow down their requests
- Callers can ignore a prompt.
- C. A larger serving endpoint
- More capacity accepts more calls rather than limiting them.
- D. Usage tables alone
- Usage tables record use, and they do not limit it.