TensorRT-LLM
NVIDIA's optimised inference library for LLMs on GPUs
Verdict
Highest raw throughput for NVIDIA GPUs. Kernel fusion, INT4/FP8 quantisation, and in-flight batching push tokens/sec to the limit. Pairs with Triton for production serving. Requires NVIDIA hardware.
Other Model Serving
- BentoMLStable
Build, ship, and scale AI applications with Python
- LiteLLMProduction
100+ LLM providers behind a single OpenAI-compatible API
- MLflowStable
Open-source MLOps platform — tracking, registry, and serving
- NVIDIA TritonProduction
NVIDIA's production model serving platform — any framework, any GPU
- OllamaStable
Run open-weight LLMs locally — one-command setup
- Ray ServeStable
Scalable model serving on Ray distributed compute