Why package an inference service in a container?
Why B is the answer
For one reproducible runtime across environments
Containers carry dependencies. The host still needs its GPU driver.
In detail
- Containers give one reproducible runtime.
- The host still needs its GPU driver.
- The host still needs its GPU driver.
Use case
An illustrative scenario, not a real incident.
A service runs the same in test and production, because the container carries its dependencies.
See it in different ways
flowchart LR A["Model"] --> B["Serving container"] B --> C["Kubernetes"] C --> D["Autoscaling replicas"] D --> E["Monitoring"]
sequenceDiagram participant U as Users participant K as Kubernetes participant T as Triton or NIM participant G as GPUs U->>K: Requests K->>K: Scales replicas on load K->>T: Routes the request T->>G: Batched inference
mindmap
root(("GPU container"))
n1["In the container"]
n2["Dependencies"]
n3["Service code"]
n4["On the host"]
n5["GPU driver"]Key point: Containers give a reproducible runtime.
Why the other options are wrong
- A. To shrink the number of weights in the model
- Weights are unchanged.
- C. To improve model accuracy on unseen data
- Containers do not change accuracy.
- D. To replace the need for a GPU driver on the host
- The host driver is still needed.