A job outgrows the GPUs in one server. How do you scale?
Why A is the answer
Scale out across servers over a fast network
Larger jobs spread across nodes, joined by a high-speed fabric.
In detail
- Larger jobs scale out across nodes over a fast network.
- Scaling up has a limit.
Use case
An illustrative scenario, not a real incident.
A model outgrows one server, so it spreads across several, joined by a fast fabric.
See it in different ways
flowchart TD A["GPUs in one server"] -->|"NVLink and NVSwitch"| B["Scale up"] B --> C["Server"] C -->|"InfiniBand or Ethernet"| D["Scale out to a cluster"]
mindmap
root((AI cluster))
Compute
GPU nodes
NVLink
Fabric
InfiniBand
Ethernet
Storage
High throughput
Management
Out-of-band networkflowchart TD
Q{"Can one server still fit it?"}
Q -->|"Yes"| A0["Scale up"]
Q -->|"No"| A1["Scale out"]Key point: Scale out over a fast network.
Why the other options are wrong
- B. Move the job onto the server's CPU cores instead
- CPU cores are far slower.
- C. Lower the rack's cooling setpoint by a few degrees
- Cooling does not add capacity.
- D. Replace the storage with slower archive drives
- Slower storage makes it worse.