Why networking matters
Read network capacity as a market constraint.
← Back to Compute College
Compute College
What is InfiniBand?
InfiniBand is high-performance networking used to connect servers in many large AI clusters.
Plain-English definition
InfiniBand is a high-performance networking technology often used to connect servers in large AI clusters. It moves data between machines with high bandwidth and low latency so distributed training and other coordinated workloads can use many GPUs more effectively.
Memory trick: InfiniBand is the highway system for a GPU city: thousands of buildings are useful only when traffic can move between them quickly.
Why it matters
Large jobs frequently span multiple servers rather than remaining inside one node. Those GPUs must exchange data during the job; if the network slows coordination, accelerators sit waiting while the buyer continues paying. Network fabric therefore changes usable compute supply and effective training cost.
- Large distributed workloads depend on server-to-server communication as well as GPU capability.
- Network equipment and topology can themselves become deployment bottlenecks.
- A cluster suitable for one workload may be uneconomic for another if communication dominates runtime.
Simple example
Consider an illustrative 256-GPU training job priced at $7 per GPU-hour. It costs $1,792 per running hour. If suitable networking completes the job in 100 hours, raw cost is $179,200. If network bottlenecks stretch the same useful work to 130 hours, raw cost rises to $232,960, an additional $53,760 before overhead.
- The hourly quote does not reveal how long a distributed job will actually run.
- Network effects matter most when many nodes must synchronize frequently.
- Treat runtime comparisons as workload-specific measurements, not universal claims about one fabric.
Example figures are illustrative calculations, not current quoted market prices.
Common mistake
Do not evaluate a large cluster solely by accelerator model and GPU count. The buyer purchases completed work, not a list of components. Slow communication, unsuitable topology, or insufficient storage flow can erase the apparent savings of a cheaper accelerator quote.
Practical takeaway
What you can do with this
When comparing large capacity, collect the network fabric, topology, cluster size available at once, and evidence from a representative workload. For investors and analysts, treat networking infrastructure as part of deliverable AI supply, alongside chips and power.
- Procurement teams: require network specifications and workload comparison methodology in material cluster bids.
- ML teams: measure scaling efficiency as jobs grow from one node to multiple nodes.
- Analysts: distinguish installed accelerators from high-quality distributed capacity available to buyers.
- Operators: find whether delays come from network, storage, scheduling, memory, or the model itself.
Decision check: do not approve a distributed-training quote unless the planned job size and network assumptions are visible in the cost estimate.
Compute College track
Power & Data Centers
Step 8 of 17: What is InfiniBand