Tuning the GPU Interconnect in Multi-Node Language Model Pretraining
Training a language model across several machines costs something that training on one machine does not. At every optimizer step, each GPU computes gradients, the corrections the model learns from, and every GPU’s copy has to be added together. On a multi-node job that sum crosses a network. This article reports what happened when I tried to make that network faster, pretraining a 2.06-billion-parameter model on two multi-node GPU clusters, each linked by a high-speed fabric, the fast network that carries traffic between nodes.
The short answer is that one change was worth a great deal, and every change after it was worth nothing, could not be measured, or crashed the job. The most useful results were therefore the negative ones. That is a common shape for infrastructure work and a rare shape for a written account of it, so the negative results are the subject here rather than an appendix.
Allocation time is billed, so every test used two nodes rather than the six the production job runs on. That sounds disqualifying, because a two-node job barely exercises the network. It is not, for a reason given in the section on scaling below, and that workaround is the most portable thing in this article.
The one change that mattered
Figure 1: Training throughput per GPU before and after installing a fabric-aware communication plugin, measured inside real pretraining slices at 8 GPUs across 2 nodes with the production batch shape. The single-node reference is the median of five separate slices on the same machine and has no cross-node traffic; it is a ceiling rather than a controlled comparison, since it differs from the two-node arms in gradient accumulation depth as well as in node count, a confound quantified later in this article. Direct labels give tokens per second per GPU.
NCCL, the library PyTorch uses to add gradients across GPUs, picks a network at start-up. On this cluster it picked the wrong one. With no plugin that understands the fabric, it fell back to TCP over the management ethernet, a ten-gigabit link meant for administration rather than for data. Installing aws-ofi-nccl v1.6.0 and pointing it at the fabric multiplied training throughput by 1.45.
This is the only change from the entire exercise that reached production, and it is worth being precise: it is not a tuning result but a configuration defect. Nothing was optimized. A default was wrong, and the fix was to stop using it.
Two details cost time and are worth recording. A newer plugin, v1.9.1-aws, was worse: 1.06 GB/s on the microbenchmark against v1.6.0’s 1.92, close to the socket baseline it was supposed to replace. Newer was not better. The hypothesis that predicted it would be, that NCCL 2.27.5 advertises plugin interfaces the older build does not implement and would therefore fall back to a degraded path, was simply wrong.
Separately, requesting the Tree algorithm on its own crashes training outright, failing to find an algorithm for AllGather. A collective is one of these coordinated operations across every GPU, and the memory-saving optimizer this job uses performs collectives the microbenchmark never exercised. The production configuration names both algorithms permissively.
What the benchmark got wrong
Before touching real training I built a microbenchmark: repeatedly all-reduce a 4.1 GB buffer, the size of this model’s gradient, across eight ranks on two nodes, and time it. An all-reduce sums a value across every GPU and hands the result back to all of them; a rank is one GPU’s process. The benchmark ranked the candidate settings clearly, and it was substantially misleading about all of them.
Figure 2: Speedup predicted by a standalone all-reduce benchmark against speedup delivered in real training, for the three settings measured both ways. The dashed line is parity, where a benchmark result would transfer exactly. The Tree algorithm has no training value because it crashes the job. Predicted ratios are medians against the socket baseline for the plugin and against the plugin baseline for the buffer setting.
A standalone benchmark measures the sum in isolation, whereas in training the sum runs at the same time as the backward pass. Only the part that fails to hide behind computation shows up in the step time. The plugin’s benchmark advantage was large enough that a fraction of it still showed, at 1.45 rather than the predicted 1.86. Larger internal buffers, which the benchmark rated at 4.49 times, made an already-hidden sum faster and delivered a 1.2 percent loss, which is to say nothing.
Three of the four settings the microbenchmark called wins were null or harmful in production. The lesson is not that benchmarks are useless: the ordering did survive, and ordering is what a cheap harness is for. It is that a benchmark can rank candidates but cannot price them, and the price is what decides whether to adopt.
There is a harder version of this failure, which is that for several days the benchmark had no measurement floor at all.
Figure 3: Reproducibility of each harness used in this work, as the spread across repeated runs of one unchanged configuration. Any reported effect smaller than a harness’s own spread is unreadable by that harness. The two wide harnesses were used to rank settings before their spread was known, which invalidated several early claims.
The all-reduce benchmark varies by 12 percent across six identical runs. Before that was known I had reported a 1.6 percent difference between two memory-registration settings as a ranking. It was noise. The campaign this work belongs to has an explicit rule: measure the run-to-run floor before quoting any difference. I had enforced it all week on the model experiments while exempting the benchmark, on the unexamined grounds that a benchmark is instrumentation rather than an experiment. It is an experiment.
The production slice logs have the same problem, and it is worse there, because slices are expensive and the temptation to read just one is correspondingly stronger. Two slices of the same script at the same shape with the same twenty-four ranks differ by 12.5 percent, larger than any effect this article reports apart from the plugin. No production claim here rests on a single slice.
The in-job control is the harness that works. Running the control arm and the treatment arm inside one allocation, each from a fresh checkpoint directory, reproduced the control to 0.09 percent across two jobs. That precision is what makes a 14.5 percent effect readable, and it is available only because both arms see the same node, the same filesystem contention and the same fabric neighbors.
A catalog of null results
Figure 4: Every communication setting measured inside real training, as a percentage change against its own control in the same job. Regime A uses the production batch shape, 66 gradient accumulation micro-steps per optimizer step; regime B shrinks the batch to 11 micro-steps, which reproduces the collective exposure of a 48-GPU run on two nodes. A micro-step is one forward-and-backward pass; the optimizer updates the weights once per group of them. Five settings are absent from the axis: four produced no throughput number at all, and the 100 MB gradient bucket ran but its control arm did not, so that ladder is not internally controlled and no delta from it is quoted. The bfloat16 result is a numerical change rather than a transport change and has not been validated for quality.
Beyond the plugin, the two settings that produced a clean number produced approximately zero. Internal buffer size and the number of communication channels each landed within a percent or two of the control arm in their own job.
Four settings produced no usable number, for reasons worth separating. Requesting the Tree algorithm alone crashes. Enabling direct memory access between the GPU and the network adapter fails with a mapping error from the fabric driver.
A 400 MB gradient bucket, the chunk size in which gradients are handed to the network, never reached steady state. Its memory occupancy climbed on every step, from 88.4 percent at step two to 92.1 percent at step three. This job holds gradients inside the communication buckets, so a large bucket cap demands many large contiguous regions on a card that is already nearly full.
The 100 MB bucket ran and produced a rate, but the control arm of the same job did not. That ladder therefore has no internal comparison, and I quote no delta from it. The alternative would be to compare it against a figure from a different session, and the slice-to-slice spread established above is twenty-five times larger than the difference in question.
Taken singly these are unremarkable. Taken together they say something specific. At 66 micro-steps per optimizer step, the gradient sum hides so completely behind the backward pass that neither its speed, nor its algorithm, nor the number of separate transfers it is split into is observable in the step time. There is no headroom at that shape to find.
The most informative failure is the one that lost the most. LL128 is a low-latency protocol that trades bandwidth for latency, and it cost 15.5 percent. A configuration that gives up bandwidth to buy latency, and loses, is evidence that the regime is bandwidth-bound rather than latency-bound. That single fact retrospectively explains three earlier null results at once: bucket size did nothing because transfer launch latency was never the constraint, and the Tree algorithm’s fewer network hops bought nothing for the same reason. A negative result used as an instrument is worth more than the positive result it was hunting.
One qualification keeps that from being tidy. If the link were simply starved of bandwidth, raising the number of parallel communication channels should have helped, and it did not. The defensible statement is that the regime is bandwidth-bound at the level of the payload rather than starved of channels.
How much of a step is communication
Every estimate in this campaign of the fraction of step time spent communicating had been an inference. One measurement finally produced a number, from the only intervention that reduced how much data was sent rather than how fast it was sent.
Summing gradients in bfloat16, a 16-bit number format, rather than in 32-bit single precision halves the bytes on the wire. It returned 14.5 percent, not 100 percent. Solving for the communication fraction gives approximately 25 percent of step time in the exposed regime, of which halving the payload recovered half. The arithmetic is elementary; the point is that it is arithmetic on a measurement rather than on an assumption.
This result is not adoptable on throughput evidence alone, which is why it is reported separately from the others. Every other setting in this article is quality-neutral by construction: changing which network carries a sum, or how it is chunked, cannot change the sum. Compressing gradients before summing them changes the arithmetic the model trains on. The comparison that would settle it is a matched pair of runs differing only in this setting, read on training loss rather than on throughput.
That comparison has not produced a valid result. Its first attempt was void, for a reason worth recording because it will recur in anyone’s short probe derived from a long training script. It inherited --warmup_steps 8000 from the production command. Within the roughly twenty-five steps a short wall-clock limit allows, the learning rate therefore never exceeded about 1.9e-6, the weights had barely left their initialization, and both arms reported a loss identical to four decimal places. Two nearly untrained models cannot diverge numerically, whatever precision sums their gradients. Any probe derived from a production configuration must override the production warmup schedule, or it measures initialization.
What does not scale
Figure 5: Per-GPU training throughput against the number of nodes the job spans, from production slice logs at fixed configuration. The 6-node point is shown for both the best and the worst slice of six, a 12.5 percent spread at identical configuration. The second-cluster point is a different machine measured at a single node and is included because it establishes that per-GPU compute throughput is the same on both, so the decline across the series is not hardware.
The tuned two-node configuration reaches 6,212 tokens per second per GPU against 7,171 on a single node. At the six nodes the production job actually runs on, the same configuration reaches 4,626 on its best slice and 4,111 on its worst. The per-GPU cost of spanning nodes therefore grows from roughly 13 percent to roughly 35 percent between two nodes and six. Two-node results can rank candidate settings but cannot size the win, so every number in the sections above should be read as a ranking.
The obvious alternative explanation can be ruled out. The second cluster reaches 7,084 tokens per second per GPU on a single node against the first cluster’s 7,171, a difference of 1.2 percent. Both use the same generation of accelerator, and neither configuration crosses a node. Per-GPU compute is effectively identical on the two machines, so the decline across the series above is communication and not hardware.
One correction belongs here rather than in a footnote, because I quoted the affected number repeatedly before checking it. The two-node figure of roughly 13 percent is confounded. The single-node reference runs 132 micro-steps per optimizer step and the two-node measurement runs 66, because the same global batch is divided among twice as many devices. Halving the micro-step count halves the window available to hide the sum, which is a throughput cost with no network involvement whatsoever. I attributed the entire gap to crossing nodes. The control that separates the two effects needs a single node, so that it contains no cross-node traffic at all, and must vary only the micro-step count by shrinking the global batch. It has not been run.
That same relationship is why two-node testing was viable. The number of micro-steps is the global batch divided by the number of devices, and both terms are chosen by the experimenter rather than imposed by the hardware. Shrinking the batch on two nodes reproduces the micro-step count, and therefore the collective exposure, of a job six times the size. Regime B in the catalog above is exactly that: an eight-GPU job configured to expose its communication the way a forty-eight-GPU job would. Buying twelve nodes to discover that a setting is null is avoidable.
What remains open, and what to take away
Three questions are unresolved. The single-node micro-step control has not been run, so the split between crossing nodes and dividing the batch remains unmeasured. The quality of bfloat16 gradient compression, the only remaining lever with a measured gain, is unvalidated. And the exposed cost at production scale has not been broken down: the candidates that no transport setting would touch include reading training data from a shared filesystem, checkpoint writes, and slow individual ranks holding up each collective.
The transferable conclusions are narrower than the effort behind them, which is the usual result and worth stating plainly.
Check that the library chose the right network before tuning anything about how it uses that network. The one large win here was a wrong default rather than a badly chosen parameter, and it was invisible in every metric except aggregate throughput.
Measure the reproducibility of a harness before quoting a difference it reports. Two of the three harnesses used here have a spread wider than most of the effects they were asked to resolve, and both were used in that state before anyone checked.
Prefer a control in the same job to a remembered number from a previous one. The 0.09 percent in-job reproducibility and the 12.5 percent slice-to-slice spread are the same measurement of the same configuration, differing only in whether the comparison shared an allocation.
Reproduce the regime rather than the scale. A collective that hides completely behind computation can be neither optimized nor measured; one deliberately exposed can be both, and exposing it is a matter of arithmetic on the batch shape rather than of buying nodes.
Finally, these results describe one fabric, one model size and one framework. The plugin defect is likely to generalize wherever a specialized interconnect meets a library that does not recognize it. The catalog of nulls almost certainly does not: a job with fewer accumulation steps, a larger model, or a slower network would expose its communication differently, and the settings that were unreadable here would become readable there. The method is the portable part.