is easy. You load the weights, load the information, and watch for it to complete. For many fashions, that’s high-quality, and the one actual price is time. When the wait will get too lengthy, the same old repair is so as to add GPUs. Every one trains on a unique slice of the information in parallel, and the work finishes quicker. Nothing in regards to the mannequin state modifications; you’ve simply thrown extra arms at it.
That covers the widespread case, the place the issue is time. Giant fashions add a second downside that extra arms can’t resolve: house. Over the previous 12 months, a significant a part of my work has concerned coaching and fine-tuning giant basis fashions throughout completely different domains, together with single-cell fashions and vision-and-language transformer fashions. When you get previous a number of billion parameters, the mannequin, its gradients, and its optimizer states now not match on a single GPU. You’ll be able to’t simply replicate the work throughout GPUs as a result of a duplicate doesn’t match on any of them. You need to cut up the mannequin itself.
These two issues map onto two methods. DDP (Distributed Knowledge Parallel) assaults time. Each GPU retains a full copy of the mannequin and trains on a unique slice of the batch. FSDP (Absolutely Sharded Knowledge Parallel) assaults house. The mannequin is cut up into items in order that no GPU has to carry the whole mannequin.
There aren’t actually simply two choices, although. These are simply the 2 ends of 1 dial. In between these sit the ZeRO phases from DeepSpeed (Microsoft’s coaching library), which allow you to slide from one finish to the opposite, step-by-step, buying and selling a little bit reminiscence for a little bit communication at every cease quite than leaping straight from “everyone holds all the things” to “no person does.”
The place to set that dial and what it truly controls is roofed within the first a part of the article. However there’s a second factor that decides how properly any of those methods work. Every considered one of them works by shifting knowledge between GPUs. And how briskly that knowledge strikes will depend on the bodily wires connecting them. This was one thing I didn’t take into consideration till the identical code, on the identical sort of GPUs, ran a number of occasions slower on one node than on one other. So within the second half, we’ll talk about the {hardware} beneath it, and the way the wiring must also be one thing you think about. (For people who are usually not a lot into {hardware}, belief me, it’s not that dangerous!)
One be aware earlier than we begin. Every part right here is measured, not simply concept. The technique comparisons ran on 4 A100 80GB GPUs; the {hardware} experiments ran on H200 machines, together with two which have the very same GPUs wired collectively in numerous methods, which seems to be the entire level.
Half 1: The Software program: What Every GPU Holds
Each distributed coaching technique is actually a solution to 1 query: what does every GPU hold its personal copy of? For that to make sense, let’s have a look at an instance.
Take Mistral-7B and fine-tune it with Adam in BF16 blended precision. The parameters take 14 GB: 7 billion of them, two bytes every. The gradients take one other 14 GB. Then now we have the optimizer states, that are the most costly half. Adam retains two working averages for each parameter, momentum and variance, each saved in FP32, which works out to roughly 58 GB. Add it up, and we attain 87 GB earlier than the mannequin has seen a single batch or a single activation has been computed.
For an A100 GPU that has 80 GB of VRAM, the mannequin doesn’t match. So we distribute. And the query, as talked about at the start of this part, is: what does every GPU maintain?
Distributed Knowledge Parallel (DDP): Everybody Holds Every part
The straightforward reply is that each GPU holds all of it. That is DDP, and it’s the one most individuals first encounter (which was the case for me as properly). Every GPU retains an entire copy of the mannequin: all of the parameters, all of the gradients, all of the optimizer states. What will get divided is the information. Every GPU takes a unique slice of the batch, runs its personal ahead and backward move, and computes gradients from its personal slice.
The catch is that these gradients are all completely different as a result of every GPU noticed completely different knowledge. If each GPU simply stepped its optimizer now, the copies would drift aside, and also you’d now not have one mannequin, you’d have 4. So earlier than stepping, the GPUs common their gradients. Each GPU sends its gradients out, receives everybody else’s, they usually all find yourself with the identical averaged gradient. They then take an equivalent optimizer step and the copies keep in sync. That averaging is the one piece of communication DDP wants, and it occurs as soon as per step. It’s known as all-reduce, and we’ll come again to it later once we begin counting what every technique prices the wire.
DDP is quick and easy exactly as a result of it communicates so little. One all-reduce per step, and PyTorch overlaps even that with the backward move, so on good {hardware}, you barely discover it. However its limitation is constructed into the design. Each GPU holds all the things, which suggests including GPUs does nothing for the reminiscence downside. If 87 GB doesn’t match on one A100, it doesn’t match on 4 both, as a result of every of the 4 remains to be holding the complete 87 GB. Including GPUs to DDP buys you throughput, but it surely doesn’t cut back the reminiscence every GPU has to carry.
Absolutely Sharded Knowledge Parallel (FSDP): No person Holds Every part
If the issue is that each GPU holds all the things, the apparent repair is to verify no GPU does. That is FSDP. As an alternative of every GPU maintaining a full copy, the mannequin is cut up into items, and every GPU owns only one piece of the parameters, one piece of the gradients, and one piece of the optimizer states. On 4 GPUs, each holds roughly 1 / 4 of the 87 GB, which permits the mannequin to suit.
However this raises a right away downside. You’ll be able to’t run a layer by a GPU that solely has 1 / 4 of that layer’s weights. So FSDP reassembles every layer simply earlier than it’s wanted and takes it aside once more proper after. When the ahead move reaches a layer, the GPUs commerce their items so that each GPU briefly has that layer’s full weights, the layer runs, after which every GPU throws away the items it doesn’t personal. The subsequent layer comes up and the identical factor occurs once more. The complete mannequin is rarely resident on any single GPU; just one layer is, and solely for the immediate it’s computing. The identical factor occurs in reverse in the course of the backward move.
That buying and selling has names, the identical manner DDP’s averaging did. Gathering everybody’s items right into a full layer is an all-gather, and scattering the summed gradients again to their house owners is a reduce-scatter. Take into account this distinction. DDP communicates as soon as per step. FSDP communicates consistently. An all-gather earlier than each layer within the ahead move, one other earlier than each layer within the backward move, and a reduce-scatter after. That’s principally the commerce that you simply do when switching between them. FSDP buys reminiscence and pays for it in communication.
One clarification issues right here, as a result of it could possibly be sort of complicated. FSDP remains to be knowledge parallelism. Each GPU runs the whole mannequin by itself slice of the batch, precisely like DDP. The one distinction is that the weights are saved in items and reassembled on demand as a substitute of being saved as a full copy. Every GPU nonetheless computes the entire ahead and backward move from begin to end.
That is completely different from mannequin parallelism, the place the mannequin itself is cut up throughout GPUs, and each is accountable for solely a part of the computation, one set of layers, or one slice of each layer. On this paradigm, a single ahead move has to hop between GPUs as a result of no single one can end it alone. So as to add to this, even inside mannequin parallelism, now we have layer and tensor parallelism, which dictate whether or not we divide the mannequin vertically or horizontally among the many layers. However earlier than we digress an excessive amount of, these are completely different strategies with completely different communication patterns, and it’s not what this text is about. Every part right here, DDP, FSDP, and the ZeRO phases in between, is knowledge parallel. Similar computation on each GPU, simply with completely different knowledge. Solely the storage of the mannequin differs from one technique to the following.
Coming again to it, now that we’ve mentioned the idea, let’s see what occurs in observe. To see how these two methods evaluate, I ran experiments on two fashions. DINOv2 (a imaginative and prescient basis mannequin) and Mistral-7B (a big language mannequin). Each experiments ran on 4 A100 80GB GPUs on a single node, utilizing HuggingFace Speed up with BF16 blended precision. The one factor that modified between runs was the technique.
DINOv2 on Meals-101. The primary experiment fine-tunes DINOv2 on Meals-101, a classification dataset with 101 meals classes and about 101k photographs. The duty is to foretell a picture class given a picture. I examined each the bottom (86M parameters) and huge (304M parameters) backbones, utilizing a batch dimension of 32 per GPU for the bottom mannequin and 256 per GPU for the big mannequin. This can be a case the place each methods work for the reason that mannequin suits comfortably in GPU reminiscence. However even right here, we are able to see the reminiscence distinction between the 2 approaches.
FSDP makes use of considerably much less reminiscence per GPU: 0.39 GB vs. 1.79 GB for the bottom mannequin, and 1.39 GB vs. 6.26 GB for the big mannequin. That’s roughly 4.5x much less reminiscence with FSDP. This issues much less when you have got loads of headroom, but it surely exhibits us why FSDP shall be important for bigger fashions. So we noticed the reminiscence, what in regards to the velocity?
We are able to see that DDP is constantly quicker, round 6% greater throughput for the bottom mannequin and 4% for the big one. That is anticipated since DDP has much less communication overhead. It simply has one all-reduce per step, towards FSDP’s gather-and-discard at each layer. And once we see the coaching time:
The takeaway right here is that when the mannequin suits in reminiscence, DDP is the higher selection. It’s less complicated and quicker. However discover the reminiscence hole. What occurs once we scale as much as a mannequin the place that hole truly issues?
Mistral-7B on Alpaca. For the second half, I attempted fine-tuning the Mistral mannequin on the Alpaca instruction-following dataset. With DDP, the job crashed instantly with an out-of-memory error. With FSDP, it ran with out a problem.
DDP would want an estimated 87 GB per GPU for the mannequin state (parameters, gradients, optimizers), which exceeds the 80 GB accessible on the A100. FSDP shards this throughout 4 GPUs, bringing the estimated requirement all the way down to about 22 GB per GPU. The precise peak in the course of the run was about 29 GB, which incorporates activation and different overhead. This is the reason FSDP exists. It makes coaching doable when DDP merely can’t match the mannequin.
In order that’s the rule I used to cease at. Does the mannequin, with its gradients and optimizer states, match on one GPU? If sure, use DDP. If no, use FSDP. It’s a great rule. However there are two assumptions hiding inside it. The primary is that DDP and FSDP are the one two settings. To start with, I already hinted that they aren’t. There’s a dial between them, and within the subsequent half, we’ll talk about these phases, the place you shard among the mannequin state however not all of it.
The second assumption is that we’re treating all machines the identical manner. These methods don’t behave the identical all over the place. The identical technique on the identical mannequin can run a number of occasions quicker or slower relying on how the GPUs are wired collectively. That’s what the second half of the article is about. First, although, the dial.
ZeRO: The Dial Between DDP and FSDP
To see the stops on the dial, begin with a small truth in regards to the operations from Half 1. DDP’s all-reduce, the one which averages gradients throughout GPUs, is definitely two less complicated operations glued collectively. First is reduce-scatter, which sums everybody’s gradients however offers every GPU solely its slice of the end result as a substitute of giving everybody the complete end result. Then an all-gather, which collects these slices again right into a full copy on each GPU. So all-reduce is principally reduce-scatter plus all-gather.
These are the identical two operations that FSDP makes use of. Now that we all know that the operations are shared, it is smart that the quantity of sharding could possibly be in phases. The dial is then only a query of how a lot you’re prepared to maintain sharded between steps as a substitute of reassembling.
To make that concrete, have a look at what every GPU shops per parameter. In mixed-precision Adam, there are three issues: the parameters, the gradients, and the optimizer states, with the optimizer states being by far the heaviest. DDP retains full copies of all three on each GPU. The ZeRO phases, from DeepSpeed, peel them off one by one, heaviest first.
ZeRO-1 shards solely the optimizer states. Every GPU nonetheless holds the complete parameters and full gradients, but it surely retains solely its slice of the optimizer states. Since these are the heaviest objects, this offers the most important reminiscence win, and it prices little or no in further communication, as a result of nothing you want in the course of the ahead and backward move has moved.
ZeRO-2 provides the gradients to the shards. Now every GPU retains solely its slice of the gradients and its slice of the optimizer states, however nonetheless a full copy of the parameters. Communication remains to be near DDP’s, for the reason that parameters (the factor you truly compute with) are all nonetheless native.
ZeRO-3 provides the parameters as properly. Now, no GPU holds the complete mannequin, so earlier than a layer can run, its parameters must be gathered on the fly, used, and discarded. ZeRO-3 and FSDP implement the identical fundamental concept. One is constructed by DeepSpeed, and the opposite is native to PyTorch. So, since that is the place you cease maintaining the parameters native, additionally, you will have the additional communication we mentioned earlier.
The next diagram exhibits the dial from full replication to full sharding:
Two issues within the determine are value pausing on. The primary is the reminiscence column on the appropriate. These are the measured peaks for Mistral-7B on 4 GPUs, they usually fall in a clear sample as you progress down the dial. 87 GB at DDP, then 55, 51, 40, and 37 GB. However discover it by no means drops to 1 / 4 of 87 GB, which is what you’d anticipate if 4 GPUs every held 1 / 4 of all the things. The ground is usually activations and runtime overhead. Activations are nonetheless produced independently on every GPU, so sharding the mannequin state doesn’t make them disappear.
Second, each step down that dial buys reminiscence by spending communication. ZeRO-1 spends virtually none. ZeRO-3 and FSDP spend probably the most. That’s the software program view. DDP, ZeRO, and FSDP are completely different solutions to the identical query: what ought to every GPU hold domestically, and what ought to it fetch from the others? Transferring towards sharding buys reminiscence, but it surely additionally creates extra communication.
To date, now we have solely counted how a lot knowledge has to maneuver. We now have not requested how costly that motion is. That will depend on the {hardware}. Extra particularly, it will depend on the material connecting the GPUs.
Half 2: The {Hardware}: How Quick the GPUs Can Discuss
The Wires Between the GPUs
Each technique in Half 1 had the identical hidden price line: communication. DDP pays it as soon as per step. FSDP pays it at each layer. ZeRO enables you to select how a lot to pay. However communication isn’t an summary price. It’s knowledge shifting between chips, and the trail it takes issues.
On this article, I’m solely communication inside a single node. As soon as coaching spans a number of servers, one other layer seems. GPUs now have to speak throughout InfiniBand or Ethernet, which brings a unique set of bottlenecks. That issues loads as properly, but it surely’s a separate article.
Inside one node, when GPUs must trade knowledge, how do they do it? Effectively, there are completely different roads they’ll take, and the velocity between them can differ by as much as 10×. The identical FSDP code can run at full velocity on one machine and crawl on one other with the very same GPUs, purely due to how these GPUs are linked to one another.
There are two fundamental paths GPU site visitors can take inside a server: PCIe and NVLink. PCIe is the default system path by the motherboard. NVLink is NVIDIA’s direct GPU-to-GPU interconnect.
However NVLink is simply the hyperlink. The topology issues simply as a lot. On the machines I examined, NVLink appeared in two completely different preparations. NVL bridge teams and NVSwitch. That provides us 4 phrases to maintain straight. PCIe, NVLink, NVL, and NVSwitch. They’re associated, however they aren’t the identical sort of factor. PCIe and NVLink are the paths. NVL and NVSwitch are methods of arranging the quick NVLink path throughout a number of GPUs.
PCIe: the default connection
PCIe (PCI Specific) is the usual high-speed bus that connects parts to a pc’s motherboard, the slots your graphics card, SSD, and community all plug into. Each GPU sits on it by default.
When two GPUs must trade knowledge over PCIe, it goes by the system: out of 1 GPU, throughout the shared bus, usually by the CPU and host reminiscence, and into the opposite. It really works between any two units, but it surely’s the slowest possibility, and each gadget on the bus is contending for a similar lanes. A PCIe Gen5 x16 slot strikes someplace round 64 GB/s in a single route.
NVLink: a direct GPU-to-GPU connection
NVLink is NVIDIA’s devoted interconnect. Hyperlinks that run from one GPU to a different with out going by the CPU or system reminiscence. On an H100 or H200, every GPU has 18 NVLink connections, which collectively come to roughly 450 GB/s in every route, about 7x what a single PCIe slot supplies.
However NVLink is simply the hyperlink. What issues simply as a lot is how these hyperlinks are organized throughout the GPUs in a server. Two machines can each have H200 GPUs and nonetheless behave very in a different way if one makes use of NVLink by bridges and the opposite makes use of NVSwitch.
NVL: hyperlinks inside teams, PCIe between them
The cheaper association connects NVLink solely inside small teams of GPUs, utilizing bodily bridges between playing cards. NVIDIA sells these as “NVL” playing cards, just like the H200 NVL. On the machines I examined, eight playing cards have been bridged into two teams of 4. Inside a bunch, each pair of GPUs has a quick NVLink bridge. Between teams, there’s no NVLink in any respect, and site visitors falls again to PCIe.
So the material is uneven. Some pairs of GPUs discuss over NVLink, others discuss over PCIe. And so which pair you get will depend on which bodily GPUs your job lands on, which is determined by the cluster scheduler. On an NVL machine, placement is essential as it could possibly have an effect on efficiency.
NVSwitch: each GPU linked to each different
The premium association places a change within the center. Devoted NVSwitch chips join each GPU to each different GPU at full NVLink velocity, with no quick pairs and no gradual pairs. Each GPU is one hop away from each different. That is what’s on NVIDIA’s HGX baseboards, the SXM type issue, and it’s the structure that giant coaching runs use.
To get a way of how far this concept scales, at Computex 2025, Jensen Huang confirmed off what NVIDIA calls an NVLink Backbone, a column of round 5000 cables wiring 72 GPUs right into a single all-to-all cloth, and claimed it strikes about 130 TB/s. That is greater than the height site visitors of the whole web. Take the comparability as you’ll, however the underlying level is actual. An NVSwitch cloth is a crossbar the place each GPU reaches each different at full velocity, and it scales to a complete rack. The machine I examined is a a lot smaller model of the identical concept. It has 8 GPUs as a substitute of the 72, however wired on the identical precept.
Similar GPU, two completely different machines
It’s value clearing up a confusion that may occur. After we say “H200,” we’re principally naming the GPU. H200 methods might be constructed in numerous type elements and materials: as SXM GPUs on an NVSwitch baseboard, or as NVL playing cards linked by bridges. Each are H200-class GPUs with 141 GB of reminiscence, however they aren’t the identical system. Energy limits, board design, and cooling can differ. However the level right here is that you simply can not infer the GPU-to-GPU cloth from the GPU mannequin identify alone. You need to have a look at the topology. Listed here are the 2 machines used for the experiments under. You’ll be able to verify the wiring in your node with nvidia-smi topo -m.
On the SXM node, each pair reads NV18. That is uniform, full velocity, each GPU is equal. On the NVL node, the identical H200 playing cards cut up into two teams of 4, quick NVLink inside a bunch and a plain PCIe throughout. A job that lands on 4 GPUs inside one group runs close to full velocity. A job scattered throughout teams drops all the way down to PCIe. The subsequent part is about precisely what that prices.
Measuring the Wires
We now have two sorts of measurements right here. The primary is a microbenchmark that occasions the precise collective operations the methods depend upon. All-reduce for DDP, all-gather and reduce-scatter for FSDP, and stories the bandwidth each achieves. The second is end-to-end coaching, the identical Mistral-7B fine-tune from Half 1, now timed for steady-state throughput, so we are able to see whether or not the bandwidth variations truly present up in coaching. Every part under ran on the 2 machines described above.
Let’s begin with the cleanest doable comparability. Take two H200 SXM GPUs and run the identical collectives twice. For the primary run, allow them to use NVLink, and for the second, drive them onto PCIe as a substitute. Similar GPUs, similar code, solely the wire modifications.
The wire alone is value about 10-11x. This is identical silicon and the identical operation, the place the one distinction was the bodily connection the information travels over.
Subsequent, I measured how all-reduce bandwidth modifications as you add GPUs to the job. For this, I needed to management which bodily GPUs every run used, so I might hold a job inside a single NVL group or drive it to cross between teams.
We are able to see three behaviors right here. NVSwitch stays flat and quick. 330 GB/s at two GPUs, rising gently to 467 at eight, and it by no means issues which bodily GPUs the job will get. Each pair is NV18, so the material seems the identical regardless of how the scheduler fills it. That is the habits giant coaching runs assume.
The NVL quad is a shock, and it comes all the way down to how NVLink hyperlinks are shared. Every of those GPUs has 18 NVLink hyperlinks to distribute amongst its neighbors. Inside a quad of 4, each GPU has three neighbors and offers six hyperlinks to every one, which is the NV6 you’d see within the topology matrix. Six hyperlinks per pair, eighteen hyperlinks whole per GPU.
That sharing is why job dimension issues a lot. A job utilizing solely two GPUs of the quad talks throughout simply the six hyperlinks connecting that one pair. The opposite 12 hyperlinks on every card go to GPUs that aren’t within the job, so that they sit idle, and the pair runs at about 114 GB/s, which is a 3rd of the cardboard’s NVLink capability. Add the opposite two GPUs, and each card now talks to all three of its neighbors without delay, lighting up all 18 hyperlinks. That’s the identical variety of hyperlinks an SXM module makes use of, which is why a full quad reaches about 322 GB/s, almost matching NVSwitch.
So the NVL quad was by no means gradual. It was simply underused. A small job leaves most of its hyperlinks idle. After we fill the quad, we fill the hyperlinks.
The third line is the cross quad case, and it by no means leaves PCIe territory. 28 GB/s at two GPUs, 35 at 4, 39 at eight. Including GPUs doesn’t assist as a result of the issue isn’t what number of hyperlinks are lively; it’s that the job has to cross between quads, and there’s no NVLink there in any respect. A collective is simply as quick as its slowest required hop, so a single PCIe hyperlink between the quads drags the whole all-reduce all the way down to PCIe velocity, regardless of how briskly each different hyperlink runs. And since a quad is simply 4 GPUs broad, any job bigger than 4 on this machine is compelled to cross. Eight GPUs can’t keep away from it.
The clearest view is the 4-GPU level on the determine. The identical 4 playing cards, in the identical node, run at 322 GB/s inside one quad and about 35 GB/s throughout two, a roughly 9× distinction determined by nothing however which GPUs the job landed on. On an NVL machine, placement actually does make a distinction.
What it does to actual coaching
The bandwidth numbers show the wire issues in a microbenchmark. I additionally needed to see if it exhibits up in actual coaching. So I ran the identical Mistral-7B fine-tune from Half 1 on 4 GPUs, 3 ways: on NVSwitch, on a full NVL quad, and on an NVL job intentionally spanning two quads.
Have a look at the 2 quick materials first, NVSwitch and the complete NVL quad. They’re almost equivalent for each technique. The playing cards have been by no means the bottleneck, it was the location. (If FSDP seems prefer it edges out DDP contained in the quad, don’t learn an excessive amount of into it. Repeat runs put the noise at a number of %, so these are a tie. The trustworthy abstract is that, on both quick cloth, DDP and FSDP are even.)
The across-quads case is the price of getting placement mistaken. Throughput drops by 3 to 5x. That’s a smaller hit than ~10x bandwidth drop from determine 9, and the hole is value understanding. Coaching isn’t pure communication. Every GPU remains to be doing actual compute that doesn’t care in regards to the wire, and that compute partly hides the gradual communication behind it. So the wire’s full penalty will get considerably diluted. However 3 to 5x remains to be a really significant distinction, particularly for lengthy jobs.
And the slowdown isn’t even throughout the methods. The sample is strictly what Half 1 predicts. FSDP falls the toughest, as a result of it communicates each layer, and on a gradual wire, these per-layer transfers can’t cover behind compute. DDP holds up finest as a result of it communicates simply as soon as per step. When the wire is quick, sharding is sort of free, so FSDP retains tempo with DDP. When the wire is gradual, each further little bit of communication is uncovered, so the extra a method shards, the extra it loses.
Put all of it collectively. The entire dial, on three wires
Earlier, we mentioned that DDP and FSDP are two ends of 1 dial, with ZeRO-1, -2, and -3 because the stops between them. That was a diagram, and now I can measure it. Right here’s the identical Mistral-7B job on 4 GPUs, run at each cease on the dial, on a quick wire and on gradual ones.
Begin with the reminiscence panel on the appropriate as a result of it does precisely what the idea promised. Transferring down the dial, peak reminiscence falls in a clear sample. 87 GB at DDP, then 55 at ZeRO-1, 51 at ZeRO-2, and 40 at ZeRO-3. Every cease shards another factor, and each buys headroom. FSDP sits on the backside at 37 GB, however discover it’s not a giant step down from ZeRO-3’s 40, as a result of FSDP and ZeRO-3 shard precisely the identical issues. As I discussed above, they shard the identical issues, however the frameworks implement and schedule issues in a different way, which explains the small hole that we see. This staircase is the half that at all times works, no matter {hardware}. In case your downside is becoming the mannequin, the dial delivers, predictably.
The throughput panel is fascinating as a result of the dial behaves like two various things relying on the wire.
On the quick wire, the center of the dial loses. ZeRO-1 and ZeRO-2 surrender round 20% of DDP’s throughput, and ZeRO-3 offers up extra. However FSDP, sharding the identical issues as ZeRO-3, claws almost all of that throughput again whereas utilizing about the identical reminiscence. That’s fascinating as a result of the final two are implementations of the equivalent algorithm, and land far aside on a quick wire.
The distinction is implementation, not technique. The 2 frameworks schedule and overlap their communication in a different way, and on a quick wire, the place communication is affordable, that overhead is the principle factor you’re measuring. (That is Hugging Face speed up’s default DeepSpeed config, with no tuning. A tuned setup would seemingly slim the hole; the purpose is what the defaults provide you with, not that one framework is inherently slower.)
On a gradual wire, the dial flattens. Every part collapses towards the identical low throughput, and the variations between phases principally wash out. The costly factor on a gradual wire is communication itself, and as soon as that dominates, the query of which precise tensors you shard barely strikes the quantity. On this case, the wire is setting the tempo, not the technique.
Placing all the things above collectively, we are able to now make a extra knowledgeable choice.
If you happen to’re on NVSwitch (SXM), cease fascinated with the wire. Each GPU is equally quick, and any GPUs the scheduler offers you’re pretty much as good as another. Use DDP if the mannequin suits, FSDP if it doesn’t. And since FSDP matches DDP’s velocity right here whereas utilizing far much less reminiscence, it’s an affordable default even when the mannequin does match.
If you happen to’re on NVL and your job suits inside one bridged group, pin it there, and also you get near-NVSwitch velocity. On the machines I examined, meaning 4 GPUs, and also you management the location with CUDA_VISIBLE_DEVICES. FSDP is a powerful default on this case. It will get you virtually DDP-class velocity at roughly half the reminiscence.
If you happen to’re on NVL and your job has to span teams, anticipate a substantial slowdown, worst for the communication-heavy methods. Don’t anticipate a milder ZeRO stage to win the velocity again both, as a result of the dial flattens on a gradual wire. If the mannequin suits replicated, plain DDP is the least-bad possibility. If it doesn’t, decide the stage that matches in reminiscence and simply settle for that you simply’re going to pay the wire’s worth.
Conclusion
We’ve lined loads of floor. Within the first half, we seemed on the completely different methods. DDP, ZeRO, and FSDP. Every of them solutions the identical query another way: what ought to every GPU retailer domestically, and what ought to it fetch from the others? DDP retains all the things replicated. FSDP shards all the things. ZeRO offers you the stops in between.
Within the second half, we seemed on the {hardware} layer, the material these requests journey over. On a quick cloth, sharding might be low-cost sufficient that FSDP retains tempo with DDP whereas utilizing a lot much less reminiscence. On a gradual cloth, the identical further communication turns into seen, and may gradual issues down considerably.
That’s the primary lesson. The technique decides how a lot knowledge strikes, however the wire decides how costly that motion is. So earlier than launching a distributed job, run one command:
nvidia-smi topo -m
It tells you whether or not your GPUs are linked by NVLink, NVSwitch, or a slower host path.
Thanks for studying, and I hope you discovered this useful!
References and Additional Studying
