launched Gaudi accelerators to Amazon’s EC2 DL1 situations, we confronted a problem that threatened the whole deployment. The efficiency numbers weren’t simply disappointing; they have been disastrous. Fashions that required coaching successfully have been seeing as much as 50% of their efficiency degradation when scaling throughout a number of nodes. The issue? A community topology that routed all bytes of knowledge by host reminiscence, inflicting a bottleneck that undermined every thing Gaudi was designed to do.
I led the engineering effort to handle this concern, which finally resulted within the growth of what we now name Peer Direct. It’s a characteristic that remodeled the best way Gaudi accelerators talk in cloud environments, and its historical past has some helpful classes on distributed AI coaching at scale.
The Downside with Host NICs
Gaudi was designed with the NIC (Community Interface Card) being embedded straight within the silicon. Every chip has ten community interfaces that may deal with 100 Gbps and help RDMA with RoCE v2, permitting units to entry one another’s reminiscence straight without having the CPU or This structure is extremely environment friendly for AI coaching workloads, the place collective operations like AllReduce have to accumulate gradients from dozens or a whole bunch of units per coaching iteration.
However cloud deployments aren’t all the time compliant with excellent architectures. When Amazon examined Gaudi for DL1 situations, they needed to utilise bizarre host NICs somewhat than Gaudi’s built-in networking. The explanations have been pragmatic: price financial savings and the logistics of working round present information centre infrastructure to accommodate a brand new community topology. From their enterprise perspective, leveraging established community infrastructure made excellent sense.
From the efficiency perspective, it was a catastrophe. As a substitute of peer-to-peer RDMA transfers between Gaudi playing cards, all communication went the good distance round. Knowledge needed to be duplicated out of Gaudi’s high-bandwidth reminiscence into host DRAM, processed by the host CPU, despatched out the host NIC over TCP/IP, acquired by the far host, and duplicated again into the far Gaudi’s reminiscence. All of the added hops induced latency, stole CPU cycles, and added bandwidth restrictions that fully ruined the scalability of distributed coaching.
The efficiency shortfall was so dangerous that one questioned whether or not deployment would ever be value it in any respect. This wasn’t a matter of some trivial optimisation; it was an existential menace to the whole association with AWS.
Why Efficiency Issues This A lot
It’s value realizing why a 50% lack of efficiency is so disastrous within the life of coaching fashions, and particularly massive fashions reminiscent of GPT-5. It now takes weeks or months to coach large language fashions even on humongous clusters. In case you are messing round with fashions which have billions or trillions of parameters, each proportion level of efficiency interprets straight into time and {dollars}.
Take into account the economics. If it takes 30 days to coach a mannequin versus 15, you’re not solely ready longer; you’re paying for double the compute time. At cloud scale, with a whole bunch or 1000’s of accelerators in steady use, this provides as much as hundreds of thousands of {dollars}. Worse, it halves your iteration velocity. In an aggressive AI world the place corporations are racing to develop improved fashions, doubling the variety of checks throughout the similar time-frame may be the excellence between being in entrance and being behind.
Environmental price can be essential. Giant fashions require plenty of electrical energy to show. Higher efficiency means much less compute time, which halves vitality consumption and carbon emissions. As extra stress is mounted on the AI business to chop its carbon footprint, features in effectivity are now not a luxurious however somewhat a necessity.
The answer we designed, Peer Direct, delivered RDMA-like efficiency when the bodily community structure wasn’t appropriate for regular RDMA. We wanted direct reminiscence entry between Gaudi units on completely different techniques with out traversing host reminiscence, however on host NICs that weren’t designed for this within the first place.
The enabler was AWS Elastic Cloth Adapter, a high-performance community interface for HPC and AI workloads on EC2. EFA offers low-latency OS-bypass communications, usually sub-10 microsecond latency. EFA offers RDMA-like semantics utilizing libfabric, an in-user-space communication library offering a typical interface throughout a number of networking applied sciences.
The duty was to mix libfabric with Habana’s Collective Communication Library, HCCL, which handles all distributed coaching workloads. HCCL was constructed on the idea of native RDMA utilizing Gaudi’s on-chip NICs. We wanted to create a bridge enabling HCCL to leverage libfabric transparently for communications with out compromising its efficiency ensures and communication semantics.
The answer wanted a number of technical advances. First, we launched a reminiscence registration system that allowed libfabric to straight entry Gaudi’s high-bandwidth reminiscence. We utilised the Linux kernel DMA-BUF framework, which offers a shared mechanism for sharing machine driver buffers. When HCCL must switch information, the Gaudi driver offers a DMA-BUF file descriptor for the reminiscence area, which libfabric can utilise to create RDMA transfers straight from machine reminiscence.
Second, we included an LRU cache for reminiscence registrations. Reminiscence registration is dear; it includes kernel calls and setup operations that may trigger vital overhead. By caching the mapping of reminiscence addresses to their libfabric handles, we might reuse registrations in hot-access areas, eliminating most registration overhead from precise coaching.
The end result was a communication pipeline that seemed one thing like this: HCCL calls the OFI wrapper, which calls the cached libfabric deal with to carry out an RDMA switch straight from supply Gaudi reminiscence to vacation spot Gaudi reminiscence, with neither CPU ever being referred to as. The OFI wrapper was launched to maintain the codebase clear and keep away from direct header inclusions — it’s a light-weight library that dynamically hyperlinks to HCCL and allows using libfabric with out requiring direct integration
After the switch is full, libfabric stories by a completion queue, and HCCL continues computation with the not too long ago acquired information.
The Improvement Expertise
Constructing Peer Direct concerned venturing into new territory on tight schedules. Libfabric wasn’t but mainstream within the discipline of AI accelerators but. There wasn’t plenty of public documentation accessible, and dialogue was meagre. There was extra of an emphasis on diving into libfabric supply code and reverse-engineering based mostly on experimentation.
The communication with AWS engineers was paramount however time-zone constrained. Working with a crew twelve hours forward meant that debug iterations had 24-hour turnarounds. Each concern wanted cautious documentation and correct communication, as real-time collaboration was not attainable.
The stakes have been excessive because the total DL1 deployment was using on this performance working. Delays would have thwarted a serious product launch. No person on our crew had deep background information of libfabric internals, so we have been studying a fancy codebase whereas designing a crucial integration concurrently.
The Outcomes
After we truly deployed Peer Direct, the velocity enhancements have been all the hassle was value. We noticed a 1.5 to 2x throughput improve for collective operations on a 32MB message measurement. On bigger messages, the efficiency was much more astounding, with as much as 1.76x higher throughput at a 256MB message measurement. CPU overhead created a bottleneck that fully disappeared.
Most importantly, these microbenchmark enhancements straight translated into actual mannequin coaching efficiency. Coaching Habana’s DeepSpeed BERT mannequin with 5 billion parameters throughout 128 Gaudi units, we noticed substantial throughput achieve. Fashions utilizing extra aggressive reminiscence optimisation strategies, like ZeRO-2, that are extra collective operation dependent, benefited disproportionately from Peer Direct.
PeerDirect was one of many primary enablers for Gaudi efficiency on AWS DL1 situations, permitting high-scale distributed coaching to run effortlessly on the launch day. Past this preliminary influence, the hassle set the groundwork for future high-performance communication options and proved that cloud-native AI accelerators might stay aggressive regardless of the constraints of cloud infrastructure.
The expertise jogged my memory of an necessary lesson in techniques engineering: typically crucial efficiency enhancements don’t end result from optimising the quick path, however from sidestepping unjustified detours altogether. Throughout distributed AI coaching, having information journey straight throughout accelerators with no pointless copies and no CPU intervention is what makes a working system versus one which scales.
Key takeaways? One necessary “takeaway” from this venture is that assumptions about community topology must be examined on the earliest level within the distributed coaching course of. As most of the accelerator stacks have been constructed based mostly on an idealised surroundings, they don’t keep in mind the extra hops, translation layers, and/or cost-driven components that exist within the cloud environments. Subsequently, earlier than specializing in optimising both mannequin stage or kernel stage, engineers ought to carry out easy collective microbenchmarking throughout the specified topology. If scaling effectivity dramatically decreases with growing node counts or message sizes, the doubtless motive is the info path, not the kernel. By figuring out the host-memory detour early on, engineers can focus their efforts the place they may have the best influence.
One other necessary lesson discovered was the necessity to deal with each reminiscence registration and information switch as first-class efficiency issues. Reminiscence registration overhead can vastly exceed the time spent speaking if every information switch requires a brand new registration. The LRU cache for registered recollections was a non-glamorous addition to HCCL; nonetheless, it successfully eradicated a systemic supply of latency and made the RDMA path viable for real-world workloads. When growing distributed techniques, engineers ought to profile not solely the accessible community bandwidth but additionally the lifecycle prices related to allocating buffers, registering them, and tearing down these registrations. Small adjustments to those management paths can lead to massive will increase in end-to-end throughputs.
Lastly, the mixing methodology used on this venture offers a sample for integration. As a substitute of rewriting HCCL to make use of libfabric straight, we created a skinny abstraction layer that maintained present semantics whereas changing the underlying transport layer. This offered a number of advantages, together with minimising threat, lowering code churn, and permitting incremental testing. Groups going through an identical problem (i.e., adapting accelerator-native communication libraries to cloud-native materials) ought to try and isolate the transport layer, keep collective semantics, and create small, testable interfaces between the 2. This not solely permits for quicker growth but additionally permits for easier help of future transport backends.
Disclosure: I work as an AI Runtime Group Supervisor at Intel. The views shared on this article are my very own.
