Alibaba Cloud Reveals Innovative Data Center Design for AI and Large Language Model Training

Alibaba Cloud Reveals Innovative Data Center Design for AI and Large Language Model Training

Alibaba Cloud engineer Ennan Zhai has shared a research paper on GitHub, detailing the company's data center design for training large language models (LLMs). Titled “Alibaba HPN: A Data Center Network for Large Language Model Training,” the document outlines how Alibaba utilized Ethernet to enable communication between its 15,000 GPUs.

LLM training generates periodic data bursts that can reach up to 400 Gbps, unlike general cloud computing, which produces consistent but smaller data flows below 10 Gbps. This characteristic makes traditional load-balancing schemes, like Equal-Cost Multi-Path (ECMP), prone to issues such as uneven traffic distribution. To address this, Zhai and his team developed the High-Performance Network (HPN), employing a “2-tier, dual-plane architecture” to minimize ECMP occurrences and optimize network path selection for handling large data flows.

The HPN incorporates dual top-of-rack (ToR) switches, which provide redundancy by backing each other up, thereby mitigating single-point failures. Alibaba Cloud's data centers are divided into hosts, each equipped with eight GPUs. Each GPU has its network interface card (NIC) with two ports, forming a ‘rail.’ The host also includes an additional NIC for backend network connection. Each rail connects to two different ToR switches, ensuring continued operation even if one switch fails.

For inter-host communication, Alibaba Cloud uses Ethernet with dedicated 400 Gbps RDMA network throughput per host, totaling 3.2 Tbps bandwidth. However, intra-host communication still relies on Nvidia’s NVlink due to higher bandwidth needs. Alibaba Cloud uses single-chip ToR switches with 51.2 Tb/sec Ethernet capacity, as multi-chip solutions are less stable and have a higher failure rate. To address overheating issues, the company developed a custom vapor chamber heat sink with enhanced thermal energy transfer efficiency.

Zhai and his team will present their work at the SIGCOMM conference in Sydney, Australia, this August. The project has garnered interest from companies like AMD, Intel, Google, and Microsoft, who are working on Ultra Accelerator Link, an open-standard interconnect to rival NVlink. Alibaba Cloud has already implemented the HPN for over eight months, demonstrating its effectiveness.

Despite its advantages, the HPN's complex wiring structure is a notable drawback, as each host has nine NICs connected to two different ToR switches, increasing the risk of wiring errors. However, this technology is likely more affordable than NVlink, potentially allowing data centers to save on setup costs and avoid using Nvidia technology, especially for companies affected by U.S. sanctions in the ongoing chip war with China.