Distributed Training & Parallelism

Distributed Training & Parallelism

What it is

Distributed training is the process of splitting a machine learning model or its training data across multiple GPUs or compute nodes. Because modern Large Language Models (LLMs) often exceed the memory capacity of a single chip, parallelism allows the workload to be distributed so the model can be trained in a reasonable timeframe.

Why it matters

Scaling is the primary driver of AI performance. Without distributed training, researchers would be limited to small models that cannot capture complex patterns. To build frontier systems, you must maximize hardware utilization; if a model is too large for one GPU, you must partition it without creating massive communication bottlenecks that slow down the training process.

Core Concepts

• Data Parallelism (DP): Every GPU holds a full copy of the model, but each processes a different slice of the data. Gradients are averaged across all GPUs to keep the models synchronized.
• Tensor Parallelism (TP): A single layer (tensor) is split across multiple GPUs. Each GPU computes a portion of the matrix multiplication, requiring frequent synchronization.
• Pipeline Parallelism (PP): The model is split vertically by layers. GPU 1 handles layers 1-10, GPU 2 handles 11-20, and so on, passing the output forward like an assembly line.
• Fully-Sharded Data Parallelism (FSDP): A hybrid approach that shards model weights, gradients, and optimizer states across GPUs, reducing memory redundancy while maintaining the efficiency of data parallelism.
• Collective Communication (All-Reduce): The mechanism used to synchronize data. All-Reduce ensures every GPU ends up with the summed total of gradients from all other GPUs.

Common Mistakes

• Ignoring Communication Overhead: Assuming adding more GPUs linearly increases speed. In reality, the time spent moving data between chips (interconnect latency) can become a bottleneck.
• Imbalanced Partitioning: Creating "bubbles" in pipeline parallelism where some GPUs sit idle while waiting for data from previous stages.
• Overlooking Memory Fragmentation: Failing to account for the memory used by optimizer states, which often exceeds the memory used by the model weights themselves.

Track Connection

This concept bridges the gap between Model Architecture and Hardware Infrastructure. It provides the operational framework necessary to implement the scaling laws discussed in the Frontier Systems track, moving from theoretical model design to physical deployment on GPU clusters.