Production Serving, PD Disaggregation & Autoscaling

Production Serving: PD Disaggregation & Autoscaling

What it is

Production serving is the process of deploying Large Language Models (LLMs) to handle real-world traffic. Prefill-Decode (PD) Disaggregation is a specific architecture that separates the two phases of LLM inference: the "prefill" phase (processing the initial input prompt) and the "decode" phase (generating tokens one by one). Instead of one GPU doing both, dedicated clusters are assigned to each task.

Why it matters

In standard serving, prefill and decode compete for the same compute resources. Prefills are compute-bound and bursty, while decodes are memory-bandwidth bound and steady. This competition leads to "jitter" and high latency. Disaggregating these phases allows engineers to optimize hardware and scaling policies for each specific workload, ensuring a consistent user experience (SLA) even under heavy load.

Core concepts to master

• PD Disaggregation: Separating the compute-heavy prompt processing from the memory-heavy token generation to eliminate resource contention.
• Distributed KV Transfer: The mechanism used to move the Key-Value (KV) cache from the prefill node to the decode node so the model remembers the prompt context.
• SLA-Aware Routing: A load-balancing strategy that routes requests based on current latency targets rather than simple round-robin distribution.
• Multi-Node Autoscaling: Dynamically adding or removing GPU nodes based on request volume and queue depth to balance cost and performance.

Common mistakes

• Ignoring Transfer Overhead: Failing to account for the network latency involved in moving the KV cache between nodes, which can negate the speed gains of disaggregation.
• Static Scaling: Using fixed instance counts instead of dynamic autoscaling, leading to either wasted GPU spend or request timeouts during traffic spikes.
• Overlooking Memory Fragmentation: Neglecting how KV cache management affects the maximum batch size on decode nodes.

Connection to the track

This concept sits at the intersection of model optimization and systems engineering. It builds upon basic inference techniques (like quantization) and provides the foundation for advanced deployment strategies. Once PD disaggregation is mastered, the learner can move toward optimizing the underlying network fabric and storage layers to further reduce KV transfer latency.