Continuous Batching & Throughput Scheduling

Continuous Batching & Throughput Scheduling

What it is

In traditional static batching, a GPU processes a group of requests and waits for every single sequence in that batch to finish before starting the next set. Continuous batching (or iteration-level scheduling) removes this bottleneck. Instead of waiting for the entire batch to complete, the system inserts new requests into the batch as soon as any single request finishes its generation.

Why it matters

LLMs generate text one token at a time, and different requests have different output lengths. Static batching creates "bubbles" of wasted compute where the GPU sits idle while waiting for the longest sequence in a batch to finish. Continuous batching maximizes GPU utilization, significantly increasing the number of tokens generated per second (throughput) and reducing the time users wait for a response.

Core Concepts

• Prefill vs. Decode: The "prefill" phase processes the entire input prompt in one go (compute-bound), while the "decode" phase generates tokens one by one (memory-bound). Scheduling must balance these two distinct workloads.
• Iteration-Level Scheduling: The scheduler makes decisions at every single token generation step, allowing it to swap finished requests for pending ones in real-time.
• KV Cache Management: To enable continuous batching, the system must efficiently manage the Key-Value (KV) cache in memory so that new requests don't overwrite the data of ongoing ones.
• Tail Latency: While increasing batch size boosts throughput, it can increase the time it takes for a specific token to be generated (latency). The goal is to maximize throughput without pushing the 99th percentile latency past a usable threshold.

Common Mistakes

• Over-batching: Increasing the batch size too far can lead to "Out of Memory" (OOM) errors or cause latency to spike, making the system feel sluggish despite high throughput.
• Ignoring Prefill Spikes: Failing to account for the heavy compute cost of large prefills can cause "stutters" in the generation of tokens for existing requests in the batch.

Track Connection

This concept bridges the gap between raw model architecture and production deployment. It connects directly to Memory Management (PagedAttention), which provides the technical mechanism to store the KV cache, and Hardware Acceleration, which determines the physical limits of the GPU's compute and memory bandwidth.