KV-Cache & Paged Attention

KV-Cache & PagedAttention

What it is

The KV (Key-Value) cache is a memory optimization used during LLM inference. In a transformer, every new token generated requires looking back at all previous tokens. Instead of recomputing the mathematical representations (Keys and Values) for every past token at every step, the system stores them in a cache. PagedAttention is a memory management technique that organizes this cache into non-contiguous blocks, similar to how virtual memory works in operating systems.

Why it matters

Memory is the primary bottleneck in AI serving. Without a KV cache, inference speed slows down quadratically as the sequence grows. However, the cache itself consumes massive amounts of VRAM. Traditional caching requires contiguous memory blocks, leading to "internal fragmentation" where memory is reserved but unused. PagedAttention eliminates this waste, allowing providers to increase batch sizes and serve more users on the same hardware.

Core concepts to master

• The KV Cache: Storing previous token states to avoid redundant computation during the autoregressive generation process.
• Memory Fragmentation: The gap between the maximum possible sequence length allocated and the actual tokens generated, which wastes GPU memory.
• PagedAttention: Dividing the KV cache into fixed-size blocks that can be stored anywhere in memory, mapped via a block table.
• Prefix Sharing: The ability for multiple requests to point to the same physical memory blocks for a shared prompt, drastically reducing memory overhead for common system instructions.

Common mistakes

• Confusing Weights with Cache: Mistaking model weights (static) for the KV cache (dynamic). Weights are constant; the cache grows with the sequence length and number of users.
• Overlooking Memory Growth: Forgetting that the KV cache size scales linearly with the number of layers, heads, and the sequence length.
• Assuming Contiguity: Thinking that GPU memory must be allocated in one long string; PagedAttention proves that logical sequences can be physically fragmented.

Track connection

This concept bridges the gap between Model Architecture (how transformers work) and Inference Optimization (how to deploy them). It is the foundation for understanding high-throughput serving engines and is a prerequisite for studying advanced quantization and speculative decoding.