GPU Architecture & the Memory Wall

GPU Architecture & the Memory Wall

What it is

GPU architecture is designed for massive parallelism, using thousands of small cores grouped into Streaming Multiprocessors (SMs). While these cores can perform trillions of calculations per second (compute), they can only process data as fast as it can be moved from memory to the cores (bandwidth). The "Memory Wall" is the performance gap where the processor spends more time waiting for data to arrive than actually performing calculations.

Why it matters

In AI inference, the goal is low latency. If a model is "compute-bound," adding more Teraflops makes it faster. However, most modern LLM inference is "memory-bandwidth bound." This means the GPU cores are idling while waiting for weights to load from High Bandwidth Memory (HBM). Understanding this allows engineers to optimize for throughput via quantization or KV caching rather than simply buying faster chips.

Core Concepts

• SMs and Warps: SMs are the hardware units that execute code. They process data in "warps" (groups of 32 threads) using a SIMT (Single Instruction, Multiple Threads) model to hide latency.
• HBM vs. SRAM: HBM is large, off-chip memory with high capacity but slower access. SRAM (L1/L2 cache) is tiny, on-chip memory that is incredibly fast. The goal is to keep data in SRAM as long as possible.
• Arithmetic Intensity: The ratio of total floating-point operations (FLOPs) to the total bytes of data moved. High intensity means the GPU is compute-bound; low intensity means it is memory-bound.
• The Roofline Model: A visual tool used to determine if a kernel is limited by memory bandwidth or peak compute performance, defining the theoretical ceiling of a model's speed.

Common Mistakes

• Overestimating TFLOPS: Assuming a GPU with higher peak compute will automatically run a model faster, ignoring the bandwidth bottleneck.
• Ignoring Data Movement: Focusing solely on optimizing the mathematical operation while ignoring the cost of moving tensors between memory tiers.

Track Connection

This concept provides the hardware foundation for the rest of the AI Infrastructure track. It explains why techniques like Quantization (reducing byte size) and FlashAttention (reducing memory reads/writes) are the primary drivers of inference speed.