Home › Tracks › AI Infrastructure & Inference
Elite track
Serve frontier models fast, cheap, and at scale.
Own the serving layer: the transformer internals that decide cost, quantization, KV-cache management, continuous batching, paged attention, speculative decoding, and GPU-aware deployment. You leave able to stand up an inference platform that holds an SLO under load. Bridges to Operating Systems, Computer Architecture, and Computer Networks.
Week by week
Every week unlocks the next. Concepts route you to free, world-class material; projects turn that knowledge into something deployed.
Week 1
You cannot optimize what you do not understand. Tokens, attention, the prefill and decode phases, and exactly where compute and memory go during a forward pass.
Bridges to Computer Architecture — instruction-level parallelism and the memory wallBuilds on: nothing, start here
Week 2
Why inference is memory-bandwidth bound, not compute bound. SMs, warps, HBM versus SRAM, arithmetic intensity, and the roofline model that decides what 'fast' even means.
Bridges to Computer Architecture — parallelism, memory hierarchy, and the roofline modelBuilds on: Transformer Internals for Serving
Week 3
Deep dive into kernel-level attention optimization. Move beyond quadratic computation complexity with Subquadratic Sparse Attention (SSA) routing mechanisms that selectively process key-value tokens, pushing viable context windows up to 12 million tokens.
Bridges to Operating Systems — I/O scheduling and memory-bound versus compute-bound workBuilds on: GPU Architecture & the Memory Wall
Week 4
The KV cache is the dominant memory cost of serving. Fragmentation, paging, prefix sharing, and how PagedAttention applied virtual-memory ideas to GPU memory.
Bridges to Operating Systems — virtual memory, paging, and fragmentationBuilds on: FlashAttention & Subquadratic Sparse Attention
Week 5
Static batching wastes the GPU. Continuous (iteration-level) batching, request scheduling, prefill-decode tradeoffs, and how to push throughput without wrecking tail latency.
Bridges to Operating Systems — CPU scheduling, throughput versus latency tradeoffsBuilds on: KV-Cache & Paged Attention
Week 6
An enterprise mandate: the platform team needs a launched inference product that holds a strict latency SLO while maximizing GPU throughput. Build (or extend a serving engine into) an inference service that implements a KV cache with paged memory management, iteration-level continuous batching, and a request scheduler that balances time-to-first-token against tokens-per-second. The deliverable is not a benchmark notebook — it is a directly deployable, hyperscalable product: a real public API and a clean playground UI, CI/CD, autoscaling across replicas, production observability for tokens-per-second and latency percentiles, security on the endpoint, and full marketing (landing page, pitch, demo) so it is presentable as a real product. Measure it honestly under load. The GPU is the most expensive thing in the building; idle cycles are a defect. We are not here to babysit it; ship it as a real product.
Why it matters: Inference serving is where AI cost is won or lost; a 2x throughput gain is a direct margin gain for any company running models. Shipping a measured, SLO-holding inference server makes a builder a credible AI Infrastructure or Inference Engineer, an in-demand frontier role because it converts directly into saved spend.
A publicly hosted inference service with a stable URL and a clean playground UI, plus a public repo: the batching scheduler and KV-cache manager, an autoscaling deployment, CI/CD on every commit, production observability dashboards, a load-test harness, a benchmark report comparing static versus continuous batching across concurrency levels, a marketing landing page, a 10-slide pitch, a recorded demo, and a README explaining the memory, scheduling, and scaling design.
Market signal, who wants thisInference is now a FinOps problem: at production scale it accounts for over 80% of AI GPU spend, and software optimization alone has driven cost-per-million-tokens down 5x on new hardware within months. A funded infrastructure category has formed around exactly this product — vLLM, Runpod (FlashBoot sub-250ms cold starts), BentoML, and Yotta Labs — because self-hosting beats managed APIs on unit economics above ~100M tokens/month. Investors fund inference platforms because every company running open-weight models needs to cut serving cost without losing quality.
Week 7
Shrink a model to fit and run faster: INT8/INT4 weight quantization, GPTQ and AWQ, FP8, and the accuracy-versus-cost tradeoff measured honestly.
Bridges to Computer Architecture — number representation and fixed-point arithmeticBuilds on: Continuous Batching & Throughput Scheduling
Week 8
Cut decode latency by guessing ahead: a small draft model proposes tokens a large model verifies in parallel. Acceptance rates, draft selection, and when speculation pays off.
Bridges to Computer Architecture — speculative and out-of-order executionBuilds on: Quantization & Model Compression
Week 10
Wrap a model in a service that holds an SLO: load balancing across replicas, GPU autoscaling, cold-start mitigation, and observability for tokens-per-second and time-to-first-token.
Bridges to Distributed Systems — load balancing, replication, and capacity planningBuilds on: Speculative Decoding & Latency Optimization
Week 11
Deploy trillion-parameter open-weight models locally on non-NVIDIA hardware. Master Apple's MLX framework for 4-bit quantization, speculative decoding, and continuous batching on Apple Silicon. Build high-bandwidth local clusters using Thunderbolt 5 RDMA.
Bridges to Computer Architecture — specialized processors and hardware-agnostic compilationBuilds on: Production Serving & Autoscaling
Week 12
Quantize and deploy an open-weight mixture-of-experts (MoE) model locally. Architect a multi-node local cluster utilizing high-speed interconnects (Thunderbolt 5 RDMA) and the Apple MLX framework, implementing speculative decoding and continuous batching on native hardware.
An optimized mlx_lm or local inference server cluster achieving high tokens-per-second, utilizing 4-bit quantized layers, speculative drafting, and local node orchestration.
What's next
Each track stands alone, so there's no wrong order. If you want a suggestion, this one pairs well next.