Home › Tracks › AI Infrastructure & Inference
12-week elite track
Serve frontier models fast, cheap, and at scale.
Own the serving layer: the transformer internals that decide cost, quantization, KV-cache management, continuous batching, paged attention, speculative decoding, and GPU-aware deployment. You leave able to stand up an inference platform that holds an SLO under load. Bridges to Operating Systems, Computer Architecture, and Computer Networks.
Week by week
Every week unlocks the next. Concepts route you to free, world-class material; projects turn that knowledge into something deployed.
Week 1
You cannot optimize what you do not understand. Tokens, attention, the prefill and decode phases, and exactly where compute and memory go during a forward pass.
Bridges to Computer Architecture — instruction-level parallelism and the memory wallBuilds on: nothing — start here
Week 2
Why inference is memory-bandwidth bound, not compute bound. SMs, warps, HBM versus SRAM, arithmetic intensity, and the roofline model that decides what 'fast' even means.
Bridges to Computer Architecture — parallelism, memory hierarchy, and the roofline modelBuilds on: Transformer Internals for Serving
Week 3
The attention kernel that made long context affordable: tiling, kernel fusion, and being IO-aware about HBM traffic. The general principle of fusing memory-bound operations.
Bridges to Operating Systems — I/O scheduling and memory-bound versus compute-bound workBuilds on: GPU Architecture & the Memory Wall
Week 4
The KV cache is the dominant memory cost of serving. Fragmentation, paging, prefix sharing, and how PagedAttention applied virtual-memory ideas to GPU memory.
Bridges to Operating Systems — virtual memory, paging, and fragmentationBuilds on: FlashAttention & Kernel-Level Optimization
Week 5
Static batching wastes the GPU. Continuous (iteration-level) batching, request scheduling, prefill-decode tradeoffs, and how to push throughput without wrecking tail latency.
Bridges to Operating Systems — CPU scheduling, throughput versus latency tradeoffsBuilds on: KV-Cache & Paged Attention
Week 6
An enterprise mandate: the platform team needs a launched inference product that holds a strict latency SLO while maximizing GPU throughput. Build (or extend a serving engine into) an inference service that implements a KV cache with paged memory management, iteration-level continuous batching, and a request scheduler that balances time-to-first-token against tokens-per-second. The deliverable is not a benchmark notebook — it is a directly deployable, hyperscalable product: a real public API and a clean playground UI, CI/CD, autoscaling across replicas, production observability for tokens-per-second and latency percentiles, security on the endpoint, and full marketing (landing page, pitch, demo) so it is presentable as a real product. Measure it honestly under load. The GPU is the most expensive thing in the building; idle cycles are a defect. We are not here to babysit it; ship it as a real product.
Why it matters: Inference serving is where AI cost is won or lost; a 2x throughput gain is a direct margin gain for any company running models. Shipping a measured, SLO-holding inference server makes a builder a credible AI Infrastructure or Inference Engineer, one of the highest-paid frontier roles because it converts directly into saved spend.
A publicly hosted inference service with a stable URL and a clean playground UI, plus a public repo: the batching scheduler and KV-cache manager, an autoscaling deployment, CI/CD on every commit, production observability dashboards, a load-test harness, a benchmark report comparing static versus continuous batching across concurrency levels, a marketing landing page, a 10-slide pitch, a recorded demo, and a README explaining the memory, scheduling, and scaling design.
Market signal — who wants thisInference is now a FinOps problem: at production scale it accounts for over 80% of AI GPU spend, and software optimization alone has driven cost-per-million-tokens down 5x on new hardware within months. A funded infrastructure category has formed around exactly this product — vLLM, Runpod (FlashBoot sub-250ms cold starts), BentoML, and Yotta Labs — because self-hosting beats managed APIs on unit economics above ~100M tokens/month. Investors fund inference platforms because every company running open-weight models needs to cut serving cost without losing quality.
Week 7
Shrink a model to fit and run faster: INT8/INT4 weight quantization, GPTQ and AWQ, FP8, and the accuracy-versus-cost tradeoff measured honestly.
Bridges to Computer Architecture — number representation and fixed-point arithmeticBuilds on: Continuous Batching & Throughput Scheduling
Week 8
Cut decode latency by guessing ahead: a small draft model proposes tokens a large model verifies in parallel. Acceptance rates, draft selection, and when speculation pays off.
Bridges to Computer Architecture — speculative and out-of-order executionBuilds on: Quantization & Model Compression
Week 10
Wrap a model in a service that holds an SLO: load balancing across replicas, GPU autoscaling, cold-start mitigation, and observability for tokens-per-second and time-to-first-token.
Bridges to Distributed Systems — load balancing, replication, and capacity planningBuilds on: Speculative Decoding & Latency Optimization
Week 12
An enterprise mandate: a capable open-weight model must run within a fixed GPU budget at half the current latency, with no unacceptable accuracy loss, and it must ship as a launched product. Quantize the model (INT4/INT8 or FP8), pair it with a draft model for speculative decoding, deploy it behind an autoscaling service, and prove the result. The deliverable is directly deployable and hyperscalable: a real public API and a hyper-usable demo UI, CI/CD, autoscaling, production observability, endpoint security, a finance-grade cost-per-token report, and full marketing (landing page, pitch, demo). Deliver a deployment a finance team would sign off on: the cost-per-million-tokens must drop and you must show it did. We are not here to babysit it; ship it as a real product.
Why it matters: Every company running open-weight models in production needs someone who can cut inference cost without breaking quality. A builder who can quantize, speculate, and deploy with a defensible cost report is directly deployable as a Senior Inference Engineer, a role compensated at the ₹1-crore tier because the savings are measured in real money.
A publicly hosted deployment with a stable URL and a hyper-usable demo UI, plus a public repo: the quantization and speculative-decoding pipeline, an autoscaling serving setup, CI/CD on every commit, production observability, an accuracy report on a representative benchmark before and after compression, a marketing landing page, a 10-slide pitch, a recorded demo, and a README with the cost-per-token analysis and scaling design.
Market signal — who wants thisGPU FinOps is a defined 2026 budget line: inference is over 80% of AI GPU spend, and quantization plus speculative decoding are the highest-leverage cost cuts (FP8 alone gives 1.3-2x throughput at under 2% quality loss). Hardware-plus-software optimization has delivered 5x cost-per-token reductions, and a market of inference-cost tooling (Spheron, regolo.ai, BentoML, Yotta Labs) has formed around it. Investors fund cost-optimization products because the savings convert directly into gross margin for anyone serving open-weight models at volume.