Home › Tracks › AI Safety, Alignment & Interpretability
Elite track
Make powerful models honest, transparent, and governable.
Work on what frontier labs hire most heavily for: alignment techniques (RLHF, DPO, Constitutional AI), mechanistic interpretability that reverse-engineers what a network computes, safety evaluations and scalable oversight, adversarial robustness and red-teaming depth, and AI governance grounded in real frameworks (model cards, the EU AI Act, the NIST AI RMF). You leave able to ship an interpretability tool and a safety-and-governance pipeline. Bridges to Information Security, Theory of Computation, and Software Engineering.
Week by week
Every week unlocks the next. Concepts route you to free, world-class material; projects turn that knowledge into something deployed.
Week 1
Why a capable model can be unsafe even when it is accurate: specification gaming, reward hacking, goal misgeneralization, and the gap between what we train and what we want. The conceptual map before the techniques.
Bridges to Theory of Computation — specification, correctness, and the limits of formal goalsBuilds on: nothing, start here
Week 2
How a base model is shaped to human preferences: reward modeling, PPO, and the direct-optimization alternatives (DPO, GRPO) that drop the separate reward model. The modular SFT then preference then reasoning pipeline used in 2026.
Bridges to Machine Learning — reinforcement learning and optimization under feedbackBuilds on: The Alignment Problem & Safety Foundations
Week 4
Aligning models you can no longer fully supervise by hand: AI feedback, Constitutional AI, debate, and recursive reward modeling. How supervision scales when the model is smarter than the average human labeler.
Bridges to Distributed Systems — delegation, trust, and verification of work you cannot fully checkBuilds on: RLHF & Preference-Based Alignment
Week 5
Reverse-engineering a neural network into human-understandable algorithms: features, circuits, the residual stream, and induction heads. Reading what a transformer actually computes instead of treating it as a black box.
Bridges to Compilers — reverse engineering, intermediate representations, and program analysisBuilds on: The Alignment Problem & Safety Foundations
Week 7
Apply dictionary learning in practice. Implement Sparse Autoencoders (SAEs) and Natural Language Autoencoders (NLAs) with activation verbalizer-reconstructor reinforcement loops to audit hidden model states and detect unverbalized evaluation awareness.
Bridges to Software Engineering — instrumentation, debugging, and observability of complex systemsBuilds on: Mechanistic Interpretability Foundations
An enterprise mandate: build and launch an interpretability tool — a product that lets a user load a small open-weight transformer, run a prompt, and inspect what the model is actually computing: attention patterns, per-layer activations, the contribution of individual components, and features surfaced by a sparse autoencoder. This is reverse engineering as a product: make the internals of a black-box model legible. The deliverable is directly deployable and hyperscalable: real public hosting, a fast accessible inspector UI, CI/CD, observability, security, and full marketing (landing page, pitch, demo). Interpretability is what frontier labs hire most heavily for; ship a tool a researcher would actually use. Ship it as a real product.
Why it matters: Mechanistic interpretability is among the most heavily hired-for research directions at frontier labs, because regulators and leadership increasingly want to know what a deployed model is doing internally, not just how it scores. A builder who ships an interpretability tool that runs real circuit and feature analysis is directly deployable as an Interpretability Researcher or AI Safety Engineer, a scarce and rapidly growing role.
A publicly hosted interpretability tool with a stable URL and a fast, accessible model-inspector UI, plus a public repo: the activation-capture and analysis backend, attention and activation visualizations, an activation-patching workflow, a sparse-autoencoder feature view, CI/CD on every commit, production observability, a marketing landing page, a 10-slide pitch, a recorded demo, and a README documenting the interpretability methods, their limitations, and the scaling design.
Market signal, who wants thisInterpretability is a defined and growing 2026 research field with a dedicated ICML 2026 workshop and standard open tooling (TransformerLens, sparse autoencoders, Anthropic’s Transformer Circuits sequence). Frontier labs invest in it because understanding a model internally is increasingly a release and governance requirement, and the open-source ARENA curriculum exists specifically to train people into these roles. Investors and labs back interpretability because a model you cannot inspect is a model you cannot safely scale.
Week 8
Measuring whether a model is safe to release: capability and propensity evals, dangerous-capability assessments, honesty and sycophancy tests, and the benchmark-overfitting trap where high public scores hide real-world failure.
Bridges to Software Engineering — test design, coverage, and validation under uncertaintyBuilds on: Constitutional AI & Scalable Oversight
Week 9
Attacking a model to make it safer: jailbreaks, prompt injection and indirect injection, automated adversarial attack generation, and the OWASP LLM and Agentic-Security taxonomies that structure a serious red-team program.
Bridges to Information Security — adversarial thinking, threat modeling, and penetration testingBuilds on: Safety Evaluations & Dangerous-Capability Testing
Week 10
The compliance layer that ships with a model: model cards and system cards, the EU AI Act risk tiers and its August 2026 transparency rules, and the NIST AI RMF Govern-Map-Measure-Manage core that frames responsible AI in practice.
Bridges to Information Security — policy, compliance, and audit-ready documentationBuilds on: Safety Evaluations & Dangerous-Capability Testing
Week 11
Working like a researcher: reading a paper for its claim and evidence, reproducing a result from scratch, and designing the ablations that isolate what actually caused a gain. The skill frontier and research-adjacent roles assume.
Bridges to Software Engineering — the scientific method, controlled experiments, and reproducibilityBuilds on: Interpretability in Practice: SAEs, NLAs & Activation Patching
Week 12
An enterprise mandate: a company is about to ship an AI feature and has no defensible answer to "is this safe and is it compliant?". Build and launch a product with two interlocking systems: a safety-evaluation pipeline that runs capability, propensity, honesty, and adversarial red-team batteries against a model and scores how it behaves; and a governance layer that turns those results into an audit-ready model card and a risk classification mapped to the NIST AI RMF and the EU AI Act. The deliverable is directly deployable and hyperscalable: real public hosting, CI/CD, a hyper-usable dashboard a safety or compliance lead reads at a glance, security, and full marketing (landing page, pitch, demo). Deliver something an organization can run before release, not after an incident. Ship it as a real product.
Why it matters: AI safety evaluation and governance is becoming a release gate, not an afterthought: the EU AI Act transparency rules apply from August 2026 and the NIST AI RMF is the de facto governance structure organizations adopt. A builder who can ship an evaluation-plus-governance pipeline is directly deployable as an AI Safety Engineer, an Evaluations Engineer, or an AI Governance specialist, scarce roles because they combine ML, security, and policy.
A publicly hosted product with a stable URL and a hyper-usable safety-and-governance dashboard, plus a public repo: the safety-evaluation pipeline with capability, propensity, honesty, and red-team batteries, the auto-generated model card and risk-classification layer, CI/CD on every commit, production observability, a marketing landing page, a 10-slide pitch, a recorded demo, and a README documenting the evaluation methodology, the governance mapping, and the scaling design.
Market signal, who wants thisAI governance and safety evaluation is a defined 2026 enterprise requirement: the EU AI Act transparency obligations take effect in August 2026, NIST released an AI RMF profile for critical infrastructure in April 2026, and enterprises are explicitly investing in audit-ready model cards, risk classification, and incident response. Research shows public benchmark scores can hide real-world failure, making independent evaluation valuable. Investors and enterprises fund safety-and-governance tooling because shipping AI into regulated industries without it is now a legal and reputational liability.
What's next
Each track stands alone, so there's no wrong order. If you want a suggestion, this one pairs well next.