Home › Tracks › AI Safety, Alignment & Interpretability
Elite track
Make powerful models honest, transparent, and governable.
Work on what frontier labs hire most heavily for: alignment techniques (RLHF, DPO, Constitutional AI), mechanistic interpretability that reverse-engineers what a network computes, safety evaluations and scalable oversight, adversarial robustness and red-teaming depth, and AI governance grounded in real frameworks (model cards, the EU AI Act, the NIST AI RMF). You leave able to ship an interpretability tool and a safety-and-governance pipeline. Bridges to Information Security, Theory of Computation, and Software Engineering.
Week by week
Every week unlocks the next. Concepts route you to free, world-class material; projects turn that knowledge into something deployed.
Week 1
Why a capable model can be unsafe even when it is accurate: specification gaming, reward hacking, goal misgeneralization, and the gap between what we train and what we want. The conceptual map before the techniques.
Bridges to Theory of Computation — specification, correctness, and the limits of formal goalsBuilds on: nothing, start here
Week 2
How a base model is shaped to human preferences: reward modeling, PPO, and the direct-optimization alternatives (DPO, GRPO) that drop the separate reward model. The modular SFT then preference then reasoning pipeline used in 2026.
Bridges to Machine Learning — reinforcement learning and optimization under feedbackBuilds on: The Alignment Problem & Safety Foundations
Week 4
Aligning models you can no longer fully supervise by hand: AI feedback, Constitutional AI, debate, and recursive reward modeling. How supervision scales when the model is smarter than the average human labeler.
Bridges to Distributed Systems — delegation, trust, and verification of work you cannot fully checkBuilds on: RLHF & Preference-Based Alignment
Week 5
Reverse-engineering a neural network into human-understandable algorithms: features, circuits, the residual stream, and induction heads. Reading what a transformer actually computes instead of treating it as a black box.
Bridges to Compilers — reverse engineering, intermediate representations, and program analysisBuilds on: The Alignment Problem & Safety Foundations
Week 7
Apply dictionary learning in practice. Implement Sparse Autoencoders (SAEs) and Natural Language Autoencoders (NLAs) with activation verbalizer-reconstructor reinforcement loops to audit hidden model states and detect unverbalized evaluation awareness.
Bridges to Software Engineering — instrumentation, debugging, and observability of complex systemsBuilds on: Mechanistic Interpretability Foundations
Build an active mechanistic auditing tool using TransformerLens and Jacobian analysis. Your system must analyze internal J-space representations on the fly, intercepting and blocking adversarial prompt injections and faked evaluation responses before the model's first output token is generated.
An interactive auditing dashboard and CLI tool that intercepts, visualizes, and dynamically patches J-space activations to prevent unauthorized model behaviors.
Week 8
Measuring whether a model is safe to release: capability and propensity evals, dangerous-capability assessments, honesty and sycophancy tests, and the benchmark-overfitting trap where high public scores hide real-world failure.
Bridges to Software Engineering — test design, coverage, and validation under uncertaintyBuilds on: Constitutional AI & Scalable Oversight
Week 9
Attacking a model to make it safer: jailbreaks, prompt injection and indirect injection, automated adversarial attack generation, and the OWASP LLM and Agentic-Security taxonomies that structure a serious red-team program.
Bridges to Information Security — adversarial thinking, threat modeling, and penetration testingBuilds on: Safety Evaluations & Dangerous-Capability Testing
Week 10
The compliance layer that ships with a model: model cards and system cards, the EU AI Act risk tiers and its August 2026 transparency rules, and the NIST AI RMF Govern-Map-Measure-Manage core that frames responsible AI in practice.
Bridges to Information Security — policy, compliance, and audit-ready documentationBuilds on: Safety Evaluations & Dangerous-Capability Testing
Week 11
Working like a researcher: reading a paper for its claim and evidence, reproducing a result from scratch, and designing the ablations that isolate what actually caused a gain. The skill frontier and research-adjacent roles assume.
Bridges to Software Engineering — the scientific method, controlled experiments, and reproducibilityBuilds on: Interpretability in Practice: SAEs, NLAs & Activation Patching
Week 12
An enterprise mandate: a company is about to ship an AI feature and has no defensible answer to "is this safe and is it compliant?". Build and launch a product with two interlocking systems: a safety-evaluation pipeline that runs capability, propensity, honesty, and adversarial red-team batteries against a model and scores how it behaves; and a governance layer that turns those results into an audit-ready model card and a risk classification mapped to the NIST AI RMF and the EU AI Act. The deliverable is directly deployable and hyperscalable: real public hosting, CI/CD, a hyper-usable dashboard a safety or compliance lead reads at a glance, security, and full marketing (landing page, pitch, demo). Deliver something an organization can run before release, not after an incident. Ship it as a real product.
Why it matters: AI safety evaluation and governance is becoming a release gate, not an afterthought: the EU AI Act transparency rules apply from August 2026 and the NIST AI RMF is the de facto governance structure organizations adopt. A builder who can ship an evaluation-plus-governance pipeline is directly deployable as an AI Safety Engineer, an Evaluations Engineer, or an AI Governance specialist, scarce roles because they combine ML, security, and policy.
A publicly hosted product with a stable URL and a hyper-usable safety-and-governance dashboard, plus a public repo: the safety-evaluation pipeline with capability, propensity, honesty, and red-team batteries, the auto-generated model card and risk-classification layer, CI/CD on every commit, production observability, a marketing landing page, a 10-slide pitch, a recorded demo, and a README documenting the evaluation methodology, the governance mapping, and the scaling design.
Market signal, who wants thisAI governance and safety evaluation is a defined 2026 enterprise requirement: the EU AI Act transparency obligations take effect in August 2026, NIST released an AI RMF profile for critical infrastructure in April 2026, and enterprises are explicitly investing in audit-ready model cards, risk classification, and incident response. Research shows public benchmark scores can hide real-world failure, making independent evaluation valuable. Investors and enterprises fund safety-and-governance tooling because shipping AI into regulated industries without it is now a legal and reputational liability.
What's next
Each track stands alone, so there's no wrong order. If you want a suggestion, this one pairs well next.