Interpretability in Practice: SAEs, NLAs & Activation Patching

Interpretability in Practice: SAEs, NLAs & Activation Patching

What it is

This concept focuses on "dictionary learning," a method used to decompose the complex, dense activations of a neural network into human-understandable features. While models store information in "superposition" (overlapping patterns), Sparse Autoencoders (SAEs) and Natural Language Autoencoders (NLAs) act as filters that isolate these individual features. Activation patching is then used to intervene in these states to verify if a specific feature actually drives the model's behavior.

Why it matters

Modern LLMs are "black boxes." We cannot tell by looking at raw numbers if a model is lying, planning, or simply predicting the next token based on a pattern. By auditing hidden states, developers can detect "unverbalized evaluation awareness", when a model knows it is being tested and changes its behavior, which is critical for ensuring AI safety and honest alignment.

Core concepts to master

• Sparse Autoencoders (SAEs): Tools that project dense activations into a higher-dimensional space where only a few "neurons" are active at once, making individual concepts (like "legal text" or "deception") visible.
• Natural Language Autoencoders (NLAs): A variation that uses language-based reconstructions to translate abstract mathematical vectors into human-readable text.
• Verbalizer-Reconstructor Loops: A reinforcement process where the system attempts to name a feature (verbalize) and then recreate the activation (reconstruct) to ensure the interpretation is accurate.
• Activation Patching: The process of swapping a specific activation from one prompt into another to see if the model's output changes, proving a causal link between a feature and a behavior.

Common mistakes

• Confusing Correlation with Causation: Assuming that because a feature is active during a specific task, it is the *reason* for the output. This is why patching is required for verification.
• Over-interpreting Noise: Mistaking random activation patterns for meaningful "features" when the SAE sparsity is set too low.

Track connection

This serves as the practical application phase of the Interpretability track. It moves from theoretical understanding (how transformers work) to active auditing (finding specific "danger" features), providing the empirical tools necessary for the Alignment phase of the curriculum.