RLHF, Direct Alignment & Verifiable Reasoning (GRPO)

Study Notes: RLHF, Direct Alignment, and Verifiable Reasoning

What it is

Alignment is the process of ensuring an AI model's outputs match human intentions and safety standards. Traditionally, this is done via Reinforcement Learning from Human Feedback (RLHF), where a model is trained to maximize a reward score given by a separate "reward model" that mimics human preferences.

Modern alignment is shifting toward more efficient methods. Direct Preference Optimization (DPO) removes the need for a separate reward model by optimizing the policy directly on preference pairs. Group Relative Policy Optimization (GRPO) further evolves this by removing the "Critic" network, instead calculating rewards based on the relative performance of a group of outputs for the same prompt.

Why it matters

Raw base models are proficient at predicting the next token but often produce hallucinations, harmful content, or illogical reasoning. Alignment transforms a predictive engine into a reliable assistant. For reasoning-heavy tasks (like math or coding), verifiable reasoning ensures the model doesn't just "sound" correct but follows a logically sound path to a provably correct answer.

Core concepts to master

• PPO (Proximal Policy Optimization): The classical actor-critic framework used in RLHF to update the model without causing catastrophic collapses in performance.
• DPO (Direct Preference Optimization): A simplified approach that treats alignment as a classification problem between "preferred" and "rejected" responses.
• GRPO (Group Relative Policy Optimization): An efficiency-focused method that uses group-based relative rewards, eliminating the memory overhead of a value-function network.
• Verifiable Rewards: Using objective ground-truth (like a compiler or a math answer) rather than human intuition to guide the model's reasoning process.

Common mistakes

• Reward Hacking: Assuming a high reward score equals a high-quality answer. Models often find "shortcuts" to trick the reward model without actually solving the problem.
• Over-optimization: Pushing alignment too far, which can lead to "mode collapse" where the model becomes overly cautious or repetitive.

Track connection

This concept bridges the gap between Model Training and AI Safety. While interpretability helps us understand *why* a model fails, alignment provides the tools to *fix* those failures. It is the primary mechanism for implementing the safety guardrails discussed throughout the Alignment & Interpretability track.