RLHF & Preference-Based Alignment

RLHF & Preference-Based Alignment

What it is

Reinforcement Learning from Human Feedback (RLHF) is the process of fine-tuning a raw base model so its outputs align with human values, safety guidelines, and specific stylistic preferences. While a base model predicts the next token, an aligned model follows instructions and avoids harmful content.

Why it matters

Base models are trained on the internet, meaning they mirror all its biases, inaccuracies, and toxicity. Without alignment, a model might provide a correct technical answer but do so in a rude manner, or provide a helpful answer to a dangerous request. Alignment transforms a statistical predictor into a reliable assistant.

Core Concepts

• The Traditional Pipeline: This typically involves Supervised Fine-Tuning (SFT) to teach the model the basic format of a conversation, followed by a preference phase to refine the quality of those responses.
• Reward Modeling & PPO: In standard RLHF, a separate "Reward Model" is trained to predict which of two responses a human would prefer. Proximal Policy Optimization (PPO) then uses this reward model to update the main AI, effectively "gaming" the reward model to maximize quality.
• Direct Optimization (DPO & GRPO): Newer methods like Direct Preference Optimization (DPO) remove the need for a separate reward model. They optimize the policy directly on preference pairs, reducing computational overhead and instability.
• Reasoning Pipelines: Modern workflows often add a final layer of reasoning optimization, where the model is rewarded not just for the final answer, but for the logical steps (Chain-of-Thought) it took to get there.

Common Mistakes

• Reward Hacking: This occurs when a model finds a loophole to get a high score from the reward model without actually solving the task (e.g., adding overly polite phrases that humans like but don't add value).
• Over-optimization: Pushing alignment too hard can lead to "mode collapse," where the model becomes bland, repetitive, or refuses harmless prompts due to excessive safety caution.

Track Connection

This concept bridges the gap between raw model architecture and AI Safety. It is the primary mechanism for implementing the safety guardrails discussed in the Alignment track and provides the behavioral data needed for Interpretability research to understand why a model chooses one response over another.