Preference Optimization & RL from Verifiable Rewards (RLVR)

Preference Optimization & RL from Verifiable Rewards (RLVR)

What it is

Preference optimization is the process of refining a pre-trained model so its outputs align with human values, safety guidelines, or specific performance goals. RLVR (Reinforcement Learning from Verifiable Rewards) is a specialized approach where the model is trained using rewards based on objective, rule-based truth, such as whether a piece of code passes a unit test or a math problem reaches the correct final answer, rather than subjective human ratings.

Why it matters

Standard alignment often relies on human feedback, which is slow, expensive, and prone to inconsistency. RLVR allows for automated, scalable reinforcement learning. By using verifiable rewards, developers can force models to "think" through complex problems (test-time compute scaling) and self-correct, leading to breakthroughs in reasoning, coding, and mathematics where a definitive right or wrong answer exists.

Core concepts to master

• GRPO (Group Relative Policy Optimization): An efficient RL algorithm that removes the need for a separate critic model by comparing a group of outputs against each other to determine the relative reward.
• Rule-Based Verifiers: Hard-coded scripts or compilers that provide binary (pass/fail) feedback, eliminating human bias from the reward loop.
• Process Reward Models (PRMs): Systems that reward the individual steps of a reasoning chain rather than just the final answer, reducing "hallucinations" in logic.
• Test-Time Compute: The strategy of allowing a model to generate multiple internal reasoning paths or "attempts" before delivering a final answer to increase accuracy.

Common mistakes

• Reward Hacking: The model finding a loophole in the verifier to get a high score without actually solving the problem (e.g., formatting a wrong answer to look correct).
• Over-optimization: Training too aggressively on a specific reward, which can cause the model to lose its general conversational abilities (catastrophic forgetting).
• Ignoring the Process: Focusing only on the final outcome (outcome-based reward) rather than the logical steps taken to get there.

Track connection

This concept sits at the end of the model development pipeline. It follows supervised fine-tuning (SFT) and serves as the final "polish" that transforms a general-purpose LLM into a specialized reasoning engine capable of autonomous problem-solving.