Safety Evaluations & Dangerous-Capability Testing

Safety Evaluations & Dangerous-Capability Testing

What it is

Safety evaluations are systematic tests used to determine if an AI model is safe for deployment. Rather than testing general intelligence, these evaluations specifically look for "dangerous capabilities", the ability of a model to assist in creating biological weapons, executing cyberattacks, or manipulating humans. The core idea is to move from intuitive "vibes" about safety to empirical, measurable evidence.

Why it matters

In real-world systems, a model that is 99% safe can still be catastrophic if the 1% failure involves a high-impact risk. Without rigorous testing, developers cannot know if a model has developed emergent properties that allow it to bypass safety filters or provide actionable instructions for illegal acts. These evaluations act as the final "go/no-go" gate before a model reaches the public.

Core concepts to understand

• Capability vs. Propensity: Capability is whether the model *can* do something dangerous; propensity is whether it *will* do it when prompted. A model might have the knowledge to write malware but a strong safety filter that prevents it.
• Honesty and Sycophancy: Tests to see if a model tells the truth or simply tells the user what they want to hear (sycophancy) to appear helpful, which can lead to dangerous misinformation.
• The Overfitting Trap: The risk that a model is trained on the specific questions found in public benchmarks. This results in high "paper" scores while the model remains unsafe in real-world, novel scenarios.
• Red Teaming: The process of using human experts to intentionally provoke the model into failing to find hidden vulnerabilities.

Common mistakes

• Relying on public benchmarks: Assuming a high score on a known dataset equals real-world safety.
• Ignoring "jailbreaks": Testing the model only with polite prompts and ignoring adversarial inputs designed to bypass filters.
• Confusing helpfulness with safety: Mistaking a model's polite tone for an absence of dangerous capabilities.

Connection to the track

Safety evaluations provide the empirical data needed for Alignment. While interpretability helps us understand *why* a model behaves a certain way, evaluations tell us *what* the behavior actually is. Together, they allow researchers to iterate on training methods to reduce risk before release.