Supervised Fine-Tuning

Supervised Fine-Tuning (SFT)

What it is

Supervised Fine-Tuning is the process of taking a pre-trained base model, which has learned general language patterns from a massive dataset, and training it further on a smaller, curated dataset of prompt-response pairs. The core idea is to shift the model from predicting the next word in a sequence to following specific instructions or adopting a particular persona.

Why it matters

Base models are powerful but often unpredictable; they may continue a prompt as a story rather than answering a question. SFT transforms a general-purpose engine into a functional product. It allows developers to align the model with specific domain knowledge, brand voice, or complex task requirements (like JSON formatting) that cannot be reliably achieved through prompting alone.

Core concepts to master

• Instruction Tuning: The process of using datasets formatted as "Instruction $\rightarrow$ Output" to teach the model how to respond to user commands.
• Hyperparameter Tuning: Focus on learning rate and batch size. Because the model is already pre-trained, a very low learning rate is required to prevent the model from "unlearning" its base knowledge.
• Catastrophic Forgetting: A failure mode where the model becomes so specialized in the new task that it loses the general reasoning capabilities it had during pre-training.
• Overfitting: When the model memorizes the specific examples in the SFT dataset rather than learning the underlying pattern, leading to poor performance on new, unseen prompts.

Common mistakes

• Low-quality data: Using thousands of mediocre examples instead of a few hundred high-quality, diverse examples. SFT is highly sensitive to data noise.
• Over-training: Running too many epochs, which triggers catastrophic forgetting and makes the model rigid and repetitive.
• Lack of a baseline: Failing to test the base model with few-shot prompting before starting SFT, making it impossible to measure the actual improvement.

Track connection

SFT is the bridge between raw model pre-training and final alignment. In the Applied ML track, it follows the selection of a base model and precedes Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO), which further refine the model's safety and preference alignment.