Fine-Tuning
Fine-tuning modifies a model's weights by continuing training on a curated dataset. The result is a model that behaves differently at inference time — without requiring that behavior to be specified in a prompt. It is not magic: fine-tuning reshapes existing capabilities, it does not add new knowledge.
What Changes vs. What Doesn't
Fine-tuning reliably improves:
- Output format and style — consistent JSON structure, a house tone, domain-specific vocabulary
- Following complex multi-step instructions that would require long prompts to specify each time
- Task-specific behavior patterns — structured extraction, classification schemas, response formats
Fine-tuning does not reliably add:
- New factual knowledge — events after the training cutoff, proprietary data, real-time information. The model may appear to learn facts from fine-tuning data, but recall is inconsistent and hallucinates under distribution shift. Use rag for knowledge retrieval.
- Novel reasoning capabilities — if the base model can't reason through a problem type, fine-tuning on examples won't fix the underlying gap
Data Requirements
Quality beats quantity. 100 high-quality, diverse examples outperform 10,000 noisy ones.
- Diversity — cover the full input distribution you expect in production. Fine-tuning on a narrow slice degrades performance outside that slice.
- Consistency — outputs should reflect the exact behavior you want. Contradictory examples confuse the model.
- Format parity — training examples should match production prompt format exactly, including system prompt structure.
A starting point: 50–500 examples is enough to observe a clear shift in style and format. Thousands of examples are needed for reliable behavior on complex reasoning tasks.
The Alternative Argument
For most format and style problems, few-shot prompting with 3–10 well-chosen examples gets you 80% of the way to fine-tuning quality at zero infrastructure cost and instant iteration. Fine-tuning adds a training pipeline, dataset management overhead, and a deployment step for each model update.
Run the numbers:
- If your system prompt is 2,000 tokens and you run 1M requests/day, prompt-caching on the system prompt costs far less than a fine-tuning run and delivers the same latency reduction.
- If behavior is inconsistent across runs, the problem is usually prompt clarity or example quality, not the need for fine-tuning.
Fine-tune when:
- You've already optimized prompts and few-shot examples and still can't hit the quality bar
- You need to reduce system prompt length for latency or cost reasons
- You're deploying at scale where per-token prompt cost compounds significantly
- You need consistently idiomatic outputs in a specialized domain
Evaluation Is Non-Negotiable
Fine-tuning without evals is guessing. Define metrics before training, run them against the base model as a baseline, and track regression on held-out data. It is easy to fine-tune a model to be better at the training distribution and worse at the broader task.