Quick Answer
- Every major model goes through the same rough pipeline: pretraining, fine-tuning, alignment, and increasingly, reasoning training
- Pretraining teaches a model language and general knowledge, everything after that teaches it how to actually be useful and safe to talk to
- The biggest recent shift: most visible gains now come from what happens after pretraining, not from making the base model bigger
- Reasoning models (behind GPT-5.6, Fable 5, and Kimi K3) are trained with reinforcement learning to think step by step, not just prompted to
Stage 1: Pretraining
This is the expensive, slow stage most people picture when they think of "training an AI." The model reads a massive amount of text and learns to predict the next word, over and over, across trillions of examples. Nobody is grading these predictions individually, the model is just absorbing the statistical shape of language, facts, and reasoning patterns buried in the data.
The limitation: a freshly pretrained model can complete text convincingly, but it doesn't know how to be a helpful assistant yet. It's closer to a very well-read autocomplete than a chatbot.
Stage 2: Supervised Fine-Tuning
| Before this stage | After this stage |
|---|---|
| Predicts the next likely word | Answers a question helpfully |
| No sense of "instruction" vs "text to continue" | Recognizes and follows a direct request |
| Learned from raw internet-scale text | Learned from curated instruction-and-response examples |
This is where a model learns the difference between continuing a sentence and actually answering a question. Researchers show it examples of good instruction-and-response pairs, and it learns to imitate that pattern.
The catch: this teaches a model what to say based on examples, not how to figure out a good answer on its own.
Stage 3: Alignment
Once a model can follow instructions, it still needs to learn which answers humans actually prefer, not just which ones sound plausible. A few different techniques accomplish this:
1. RLHF (reinforcement learning from human feedback)
human raters compare different responses and say which is better, training a reward model that guides future behavior
2. Constitutional AI
Anthropic's approach, where written principles guide the model's own self-critique instead of relying purely on human raters for every judgment
3. RLAIF
using AI-generated feedback instead of human feedback, to scale the process without an army of human raters
4. DPO (direct preference optimization)
skips building a separate reward model entirely, adjusting the model directly against preference data
Different mechanics, same goal: shape behavior toward what's actually useful and safe.
Stage 4: Reasoning Training
This is the newest stage, and the one behind the current generation of "reasoning models." Here's roughly how it works:
-
The model attempts a problem multiple different ways in a single training round
-
Each attempt is checked against a verifiable outcome, a correct math answer, a passing test, a working piece of code
-
Whichever approach actually worked gets reinforced, without needing a separate, expensive model just to judge quality
The dominant algorithm behind this is GRPO (Group Relative Policy Optimization), popularized by DeepSeek-R1's breakthrough results and now used broadly across the industry. The scale of improvement is real and measurable: through pure reinforcement learning alone, DeepSeek-R1-Zero's accuracy on the AIME 2024 math competition climbed from 15.6% to 71.0%, and to 86.7% with majority voting, with no additional human-labeled examples involved. Newer setups extend the same idea to agentic behavior, rewarding a model for correctly deciding when to search, call a tool, or take multiple steps toward a goal.
The New Lever: Test-Time Compute
Think of it like the difference between answering a question off the top of your head versus taking five minutes to actually work through it first, same person, same knowledge, different amount of thinking applied.
That's test-time compute: letting a model run more internal reasoning steps at the moment you ask it something, instead of making the underlying model itself bigger. The effect can be dramatic, in one widely cited study, a small 3-billion-parameter model given enough test-time compute outperformed a model over 20 times its size that answered instantly.
This is the actual mechanism behind effort settings now showing up across the industry: Claude Opus 5's low, medium, and high toggle, and GPT-5.6 Sol's expanded scale of none, low, medium, high, xhigh, and max. Switching that setting doesn't switch models, it controls how much reasoning the same model does before answering.
Distillation: Where the Cheaper Tiers Come From
This explains something that otherwise looks strange: how a company can release a flagship model and a noticeably cheaper, faster version of it in the same week.
The idea
a large, expensive "teacher" model's knowledge and reasoning patterns get transferred into a smaller "student" model, instead of training the smaller model from raw data all over again
Why it's worth it
a distilled model captures a meaningful share of the teacher's capability at a fraction of the running cost
Where you've seen this
it's a big part of how tiered families like Sol, Terra, and Luna, or Opus, Sonnet, and Haiku, can share a lineage while landing at very different price points
Why This Actually Matters for You
- Why two models can have wildly different personalities despite similar raw capability: alignment training shapes tone and behavior as much as raw pretraining does
- Why "reasoning mode" costs more and takes longer: it genuinely does more computational work per answer, not just a marketing toggle
- Why newer models keep improving at agentic tasks specifically: a direct result of reinforcement learning setups built around tool use and multi-step decisions, not just bigger training data
None of this requires understanding the math. It just helps to know that "smarter" isn't one dial being turned up, it's several distinct stages, each teaching a model something a raw pile of text never could on its own.
Sources & Official References
- Amazon Web Services, on RLHF, RLAIF, and DPO: aws.amazon.com
- Daily Dose of Data Science, How to Fine-Tune LLMs in 2026, on GRPO and reinforcement fine-tuning: blog.dailydoseofds.com
- Anthropic, on Constitutional AI: anthropic.com
- Taskade, on reasoning models and DeepSeek-R1's measured results: taskade.com
- Ernie's Leisure Code, on how effort settings work in GPT-5.6 Sol: ernie55ernie.github.io
This is a general, simplified explainer of publicly documented AI training methods as of 2026. Specific techniques vary by lab and model, and some details are proprietary and not fully disclosed by any provider.
