LLM 101 study guide · Lesson 3 of 10
The Training Pipeline
Pretraining to RLHF · Alignment, inference and compute
This is the lesson's study guide from the FlashFeed app, plus its key facts and a self-test.
What you'll be able to do
- Explain self-supervised pretraining and why the resulting base model is not yet a helpful assistant.
- Describe supervised fine-tuning as training on prompt/response pairs to make a model follow instructions.
- Explain how RLHF and its reward model align a model, and how RLAIF and Constitutional AI scale that alignment.
- Identify the instruct/chat model as the aligned product of the full base-plus-SFT-plus-RLHF pipeline.
- Distinguish training from inference and relate the FLOPs compute cost to why training a large model is costly but querying it is cheap.
A chat assistant like ChatGPT isn't built in one step. It's the end of a pipeline that starts with a raw text-predictor and, through several distinct stages, shapes it into something helpful, honest, and safe to deploy. Understanding those stages — and the sharp line between building a model and running one — clears up most of the confusion beginners have about how these systems work.
Pretraining to RLHF
It all begins with pretraining. The model trains on a massive, diverse corpus — trillions of tokens from the web, books, and code — using self-supervised next-token prediction, with no human labels required. This is where the overwhelming majority of compute is spent and where the model soaks up its broad world knowledge. The result is a base model: a system that has learned to predict text but has not been taught to follow instructions or be helpful. Hand it a question and it might continue with more questions, because all it knows how to do is extend text. Base models are the raw material, not a finished product, and aren't safe to hand to general users.
The next stage, supervised fine-tuning (SFT), trains the base model on curated (prompt, ideal-response) pairs written or selected by humans, teaching it to follow instructions rather than merely continue text. That alone transforms a text-predictor into something that answers questions. Then comes the stage that made modern assistants feel aligned: RLHF, Reinforcement Learning from Human Feedback. Human raters compare pairs of model outputs; those preferences train a separate reward model that scores how "good" a response is; and the LLM is fine-tuned with reinforcement learning to maximize that score. The reward model's quality effectively caps the quality of alignment — a flawed one just teaches the model to game it. Because human labeling is a bottleneck, a variant called RLAIF (Reinforcement Learning from AI Feedback) has an AI generate the preference labels instead, enabling alignment at larger scale.
Alignment, inference and compute
Anthropic's Constitutional AI builds on that idea: the model critiques and revises its own outputs against a written set of principles — a "constitution" — and an AI then judges pairs of its responses against those principles, producing the preference labels it trains on. The payoff is less reliance on human harm-labeling and a set of guiding principles that are explicit and auditable rather than buried in opaque preference data. The product of this whole pipeline — base model, plus SFT, plus RLHF or equivalent — is an instruct or chat model: one that follows instructions, refuses harmful requests, and converses naturally. This is the kind of model nearly everyone actually interacts with, and it behaves very differently from the base model underneath it.
The single most clarifying distinction in all of this is training versus inference. Training updates the model's weights through gradient descent over billions of examples — done once or rarely, on massive GPU clusters, over weeks or months, at a cost of millions of dollars. Inference is running the finished, frozen model on a prompt — cheap per call, and done billions of times a day. This is precisely why training a large model is expensive while querying one through an API is cheap. It also dispels a common myth: during a chat, the model is not learning. Inference is read-only; weights never change, the knowledge is fixed at the training cutoff, and any apparent "memory" is just text sitting in the current context window. The scale of the training side is measured in FLOPs (floating-point operations) — large models take on the order of 10²³ to 10²⁶ of them (GPT-3 about 3×10²³, GPT-4-class models above 10²⁵) — and those budgets, tied to scaling laws, determine what any organization can afford to build.
Key terms
- pretraining
- Stage one: the model trains on a massive, diverse text corpus (trillions of tokens from the web, books, and code) using self-supervised next-token prediction, with no human labels. This produces a 'base model.' It matters because pretraining is where the vast majority of compute is spent and where the model acquires its broad world knowledge.
- base model
- A pretrained-only LLM that has learned to predict text but hasn't been tuned to follow instructions or be helpful; it simply continues a prompt, often in unexpected ways. It matters because it is the foundation every chat model is built on — and is not safe to deploy directly to general users.
- supervised fine-tuning (SFT)
- Stage two: the base model is trained on a curated set of (prompt, ideal-response) pairs written or selected by humans, teaching it to follow instructions rather than just continue text. It matters because SFT is what turns a raw text-predictor into an assistant that answers questions in a useful format.
- RLHF
- Reinforcement Learning from Human Feedback — stage three: human raters compare pairs of outputs, those preferences train a reward model, and the LLM is then fine-tuned with reinforcement learning to maximize the reward model's score. It matters because RLHF is what makes ChatGPT-style models markedly more helpful and aligned than raw base models.
- reward model
- A separate neural network trained on human preference data that scores how good a model response is, used as the optimization target during the RL phase of RLHF. It matters because the reward model's quality caps the quality of alignment — a flawed one teaches the LLM to 'game' it rather than genuinely improve.
- RLAIF
- Reinforcement Learning from AI Feedback — a variant of RLHF where an AI model, rather than human raters, generates the preference labels used to train the reward model. It matters because it enables alignment at scales where human labeling would become the bottleneck.
- Constitutional AI (CAI)
- Anthropic's alignment method in which the model critiques and revises its own outputs against a written set of principles (a 'constitution'), then trains on those AI-generated preferences via RLAIF. It matters because it reduces reliance on human harm-labeling and makes the model's guiding principles explicit and auditable.
- instruct / chat model
- An LLM that has gone through SFT and RLHF (or equivalent) on top of a base model, making it follow instructions, refuse harmful requests, and converse naturally — for example, ChatGPT. It matters because this is the type of model most people interact with; base models and chat models behave very differently.
- inference
- Running a trained, frozen model on a new input to produce an output, with no weights changed. It matters because inference is what happens every time you send a message; the model's knowledge is locked at training time unless supplemented by tools or RAG.
- FLOPs
- Floating Point Operations — a measure of computational work. Training a large LLM has taken on the order of 10^23 to 10^26 FLOPs (GPT-3 about 3×10^23; GPT-4-class models above 10^25), equal to tens of thousands to tens of millions of GPU-hours and up to tens of millions of dollars of compute. It matters because FLOPs budgets determine what model sizes an organization can afford, and scaling laws relate FLOPs to expected performance.
Key concepts
Pretraining
- The stage where the overwhelming majority of a model's compute is spent.
- It trains on trillions of tokens from web, books, and code using self-supervised next-token prediction, with no human labels.
- Its output is a base model with broad world knowledge that hasn't yet been taught to follow instructions.
Supervised fine-tuning
- The pipeline stage that first turns a raw text-predictor into something that answers questions.
- It trains a base model on curated pairs of a prompt and its ideal response, written or selected by humans.
- It comes right after pretraining and just before the RLHF stage, teaching the model to follow directions rather than merely continue text.
RLHF
- The training stage that made modern assistants finally feel aligned.
- Human raters compare pairs of outputs; those preferences train a reward model the LLM is then tuned to satisfy.
- It is the stage that follows supervised fine-tuning; a variant swaps the human raters for an AI that generates the preference labels.
Constitutional AI
- Anthropic's alignment approach, built on top of the AI-feedback idea.
- The model critiques and revises its own outputs against an explicit, written set of guiding principles.
- The payoff is far less reliance on human harm-labeling, and a value set that's auditable rather than buried in opaque preference data.
Instruct or chat model
- The finished product of the whole pipeline: base model, plus SFT, plus RLHF or its equivalent.
- It follows directions, refuses harmful requests, and converses naturally.
- It's the kind of model nearly everyone actually interacts with, behaving very differently from the raw base model underneath.
Key facts
Training vs. inference — the key distinction
TRAINING updates a model's weights with gradient descent over billions of examples — done once or rarely, on massive GPU clusters over weeks or months. INFERENCE runs the finished, frozen model on a prompt — cheap per call, done billions of times a day. This is why training a large model is costly but querying an API is cheap.
Constitutional AI's auditability payoff
Constitutional AI's written constitution gives explicit, auditable guiding principles instead of preferences buried in opaque data — on top of reducing reliance on human harm-labeling.
Base model plus alignment
An instruct or chat model is produced by taking a pretrained base model and adding supervised fine-tuning plus RLHF, or an equivalent alignment step.
Step by step
Fine-tune an LLM on instruction data
- Curate a small, high-quality dataset of (prompt, ideal response) pairs.
- Format every example the same way, so the model always sees the prompt and the ideal response in a consistent layout.
- Start from the pretrained base model and train it on the pairs, so its weights update toward following instructions.
- Train for a few epochs with a low learning rate.
- Test the model on held-out prompts it never trained on, and check that it answers them rather than just continuing the text.
- Deploy the fine-tuned model, which now follows instructions instead of merely continuing text.
Self-test
Answer each question from memory, then check your work against the answer key.
1. Which statement about the pretraining stage is correct?
- AIt uses human preference rankings to train a reward model
- BIt uses self-supervised next-token prediction with no human labels, and consumes most of the total training compute
- CIt trains on human-written (prompt, ideal-response) pairs to teach instruction-following
- DIt is the cheapest and fastest stage of the pipeline
2. Pretraining is described as "self-supervised." What does that mean in this context?
- AHumans write ideal responses that the model is trained to copy
- BThe model supervises a smaller student model and distills its knowledge into it
- CThe training signal comes from the text itself — predicting the next token — so no human labels are needed
- DHuman raters supervise the model by ranking its outputs against each other
3. You hand a raw base model the question "What causes tides?" What is its most likely behavior?
- AIt may continue with more questions, since all it knows how to do is extend text
- BIt answers helpfully and concisely, then offers a follow-up
- CIt refuses, because base models are trained to decline factual questions
- DIt returns an error, because base models cannot process prompts
4. Which is NOT true of the base model produced by pretraining?
- AIt is unsafe to hand to general users as a finished product
- BIt has soaked up broad world knowledge
- CIt has learned to predict text
- DIt has been taught to follow instructions and be helpful
5. What is the pretraining corpus described as, and how much of the pipeline's compute does the stage consume?
- AMillions of human preference comparisons; roughly half the compute
- BBillions of tokens of code only; the smallest share of compute
- CThousands of curated prompt/response pairs; a small fraction of compute
- DTrillions of tokens from the web, books, and code; the overwhelming majority of compute
6. What data does supervised fine-tuning train on?
- AAI-generated critiques of the model's own responses
- BPairs of model outputs ranked against each other by human raters
- CCurated (prompt, ideal-response) pairs written or selected by humans
- DA massive unlabeled corpus of web text scraped at random
7. What capability does SFT add that pretraining alone does not?
- AFollowing instructions rather than merely continuing text
- BPredicting the next token in a sequence
- CAbsorbing broad world knowledge from a large corpus
- DRunning cheaply at inference time without changing weights
8. Where does SFT sit in the training pipeline?
- AIt runs in parallel with pretraining on the same corpus
- BStage two — after pretraining produces the base model, before RLHF
- CStage three — after RLHF has already aligned the model
- DStage one — it is the very first training the model receives
9. A team has a base model that rambles instead of answering. Which stage is designed to fix exactly that, and why?
- APretraining — more tokens will eventually teach it to answer directly
- BSFT — training on human-written ideal responses teaches it to answer the prompt
- CInference — running the frozen model more often teaches it to answer directly
- DReward modeling alone — a scoring network by itself changes the model's behavior
10. Which is NOT part of the description of supervised fine-tuning?
- AHuman raters compare pairs of outputs to express which they prefer
- BIt teaches the model to follow instructions
- CIt starts from the base model
- DIts training pairs are written or selected by humans
11. Put the RLHF procedure in the correct order.
- ARaters write ideal responses → the LLM copies them → a reward model scores the copies
- BThe LLM is fine-tuned with RL → raters compare the results → a reward model is discarded
- CRaters compare output pairs → preferences train a reward model → RL fine-tunes the LLM to maximize its score
- DA reward model is trained first → raters check its scores → the LLM is frozen
12. What is the reward model, and what is its limitation?
- AThe LLM itself, rescoring its own outputs; it is capped by the context window
- BA copy of the base model; it is capped by the training cutoff date
- CA rule-based filter of banned words; it is capped by the size of the block list
- DA separate network scoring response quality; its quality caps the quality of alignment
13. RLAIF differs from RLHF in which single respect?
- AIt trains the reward model on the SFT dataset instead of on preference comparisons
- BIt runs before pretraining rather than after it
- CAn AI generates the preference labels instead of human raters
- DIt replaces reinforcement learning with supervised fine-tuning
14. In Constitutional AI, how does the model use a written set of principles to improve itself?
- AIt compares its outputs to a frozen copy of the base model
- BIt critiques and revises its own outputs against a written set of principles
- CIt polls a panel of human harm-labelers for every response
- DIt retrieves rulings from a legal database at inference time
15. What payoff does Constitutional AI's written constitution deliver?
- AGuiding principles that are explicit and auditable rather than buried in opaque preference data
- BA guarantee that the model can never produce a harmful output
- CLower inference cost per API call for the finished model
- DElimination of the reward model from the alignment pipeline
16. Which combination produces an instruct/chat model?
- AA base model given a longer context window
- BSFT + a reward model, with no pretraining stage
- CBase model + pretraining on a second corpus
- DBase model + SFT + RLHF (or equivalent)
17. Which behavior is characteristic of an instruct/chat model but NOT of the base model underneath it?
- APredicting likely next tokens
- BRunning with frozen weights at inference
- CRefusing harmful requests
- DDrawing on broad world knowledge from its corpus
18. Which trio of behaviors characterizes an instruct/chat model such as ChatGPT?
- APredicts next tokens, extends text, continues prompts in unexpected ways
- BCritiques its own outputs, revises them, trains on its own preferences
- CScores responses, ranks output pairs, serves as an optimization target
- DFollows instructions, refuses harmful requests, converses naturally
19. Which kind of model does nearly everyone actually interact with?
- AThe reward model, since it decides what counts as a good answer
- BThe instruct/chat model, which behaves very differently from the base model underneath
- CThe base model, since it holds all the world knowledge
- DA model still in training, whose weights update as you chat
20. A startup downloads a base model and ships it to consumers as a chatbot with no further training. What should the startup expect?
- AIt won't reliably follow instructions or refuse harm — base models are raw material, not a finished product
- BIt will be unable to generate any text at all without SFT
- CIt will behave identically to ChatGPT, since the world knowledge is already there
- DIt will keep learning from user chats and align itself over time
21. During a normal chat, does an LLM learn from your messages and update itself?
- ANo — the model never actually processes your messages; it only replays memorized training text
- BYes — each message updates the model's weights through gradient descent
- CYes — the model fine-tunes itself on new facts after every conversation
- DNo — inference is read-only; the weights stay frozen and any 'memory' is just text in the current context window
22. Which pairing correctly matches each process to its description?
- ATraining updates weights via gradient descent; inference runs the frozen model on a prompt
- BTraining runs the frozen model on a prompt; inference updates weights via gradient descent
- CBoth training and inference update weights, differing only in scale
- DNeither changes weights; they differ only in how much text they output
23. Why is training a large model expensive while querying it through an API is cheap?
- AAPI queries are subsidized, while training is billed at true cost
- BTraining runs on massive GPU clusters for weeks or months at a cost of millions; inference is cheap per call
- CTraining runs billions of times a day, while queries run only once
- DTraining must re-download the corpus on every request
24. Roughly how many FLOPs does training a large LLM take?
- AOn the order of 10^28 to 10^31
- BOn the order of 10^12 to 10^14
- COn the order of 10^23 to 10^26
- DOn the order of 10^17 to 10^19
25. What does FLOPs stand for, and why do FLOPs budgets matter?
- AFast Local Optimization Passes — they set how quickly a model answers a prompt
- BFull-Length Output Predictions — they cap how long a response can be
- CFrozen Layer Operations — they count the layers skipped during inference
- DFloating Point Operations — tied to scaling laws, they determine what an org can afford to build
Go deeper
Answer key
- 1. B It uses self-supervised next-token prediction with no human labels, and consumes most of the total training computePretraining runs self-supervised next-token prediction over trillions of tokens with no human labels, giving the model its broad world knowledge — and it's where most compute is spent. Learning from labeled pairs is SFT; preference rankings feed RLHF.
- 2. C The training signal comes from the text itself — predicting the next token — so no human labels are neededSelf-supervised means the corpus supplies its own labels: the next token is the answer. Ranking outputs is RLHF; writing ideal responses is SFT.
- 3. A It may continue with more questions, since all it knows how to do is extend textA base model only predicts continuations — a question can plausibly be followed by more questions. Helpful answering comes from SFT; refusal behavior comes from RLHF.
- 4. D It has been taught to follow instructions and be helpfulPretraining yields broad world knowledge and text prediction, but explicitly does NOT teach instruction-following or helpfulness — that requires SFT and RLHF. Base models are raw material, not safe for general users.
- 5. D Trillions of tokens from the web, books, and code; the overwhelming majority of computePretraining uses trillions of tokens from web, books, and code, and it is where the overwhelming majority of compute is spent. Curated pairs are SFT; preference comparisons are RLHF.
- 6. C Curated (prompt, ideal-response) pairs written or selected by humansSFT uses curated prompt/ideal-response pairs. Unlabeled web text is pretraining, ranked output pairs are RLHF, and AI self-critiques are Constitutional AI.
- 7. A Following instructions rather than merely continuing textSFT teaches instruction-following — it transforms a text-predictor into something that answers questions. Next-token prediction and world knowledge already come from pretraining.
- 8. B Stage two — after pretraining produces the base model, before RLHFThe pipeline runs pretraining (stage one) → SFT (stage two) → RLHF (stage three). SFT starts from the base model pretraining produced.
- 9. B SFT — training on human-written ideal responses teaches it to answer the promptSFT is the stage that turns a rambling text-predictor into something that follows instructions. More pretraining just improves prediction; a reward model only scores — the RL phase is what updates the LLM.
- 10. A Human raters compare pairs of outputs to express which they preferComparing pairs of outputs is the human-preference step of RLHF, not SFT. SFT trains on human-authored prompt/ideal-response pairs starting from the base model.
- 11. C Raters compare output pairs → preferences train a reward model → RL fine-tunes the LLM to maximize its scoreRLHF runs: human pairwise comparisons → those preferences train a separate reward model → reinforcement learning fine-tunes the LLM to maximize that reward model's score.
- 12. D A separate network scoring response quality; its quality caps the quality of alignmentThe reward model is a separate network trained on human preference data, and its quality effectively caps alignment quality — a flawed one just teaches the model to game it.
- 13. C An AI generates the preference labels instead of human ratersRLAIF is RLHF with AI-generated preference labels, which removes the human-labeling bottleneck and enables alignment at larger scale. In the standard formulation the reward model and RL phase remain.
- 14. B It critiques and revises its own outputs against a written set of principlesIn Anthropic's Constitutional AI the model self-critiques and revises its outputs against a written constitution, and AI-generated preference judgments against those principles then train the model — reducing reliance on human harm-labeling.
- 15. A Guiding principles that are explicit and auditable rather than buried in opaque preference dataThe payoff is less reliance on human harm-labeling plus principles that are explicit and auditable, unlike opaque preference data. CAI is an alignment method — it does not change per-call inference cost.
- 16. D Base model + SFT + RLHF (or equivalent)The instruct/chat model is the product of the full pipeline: a pretrained base model, plus SFT, plus RLHF or an equivalent alignment step. A bigger context window changes nothing about alignment.
- 17. C Refusing harmful requestsRefusal behavior is installed by alignment (RLHF or equivalent). Token prediction, world knowledge, and frozen-weight inference are all true of the base model too.
- 18. D Follows instructions, refuses harmful requests, converses naturallyThe instruct/chat model is defined by exactly those three behaviors. Extending text unexpectedly describes the base model; scoring and ranking describe the reward model; self-critique describes Constitutional AI.
- 19. B The instruct/chat model, which behaves very differently from the base model underneathThe instruct/chat model is what nearly everyone interacts with, and it behaves very differently from the base model it is built on. Base models aren't safe to hand to general users, and no model learns as you chat: inference is read-only.
- 20. A It won't reliably follow instructions or refuse harm — base models are raw material, not a finished productBase models are the raw material, not a finished product, and aren't safe for general users — they lack instruction-following and refusal. They also don't learn during chats; inference is read-only.
- 21. D No — inference is read-only; the weights stay frozen and any 'memory' is just text in the current context windowTraining updates weights (rare, costly); inference just runs the frozen model and is read-only. The model isn't learning as you chat — its knowledge is fixed at the training cutoff, and any 'memory' is only the text sitting in the context window.
- 22. A Training updates weights via gradient descent; inference runs the frozen model on a promptTraining updates weights through gradient descent over billions of examples; inference runs the finished, frozen model. That asymmetry is the single most clarifying distinction between the two.
- 23. B Training runs on massive GPU clusters for weeks or months at a cost of millions; inference is cheap per callThe cost asymmetry is structural: training is done once or rarely on huge clusters over weeks or months for millions of dollars, while inference is cheap per call — and it's inference that runs billions of times a day.
- 24. C On the order of 10^23 to 10^26Training a large model takes roughly 10^23 to 10^26 FLOPs (GPT-3 about 3×10^23, GPT-4-class models above 10^25) — budgets that determine what any organization can afford to build.
- 25. D Floating Point Operations — tied to scaling laws, they determine what an org can afford to buildFLOPs are Floating Point Operations, a measure of computational work. Because they tie to scaling laws, FLOPs budgets determine what model sizes an organization can afford to train.
Image credits
- Training vs. inference — the key distinction: FlashFeed original diagram · CC-BY-4.0
- Base model plus alignment: FlashFeed original diagram · CC-BY-4.0