LLM 101 study guide · Lesson 9 of 10
Quick Recall
Core terms
This is the lesson's study guide from the FlashFeed app, plus its key facts and a self-test.
What you'll be able to do
- Recall the core architecture terms — Transformer, token, self-attention, and context window — instantly.
- Recall the training terms — pretraining, RLHF, and Chinchilla's ~20-tokens-per-parameter rule.
- Recall the usage terms — RAG, inference, hallucination, few-shot, and emergence.
This final lesson is a fast drill on the terms that anchor everything else in the pack. Once these come without hesitation, you can read an article about AI, follow a technical discussion, or evaluate a product claim and know exactly what's being said. Know this dozen cold and you carry the working core of LLM literacy in your head.
Core terms
Start with the foundations of how the technology works. The Transformer is the 2017 architecture behind nearly every modern LLM — the thing that replaced recurrence with attention and unlocked scale. Its key mechanism, self-attention, is how tokens weigh one another inside a Transformer layer. A token is the unit these models actually process, roughly three-quarters of an English word, and the currency in which context limits and pricing are measured. Pretraining is the first and most expensive stage of building a model: self-supervised next-token prediction over a huge corpus, where the model absorbs its broad knowledge. And RLHF — Reinforcement Learning from Human Feedback — is the alignment step that turns a raw predictor into a helpful assistant by training it on human preferences.
Then the concepts you meet when using a model. RAG (Retrieval-Augmented Generation) means retrieving relevant documents into the prompt so the model answers from grounded sources rather than memory — one of the most widely used defenses against hallucination, which is itself the term for fluent output that is confidently fabricated. Inference is what happens every time you send a message: running the frozen model with no weights changing. The context window is the model's token-limited working memory, the bound on everything it can "see" at once. Few-shot prompting means giving the model examples in the prompt to teach it a task on the fly. Chinchilla is DeepMind's 2022 compute-optimal training result, named for its 70-billion-parameter model, usually boiled down to a rule of thumb of about twenty training tokens per parameter. Two more terms from earlier lessons complete the set: self-attention is how tokens weigh one another inside a Transformer layer, and emergence is the name for capabilities that appear only once a model passes a certain scale (researchers still debate how sharp that threshold really is).
Drill these until each is instant: Transformer and 2017, token and the unit of text, pretraining and next-token training, RLHF and human-feedback alignment, RAG and retrieval grounding, inference and the frozen model, hallucination and confident fabrication, the context window and working memory, Chinchilla and the 20-to-1 ratio, few-shot and in-prompt examples, self-attention and tokens weighing one another, and emergence and capabilities that appear only at scale. Together they form the spine of everything in this pack — the history, the mechanics, the training pipeline, and the craft of actually using these models well.
Key concepts
Token
- The basic unit these models actually process.
- Roughly three-quarters of a word.
- It's the currency in which context limits and pricing are measured.
Inference
- What happens every single time you send a message to a model.
- Running the frozen model with no weights changing at all.
- It's the opposite of the pipeline stages that update parameters — the model just computes an answer.
Matched pairs
Each item on the left goes with the one on the right.
- Transformer
- The 2017 architecture behind nearly every modern LLM
- Token
- The unit LLMs process — about ¾ of a word
- Pretraining
- Self-supervised next-token training on huge corpora
- RLHF
- Aligning a model with human preference feedback
- RAG
- Retrieving documents into the prompt for grounding
- Inference
- Running a frozen model — no weights change
- Hallucination
- Fluent output that is confidently fabricated
- Context window
- The model's token-limited working memory
- Chinchilla
- ~20 training tokens per parameter is compute-optimal
- Few-shot
- Giving examples in the prompt to teach a task
- Self-attention
- How tokens weigh one another inside a Transformer layer
- Emergence
- Capabilities that appear only past a certain model scale
Self-test
Answer each question from memory, then check your work against the answer key.
1. The 2017 Transformer architecture unlocked scale in modern LLMs by replacing what with attention?
- APooling layers
- BRecurrence
- CWord embeddings
- DFeed-forward layers
2. A token is the unit an LLM actually processes. Roughly how much text does one token represent?
- AAbout three-quarters of a word
- BAbout one and a half words
- CAbout a third of a word
- DExactly one word
3. Which term names the model's token-limited working memory — the bound on everything it can "see" at once?
- AFew-shot
- BSelf-attention
- CContext window
- DPretraining
4. Inside a Transformer layer, what does self-attention actually do?
- ARetrieves outside documents into the prompt
- BProcesses the sequence one step at a time, left to right
- CAligns the model with human preference feedback
- DLets tokens weigh one another to decide what matters
5. Three of these terms describe the architecture or its units of text. Which one does NOT?
- ARLHF
- BToken
- CContext window
- DSelf-attention
6. Which term describes how tokens weigh one another inside a Transformer layer?
- AInference
- BFew-shot
- CPretraining
- DSelf-attention
7. What is the mechanism inside a Transformer layer that lets tokens attend to one another?
- ARLHF
- BSelf-attention
- CRAG
- DChinchilla
8. Which stage of building a model is self-supervised next-token prediction over a huge corpus?
- ARetrieval-augmented generation
- BInference
- CPretraining
- DRLHF
9. In the training pipeline, what is RLHF's job?
- AExtend the model's context window
- BAbsorb broad knowledge from a huge corpus
- CGround answers in retrieved documents
- DTurn a raw predictor into a helpful assistant using human preferences
10. DeepMind's Chinchilla scaling result is often boiled down to a compute-optimal rule of thumb. Roughly how many training tokens per parameter does it call for?
- AAbout 5
- BAbout 80
- CAbout 20
- DAbout 300
11. A team plans a 7-billion-parameter model. Applying Chinchilla's compute-optimal rule, roughly how many training tokens should they budget?
- AAbout 140 billion
- BAbout 35 billion
- CAbout 7 billion
- DAbout 1.4 trillion
12. Which statement about pretraining is NOT true?
- AIt uses self-supervised next-token prediction
- BIt trains the model on rankings of human preferences
- CIt is the most expensive stage
- DIt is the first stage of building a model
13. What does the Chinchilla scaling rule recommend as the compute-optimal amount of training data?
- AOne training token per parameter
- BAbout 100 training tokens per parameter
- CAbout 20 training tokens per parameter
- DA fixed 1 trillion tokens regardless of size
14. The Chinchilla rule of thumb is a guideline about the ratio of which two things?
- AHuman feedback examples to RLHF steps
- BAttention heads to Transformer layers
- CContext window size to token count
- DTraining tokens to model parameters
15. A support bot keeps confidently inventing product details that do not exist. Which technique most directly targets that failure?
- AFew-shot prompting
- BRAG
- CInference
- DEmergence
16. Which term describes what happens every time you send a message — running the model with no weights changing?
- AInference
- BRLHF
- CPretraining
- DEmergence
17. Which best describes a hallucination from an LLM?
- AOutput cut off when the token limit is hit
- BOutput the model declines to produce
- CFluent output that is confidently fabricated
- DOutput that repeats the prompt back verbatim
18. You put a handful of worked examples in the prompt to teach the model a task on the fly. What is that called?
- AEmergence
- BFew-shot prompting
- CRAG
- DPretraining
19. Which term names capabilities that appear only past a certain model scale?
- AEmergence
- BInference
- CHallucination
- DFew-shot
20. Teaching a model a new task by including worked examples directly in the prompt, with no retraining, is called what?
- ARLHF
- BPretraining
- CFine-tuning
- DFew-shot prompting
21. In few-shot prompting, where do the task-teaching examples live?
- AIn a separate RAG database
- BIn the retraining dataset
- CIn the model's fine-tuned weights
- DIn the prompt alongside the request
22. What term names capabilities that show up only once a model crosses a scale threshold?
- AEmergence
- BChinchilla ratio
- CHallucination
- DContext window
23. Emergence in LLMs describes which pattern?
- AHuman feedback correcting bad outputs
- BA new skill appearing only above a scale threshold
- CRetrieval improving factual grounding
- DA model forgetting older training data
24. Retrieving documents and injecting them into the prompt so a model answers from grounded sources is called what?
- AFew-shot prompting
- BPretraining
- CRAG
- DRLHF
Go deeper
Answer key
- 1. B RecurrenceThe Transformer replaced recurrence — the sequential, step-by-step processing of RNNs — with attention, letting whole sequences be processed in parallel. That shift is what unlocked the scale behind nearly every modern LLM.
- 2. A About three-quarters of a wordA token averages about ¾ of a word — it is the unit models process and the currency of context limits and pricing. Tokens are usually sub-word pieces, so "exactly one word" is wrong.
- 3. C Context windowThe context window is the token-limited working memory. Self-attention is how tokens weigh one another inside a layer, not the limit on how many tokens fit.
- 4. D Lets tokens weigh one another to decide what mattersSelf-attention is how tokens weigh one another inside a Transformer layer. Step-by-step left-to-right processing describes recurrence — exactly what the Transformer replaced.
- 5. A RLHFRLHF is a training/alignment step — it uses human preference feedback to turn a raw predictor into an assistant. Token, self-attention, and context window all describe the architecture and what it processes.
- 6. D Self-attentionSelf-attention is the mechanism by which tokens weigh one another inside a Transformer layer.
- 7. B Self-attentionSelf-attention is the Transformer mechanism that lets tokens attend to one another.
- 8. C PretrainingPretraining is the first and most expensive stage — self-supervised next-token training on huge corpora, where the model absorbs its broad knowledge. RLHF comes after, using human preferences.
- 9. D Turn a raw predictor into a helpful assistant using human preferencesRLHF — Reinforcement Learning from Human Feedback — is the alignment step: it trains on human preferences to turn a raw next-token predictor into a helpful assistant. Absorbing broad knowledge from a corpus is pretraining's job.
- 10. C About 20Chinchilla's rule of thumb is about 20 training tokens per parameter — the 20-to-1 ratio. The other ratios are far from it.
- 11. A About 140 billionChinchilla calls for ~20 training tokens per parameter, so 7B parameters × 20 ≈ 140B tokens. One token per parameter (7B) or 200-to-1 (1.4T) both miss the 20-to-1 ratio.
- 12. B It trains the model on rankings of human preferencesTraining on human preferences is RLHF, not pretraining. Pretraining is the first and most expensive stage: self-supervised next-token prediction over a huge corpus.
- 13. C About 20 training tokens per parameterThe Chinchilla result boils down to a compute-optimal rule of thumb: roughly twenty training tokens per model parameter.
- 14. D Training tokens to model parametersChinchilla names the compute-optimal ratio of training tokens to model parameters, about 20 to 1.
- 15. B RAGRAG pulls relevant documents into the prompt so the model answers from grounded sources rather than memory — one of the most widely used ways to reduce hallucination. Few-shot adds task examples, which teaches format, not factual grounding.
- 16. A InferenceInference is running the frozen model; nothing about the weights changes. Pretraining and RLHF are training stages, where weights do change.
- 17. C Fluent output that is confidently fabricatedA hallucination is fluent, confident output that is simply fabricated — which is why grounding answers in retrieved sources (RAG) is a widely used defense. Hitting a token limit is a context-window issue, not a hallucination.
- 18. B Few-shot promptingFew-shot prompting means giving examples in the prompt to teach a task on the fly. RAG also adds text to the prompt, but it adds retrieved source documents for grounding, not task examples.
- 19. A EmergenceEmergence is the term for capabilities that show up only once a model passes a certain scale. Inference is just running the frozen model, regardless of its size.
- 20. D Few-shot promptingFew-shot prompting means giving the model examples in the prompt itself to teach it a task on the fly, with no weight updates.
- 21. D In the prompt alongside the requestFew-shot prompting puts example input/output pairs right in the prompt, so the model picks up the pattern without any training step.
- 22. A EmergenceEmergence describes capabilities that appear only past a certain model scale, absent in smaller models.
- 23. B A new skill appearing only above a scale thresholdEmergence is the term for capabilities absent in smaller models that appear once a model is scaled past a certain size.
- 24. C RAGRAG (Retrieval-Augmented Generation) retrieves relevant documents into the prompt so the answer is grounded in that content.