LLM 101 study guide · Lesson 6 of 10
Key Numbers and Milestones
Model sizes and dates · Scaling, context and misconceptions
This is the lesson's study guide from the FlashFeed app, plus its key facts and a self-test.
What you'll be able to do
- Recall the parameter counts of the GPT line (GPT-1, GPT-2, GPT-3) and GPT-3's training-data size.
- Recall the parameter counts of other landmark models (BERT-large, LLaMA's 7B-to-65B family, Mistral 7B and PaLM) and compare them with GPT-3.
- Place the key release years in order: BERT, GPT-2, GPT-3, LLaMA, ChatGPT, and GPT-4.
- Recall the ChatGPT and GPT-4 launch years and ChatGPT's record two-month climb to 100 million users.
- Recall the scaling-law figures (Kaplan year, Chinchilla's ~20 tokens/parameter) and context-window growth, and why bigger isn't always better.
Some facts about LLMs are worth knowing cold — the parameter counts, dates, and ratios that anchor every other discussion. They give you a sense of scale and a timeline to hang everything else on. This lesson drills the numbers, and closes with two of the field's most stubborn misconceptions.
Model sizes and dates
The GPT line is the clearest yardstick of how fast the field scaled. GPT-1 (2018) had 117 million parameters. GPT-2 (2019) jumped to 1.5 billion. GPT-3 (2020) leapt again to 175 billion — more than a hundredfold increase, and the model that revealed in-context learning. To feed that model, GPT-3 was trained on roughly 300 billion tokens of filtered text. The pattern is worth internalizing: each generation grew its parameter count by one to two orders of magnitude, and that growth is much of what drove the capability jumps. Other landmark models fill in the scale. Google's BERT-large had about 340 million parameters (Devlin et al.; released 2018, published at NAACL 2019); Meta's original LLaMA family (2023) ran from 7 billion to 65 billion; and Google's PaLM reached 540 billion (Chowdhery et al., 2022), roughly three times GPT-3.
The dates matter just as much as the sizes. ChatGPT launched on November 30, 2022, and reached an estimated 100 million monthly users within two months (a UBS analysis of Similarweb traffic data) — which UBS analysts called the fastest ramp of any consumer internet app they had followed. GPT-4 followed on March 14, 2023. Holding these milestones in mind — GPT-3 in 2020, ChatGPT in late 2022, GPT-4 in early 2023 — gives you the spine of the modern timeline, the span of under three years in which language models went from research artifact to mass technology.
Scaling, context and misconceptions
Two research results reshaped how the field thinks about size. The Kaplan scaling laws (2020) established that model loss falls predictably with scale. The Chinchilla result (2022) refined it into a practical rule: compute-optimal training uses about 20 tokens of data per parameter, meaning many earlier giant models were badly undertrained. Alongside size, the context window has grown enormously: GPT-3 launched with just 2,048 tokens of working memory, GPT-4 with 8,192 (and a 32K variant soon after) — and later models have pushed into the hundreds of thousands and millions, transforming what tasks are possible.
Those numbers also puncture two common myths. First: a bigger model is not always better. Chinchilla showed a smaller model trained on more data can beat a larger, data-starved one, and InstructGPT's 1.3-billion-parameter model was preferred by humans over the 175-billion GPT-3 — how much data a model is trained on and how well it is aligned matter as much as raw parameter count. Second: fine-tuning is not the same as prompting with examples. Fine-tuning permanently updates the model's weights on disk through more gradient descent; prompting only supplies context within a single inference call and changes no weights at all. They differ fundamentally in cost, persistence, and mechanism — confusing the two is the root of a lot of muddled thinking about how to "teach" a model something.
Key concepts
InstructGPT
- A model used to puncture the myth that a bigger model is always better.
- Its 1.3-billion-parameter version was preferred by humans over a model more than a hundred times larger.
- Human raters favored its output over the 175-billion-parameter flagship, showing alignment can beat raw size.
By the numbers
- GPT-1 parameter count (millions)
- 117 million parameters
- GPT-2 parameter count, largest version (billions)
- 1.5 billion parameters
- GPT-3 parameter count (billions)
- 175 billion parameters
- Year GPT-3 was released
- 2020 year
- Year ChatGPT launched (November 30)
- 2022 year
- Months for ChatGPT to reach 100 million monthly users
- 2 months
- Year GPT-4 was released (March 14)
- 2023 year
- Approximate number of tokens GPT-3 was trained on (billions)
- 300 billion tokens
- Parameter count of BERT-large
- 340 million
- Parameter count of GPT-1
- 117 million
- Parameter count of GPT-2
- 1500 million
- Parameter count of GPT-3
- 175000 million
- Parameter count of LLaMA-7B
- 7000 million
- Meta's original 2023 LLaMA family topped out at how many parameters?
- 65000 million
- Parameter count of Mistral 7B
- 7000 million
- Parameter count of PaLM's largest model
- 540000 million
- Year BERT was released
- 2018 year
- Year GPT-2 was released
- 2019 year
- Year ChatGPT was released
- 2022 year
- Year LLaMA was released by Meta
- 2023 year
- Year GPT-4 was released
- 2023 year
- Chinchilla's compute-optimal training tokens per parameter
- 20 tokens per parameter
- Year the Kaplan scaling-laws paper was published
- 2020 year
- GPT-3's context window at launch (tokens)
- 2048 tokens
- GPT-4's context window at launch in 2023 (tokens)
- 8192 tokens
Self-test
Answer each question from memory, then check your work against the answer key.
1. How many parameters did GPT-1 (2018) have?
- A340 million
- B117 million
- C175 billion
- D1.5 billion
2. The largest version of GPT-2 (2019) had how many parameters?
- A117 million
- B15 billion
- C1.5 billion
- D175 billion
3. How many parameters did GPT-3 (2020) have?
- A175 billion
- B1.5 billion
- C540 billion
- D70 billion
4. GPT-3 was trained on roughly how many tokens of filtered text?
- A40 billion
- B175 billion
- C4 trillion
- D300 billion
5. Using each model's largest released version, which of these model-and-size pairings is NOT correct?
- AGPT-2: 175 billion parameters
- BGPT-1: 117 million parameters
- CGPT-2: 1.5 billion parameters
- DGPT-3: 175 billion parameters
6. Meta's original LLaMA family (2023) spanned which parameter range?
- A300 million to 1 billion
- B7 billion to 65 billion
- C1 million to 10 million
- D175 billion to 540 billion
7. LLaMA-13B, from Meta's original 2023 release, outperformed which larger model on most benchmarks?
- APaLM (540B)
- BChinchilla (70B)
- CGPT-3 (175B)
- DLLaMA-65B
8. Which pair of landmark models share the same nominal parameter count?
- AGPT-2 and InstructGPT
- BLLaMA's largest original model and GPT-3
- CGPT-1 and GPT-2
- DMistral 7B and LLaMA-7B
9. Which of these statements about landmark model sizes is accurate?
- AGPT-2 has about 175 billion parameters
- BMistral 7B has about 70 billion parameters
- CGPT-1 has about 7 billion parameters
- DThe original LLaMA topped out at 65 billion parameters
10. Which of these has the largest parameter count?
- AThe original LLaMA's largest model (65B)
- BGPT-3
- CMistral 7B
- DGPT-1
11. What is the parameter count of Mistral 7B?
- A13 billion
- B540 billion
- C7 billion
- D70 billion
12. Mistral AI's efficient open-weight model, Mistral 7B, has roughly how many parameters?
- A7 billion
- B340 million
- C175 billion
- D1.5 billion
13. Which sequence correctly orders these milestones in time?
- AGPT-3 (2020) → ChatGPT (2022) → GPT-4 (2023)
- BGPT-3 (2020) → GPT-4 (2022) → ChatGPT (2023)
- CChatGPT (2020) → GPT-3 (2022) → GPT-4 (2023)
- DGPT-4 (2020) → GPT-3 (2021) → ChatGPT (2022)
14. In what year was BERT released?
- A2017
- B2018
- C2019
- D2020
15. In what year was GPT-2 released?
- A2020
- B2021
- C2018
- D2019
16. In what year did Meta release LLaMA?
- A2020
- B2021
- C2023
- D2022
17. Which of these models was released earliest?
- ABERT
- BGPT-2
- CGPT-3
- DLLaMA
18. In what year was BERT released?
- A2019
- B2016
- C2018
- D2017
19. Which year marks BERT's release?
- A2019
- B2018
- C2020
- D2021
20. What year was GPT-2 released?
- A2020
- B2021
- C2018
- D2019
21. GPT-2, with 1.5 billion parameters, was released in which year?
- A2021
- B2017
- C2018
- D2019
22. What year was GPT-4 released?
- A2024
- B2021
- C2023
- D2022
23. On what date did ChatGPT launch?
- AMarch 14, 2023
- BNovember 30, 2022
- CNovember 30, 2021
- DJanuary 30, 2023
24. How long did ChatGPT take to reach about 100 million monthly users?
- AAbout 2 months
- BAbout 2 years
- CAbout 12 months
- DAbout 6 months
25. GPT-4 was released on March 14 of which year?
- A2021
- B2023
- C2022
- D2024
26. Roughly how long after ChatGPT's debut did GPT-4 arrive?
- AAbout three and a half months
- BAbout nine months
- CAbout two months
- DAbout two years
27. Which statement about ChatGPT's launch is NOT accurate?
- AIt reached about 100 million monthly users within two months
- BIt launched on November 30, 2022
- CIt took about a year to reach 100 million monthly users
- DIt was the fastest-growing consumer app in history at the time
28. Which finding best supports the claim that 'a bigger model is not always better'?
- AGPT-3 had far more parameters than GPT-2
- BGPT-4 launched with a larger context window than GPT-3
- CKaplan's scaling laws showed loss falls predictably as models grow
- DInstructGPT's 1.3-billion-parameter model was preferred by humans over the 175-billion GPT-3
29. What did Kaplan et al.'s 2020 scaling laws describe?
- ACompute-optimal training needs roughly 20 tokens of data per parameter
- BLoss falls predictably as a power function of compute, data, and parameters
- CLoss improves only in discrete jumps once a model crosses size thresholds
- DLoss stops falling with added compute beyond about 10 billion parameters
30. What did the Chinchilla (2022) result revise about GPT-3?
- AGPT-3's architecture, not its data, capped its performance
- BGPT-3's loss did not follow a power law at all
- CGPT-3 was significantly undertrained for its parameter count
- DGPT-3 had far too many training tokens for its size
31. GPT-3 had 175 billion parameters and was trained on about 300 billion tokens. Roughly how many training tokens per parameter is that?
- AAbout 2
- BAbout 20
- CAbout 300
- DAbout 100
32. Which statement about the Kaplan (2020) and Chinchilla (2022) scaling laws is NOT accurate?
- AChinchilla's 70B-parameter model outperformed the much larger 175B-parameter GPT-3
- BKaplan's laws described how loss falls predictably as models scale up
- CChinchilla revised Kaplan's laws two years after they appeared
- DChinchilla concluded GPT-3 had too few parameters for its training data
33. In what year was the Kaplan scaling-laws paper published?
- A2022
- B2018
- C2019
- D2020
34. The scaling-laws paper establishing that model loss falls predictably with scale appeared in which year?
- A2020
- B2021
- C2017
- D2023
35. According to the Chinchilla result, how many training tokens per parameter is compute-optimal?
- A5
- B20
- C10
- D50
36. Chinchilla showed many earlier giant models were undertrained. Which token-per-parameter ratio did it identify as compute-optimal?
- A400 to 1
- B100 to 1
- C20 to 1
- D1 to 1
Go deeper
Answer key
- 1. B 117 millionGPT-1 had 117 million parameters. 1.5 billion is GPT-2 and 175 billion is GPT-3.
- 2. C 1.5 billionGPT-2 jumped to 1.5 billion parameters — about an order of magnitude over GPT-1's 117 million. 175 billion is GPT-3, a generation later.
- 3. A 175 billionGPT-3 had 175 billion parameters, the model that revealed in-context learning. 1.5 billion is GPT-2, a generation earlier.
- 4. D 300 billionGPT-3 was trained on roughly 300 billion tokens. 175 billion is its parameter count, not its data size — a tempting mix-up.
- 5. A GPT-2: 175 billion parametersGPT-2's largest version had 1.5 billion parameters, not 340 million (that is BERT-large's size). The other three pairings are accurate.
- 6. B 7 billion to 65 billionLLaMA launched with models from 7B to 65B parameters, trained solely on public data (Common Crawl, GitHub, Wikipedia, books).
- 7. C GPT-3 (175B)LLaMA-13B beat GPT-3 (175B) on most standard benchmarks, showing that a smaller model trained on more data can beat raw parameter count.
- 8. D Mistral 7B and LLaMA-7BBoth are nominally 7-billion-parameter models (Mistral 7B is about 7.3B, LLaMA-7B about 6.7B). GPT-1 (117M) and GPT-2 (1.5B) differ, InstructGPT (1.3B) is smaller than GPT-2, and LLaMA's largest (65B) is far below GPT-3 (175B).
- 9. D The original LLaMA topped out at 65 billion parametersThe original LLaMA family (February 2023) topped out at 65 billion parameters. GPT-2 had 1.5 billion, not 175 billion (that is GPT-3), and GPT-1 had 117 million, not 7 billion.
- 10. B GPT-3GPT-3's 175 billion parameters dwarf the original LLaMA's largest model (65B), Mistral 7B (7B), and GPT-1 (117 million) -- all figures taught earlier in the course.
- 11. C 7 billionMistral 7B, one of Mistral AI's flagship open-weight models, has 7 billion parameters.
- 12. A 7 billionMistral 7B carries about 7 billion parameters, matching LLaMA-7B in scale.
- 13. A GPT-3 (2020) → ChatGPT (2022) → GPT-4 (2023)The spine of the modern timeline: GPT-3 in 2020, ChatGPT on Nov 30, 2022, and GPT-4 on March 14, 2023 — the under-three-year stretch in which LLMs went from research artifact to mass technology.
- 14. B 2018BERT was released in 2018, the same year as GPT-1. GPT-2 followed in 2019 and GPT-3 in 2020.
- 15. D 2019GPT-2 arrived in 2019, between GPT-1 (2018) and GPT-3 (2020).
- 16. C 2023LLaMA was released by Meta in 2023, the same year as GPT-4. ChatGPT had launched in late 2022.
- 17. A BERTBERT (2018) came first, then GPT-2 (2019), GPT-3 (2020), and LLaMA (2023).
- 18. C 2018BERT was released in 2018, ahead of GPT-2 (2019) and GPT-3 (2020) in the field's timeline.
- 19. B 2018BERT came out in 2018, ahead of GPT-2 (2019) and GPT-3 (2020).
- 20. D 2019GPT-2 was released in 2019, jumping to 1.5 billion parameters.
- 21. D 2019GPT-2 launched in 2019, between GPT-1 (2018) and GPT-3 (2020).
- 22. C 2023GPT-4 was released on March 14, 2023, following ChatGPT's late-2022 launch.
- 23. B November 30, 2022ChatGPT launched November 30, 2022. March 14, 2023 is GPT-4's release date, a few months later.
- 24. A About 2 monthsUBS analysts, using Similarweb data, estimated ChatGPT reached roughly 100 million monthly users within two months, which they called the fastest ramp of any consumer internet app they had followed.
- 25. B 2023GPT-4 launched March 14, 2023. 2022 is the year ChatGPT debuted, on November 30.
- 26. A About three and a half monthsChatGPT launched November 30, 2022 and GPT-4 followed March 14, 2023 — about three and a half months. Two months is how long ChatGPT took to reach 100 million users.
- 27. C It took about a year to reach 100 million monthly usersThe climb to 100 million monthly users took roughly two months, not a year — which is exactly why it set the growth record.
- 28. D InstructGPT's 1.3-billion-parameter model was preferred by humans over the 175-billion GPT-3Chinchilla (more training data for a smaller model) and InstructGPT (alignment) showed that something other than raw size can win — a 1.3B InstructGPT model was preferred over 175B GPT-3. The other statements are true but don't undercut model size.
- 29. B Loss falls predictably as a power function of compute, data, and parametersKaplan's laws gave a quantitative power-law map of loss versus compute, data, and parameters, justifying ever-larger models. The 20-tokens-per-parameter rule is Chinchilla (2022); the other two contradict the smooth power-law finding.
- 30. C GPT-3 was significantly undertrained for its parameter countChinchilla showed GPT-3 was undertrained given its 175B parameters: the 70B-parameter Chinchilla, trained on about four times as much data as the 280B Gopher at the same compute budget, outperformed both Gopher and GPT-3. Chinchilla revised the power laws rather than rejecting them.
- 31. A About 2300 billion tokens spread across 175 billion parameters is only about 1.7 tokens per parameter. Chinchilla (2022) judged that far too little data for a model that size, which is why it called GPT-3 significantly undertrained.
- 32. D Chinchilla concluded GPT-3 had too few parameters for its training dataChinchilla found the opposite: GPT-3 was undertrained, with too little data for its parameter count, not too few parameters. The other three statements are accurate; the 70B Chinchilla outperformed GPT-3 despite being far smaller.
- 33. D 2020The Kaplan scaling-laws paper, showing loss falls predictably with scale, was published in 2020.
- 34. A 2020Kaplan's scaling-laws paper was published in 2020, two years before Chinchilla refined the picture.
- 35. B 20Chinchilla's rule is about 20 training tokens per parameter for compute-optimal training.
- 36. C 20 to 1Chinchilla's compute-optimal ratio is about 20 tokens per parameter, well above what earlier giant models used.
Sources
- Devlin et al. (2019), BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — aclanthology.org/N19-1423/
- Chowdhery et al. (2022), PaLM: Scaling Language Modeling with Pathways — arxiv.org/abs/2204.02311