LLM 101 study guide · Lesson 6 of 10

Key Numbers and Milestones

Model sizes and dates · Scaling, context and misconceptions

This is the lesson's study guide from the FlashFeed app, plus its key facts and a self-test.

What you'll be able to do

  • Recall the parameter counts of the GPT line (GPT-1, GPT-2, GPT-3) and GPT-3's training-data size.
  • Recall the parameter counts of other landmark models (BERT-large, LLaMA's 7B-to-65B family, Mistral 7B and PaLM) and compare them with GPT-3.
  • Place the key release years in order: BERT, GPT-2, GPT-3, LLaMA, ChatGPT, and GPT-4.
  • Recall the ChatGPT and GPT-4 launch years and ChatGPT's record two-month climb to 100 million users.
  • Recall the scaling-law figures (Kaplan year, Chinchilla's ~20 tokens/parameter) and context-window growth, and why bigger isn't always better.

Some facts about LLMs are worth knowing cold — the parameter counts, dates, and ratios that anchor every other discussion. They give you a sense of scale and a timeline to hang everything else on. This lesson drills the numbers, and closes with two of the field's most stubborn misconceptions.

Model sizes and dates

The GPT line is the clearest yardstick of how fast the field scaled. GPT-1 (2018) had 117 million parameters. GPT-2 (2019) jumped to 1.5 billion. GPT-3 (2020) leapt again to 175 billion — more than a hundredfold increase, and the model that revealed in-context learning. To feed that model, GPT-3 was trained on roughly 300 billion tokens of filtered text. The pattern is worth internalizing: each generation grew its parameter count by one to two orders of magnitude, and that growth is much of what drove the capability jumps. Other landmark models fill in the scale. Google's BERT-large had about 340 million parameters (Devlin et al.; released 2018, published at NAACL 2019); Meta's original LLaMA family (2023) ran from 7 billion to 65 billion; and Google's PaLM reached 540 billion (Chowdhery et al., 2022), roughly three times GPT-3.

The dates matter just as much as the sizes. ChatGPT launched on November 30, 2022, and reached an estimated 100 million monthly users within two months (a UBS analysis of Similarweb traffic data) — which UBS analysts called the fastest ramp of any consumer internet app they had followed. GPT-4 followed on March 14, 2023. Holding these milestones in mind — GPT-3 in 2020, ChatGPT in late 2022, GPT-4 in early 2023 — gives you the spine of the modern timeline, the span of under three years in which language models went from research artifact to mass technology.

Scaling, context and misconceptions

Two research results reshaped how the field thinks about size. The Kaplan scaling laws (2020) established that model loss falls predictably with scale. The Chinchilla result (2022) refined it into a practical rule: compute-optimal training uses about 20 tokens of data per parameter, meaning many earlier giant models were badly undertrained. Alongside size, the context window has grown enormously: GPT-3 launched with just 2,048 tokens of working memory, GPT-4 with 8,192 (and a 32K variant soon after) — and later models have pushed into the hundreds of thousands and millions, transforming what tasks are possible.

Those numbers also puncture two common myths. First: a bigger model is not always better. Chinchilla showed a smaller model trained on more data can beat a larger, data-starved one, and InstructGPT's 1.3-billion-parameter model was preferred by humans over the 175-billion GPT-3 — how much data a model is trained on and how well it is aligned matter as much as raw parameter count. Second: fine-tuning is not the same as prompting with examples. Fine-tuning permanently updates the model's weights on disk through more gradient descent; prompting only supplies context within a single inference call and changes no weights at all. They differ fundamentally in cost, persistence, and mechanism — confusing the two is the root of a lot of muddled thinking about how to "teach" a model something.

Key concepts

InstructGPT

  • A model used to puncture the myth that a bigger model is always better.
  • Its 1.3-billion-parameter version was preferred by humans over a model more than a hundred times larger.
  • Human raters favored its output over the 175-billion-parameter flagship, showing alignment can beat raw size.

By the numbers

GPT-1 parameter count (millions)
117 million parameters
GPT-2 parameter count, largest version (billions)
1.5 billion parameters
GPT-3 parameter count (billions)
175 billion parameters
Year GPT-3 was released
2020 year
Year ChatGPT launched (November 30)
2022 year
Months for ChatGPT to reach 100 million monthly users
2 months
Year GPT-4 was released (March 14)
2023 year
Approximate number of tokens GPT-3 was trained on (billions)
300 billion tokens
Parameter count of BERT-large
340 million
Parameter count of GPT-1
117 million
Parameter count of GPT-2
1500 million
Parameter count of GPT-3
175000 million
Parameter count of LLaMA-7B
7000 million
Meta's original 2023 LLaMA family topped out at how many parameters?
65000 million
Parameter count of Mistral 7B
7000 million
Parameter count of PaLM's largest model
540000 million
Year BERT was released
2018 year
Year GPT-2 was released
2019 year
Year ChatGPT was released
2022 year
Year LLaMA was released by Meta
2023 year
Year GPT-4 was released
2023 year
Chinchilla's compute-optimal training tokens per parameter
20 tokens per parameter
Year the Kaplan scaling-laws paper was published
2020 year
GPT-3's context window at launch (tokens)
2048 tokens
GPT-4's context window at launch in 2023 (tokens)
8192 tokens

Self-test

Answer each question from memory, then check your work against the answer key.

  1. 1. How many parameters did GPT-1 (2018) have?

    1. A340 million
    2. B117 million
    3. C175 billion
    4. D1.5 billion
  1. 2. The largest version of GPT-2 (2019) had how many parameters?

    1. A117 million
    2. B15 billion
    3. C1.5 billion
    4. D175 billion
  2. 3. How many parameters did GPT-3 (2020) have?

    1. A175 billion
    2. B1.5 billion
    3. C540 billion
    4. D70 billion
  3. 4. GPT-3 was trained on roughly how many tokens of filtered text?

    1. A40 billion
    2. B175 billion
    3. C4 trillion
    4. D300 billion
  4. 5. Using each model's largest released version, which of these model-and-size pairings is NOT correct?

    1. AGPT-2: 175 billion parameters
    2. BGPT-1: 117 million parameters
    3. CGPT-2: 1.5 billion parameters
    4. DGPT-3: 175 billion parameters
  5. 6. Meta's original LLaMA family (2023) spanned which parameter range?

    1. A300 million to 1 billion
    2. B7 billion to 65 billion
    3. C1 million to 10 million
    4. D175 billion to 540 billion
  6. 7. LLaMA-13B, from Meta's original 2023 release, outperformed which larger model on most benchmarks?

    1. APaLM (540B)
    2. BChinchilla (70B)
    3. CGPT-3 (175B)
    4. DLLaMA-65B
  7. 8. Which pair of landmark models share the same nominal parameter count?

    1. AGPT-2 and InstructGPT
    2. BLLaMA's largest original model and GPT-3
    3. CGPT-1 and GPT-2
    4. DMistral 7B and LLaMA-7B
  8. 9. Which of these statements about landmark model sizes is accurate?

    1. AGPT-2 has about 175 billion parameters
    2. BMistral 7B has about 70 billion parameters
    3. CGPT-1 has about 7 billion parameters
    4. DThe original LLaMA topped out at 65 billion parameters
  9. 10. Which of these has the largest parameter count?

    1. AThe original LLaMA's largest model (65B)
    2. BGPT-3
    3. CMistral 7B
    4. DGPT-1
  10. 11. What is the parameter count of Mistral 7B?

    1. A13 billion
    2. B540 billion
    3. C7 billion
    4. D70 billion
  11. 12. Mistral AI's efficient open-weight model, Mistral 7B, has roughly how many parameters?

    1. A7 billion
    2. B340 million
    3. C175 billion
    4. D1.5 billion
  12. 13. Which sequence correctly orders these milestones in time?

    1. AGPT-3 (2020) → ChatGPT (2022) → GPT-4 (2023)
    2. BGPT-3 (2020) → GPT-4 (2022) → ChatGPT (2023)
    3. CChatGPT (2020) → GPT-3 (2022) → GPT-4 (2023)
    4. DGPT-4 (2020) → GPT-3 (2021) → ChatGPT (2022)
  13. 14. In what year was BERT released?

    1. A2017
    2. B2018
    3. C2019
    4. D2020
  14. 15. In what year was GPT-2 released?

    1. A2020
    2. B2021
    3. C2018
    4. D2019
  15. 16. In what year did Meta release LLaMA?

    1. A2020
    2. B2021
    3. C2023
    4. D2022
  16. 17. Which of these models was released earliest?

    1. ABERT
    2. BGPT-2
    3. CGPT-3
    4. DLLaMA
  17. 18. In what year was BERT released?

    1. A2019
    2. B2016
    3. C2018
    4. D2017
  18. 19. Which year marks BERT's release?

    1. A2019
    2. B2018
    3. C2020
    4. D2021
  19. 20. What year was GPT-2 released?

    1. A2020
    2. B2021
    3. C2018
    4. D2019
  20. 21. GPT-2, with 1.5 billion parameters, was released in which year?

    1. A2021
    2. B2017
    3. C2018
    4. D2019
  21. 22. What year was GPT-4 released?

    1. A2024
    2. B2021
    3. C2023
    4. D2022
  22. 23. On what date did ChatGPT launch?

    1. AMarch 14, 2023
    2. BNovember 30, 2022
    3. CNovember 30, 2021
    4. DJanuary 30, 2023
  23. 24. How long did ChatGPT take to reach about 100 million monthly users?

    1. AAbout 2 months
    2. BAbout 2 years
    3. CAbout 12 months
    4. DAbout 6 months
  24. 25. GPT-4 was released on March 14 of which year?

    1. A2021
    2. B2023
    3. C2022
    4. D2024
  25. 26. Roughly how long after ChatGPT's debut did GPT-4 arrive?

    1. AAbout three and a half months
    2. BAbout nine months
    3. CAbout two months
    4. DAbout two years
  26. 27. Which statement about ChatGPT's launch is NOT accurate?

    1. AIt reached about 100 million monthly users within two months
    2. BIt launched on November 30, 2022
    3. CIt took about a year to reach 100 million monthly users
    4. DIt was the fastest-growing consumer app in history at the time
  27. 28. Which finding best supports the claim that 'a bigger model is not always better'?

    1. AGPT-3 had far more parameters than GPT-2
    2. BGPT-4 launched with a larger context window than GPT-3
    3. CKaplan's scaling laws showed loss falls predictably as models grow
    4. DInstructGPT's 1.3-billion-parameter model was preferred by humans over the 175-billion GPT-3
  28. 29. What did Kaplan et al.'s 2020 scaling laws describe?

    1. ACompute-optimal training needs roughly 20 tokens of data per parameter
    2. BLoss falls predictably as a power function of compute, data, and parameters
    3. CLoss improves only in discrete jumps once a model crosses size thresholds
    4. DLoss stops falling with added compute beyond about 10 billion parameters
  29. 30. What did the Chinchilla (2022) result revise about GPT-3?

    1. AGPT-3's architecture, not its data, capped its performance
    2. BGPT-3's loss did not follow a power law at all
    3. CGPT-3 was significantly undertrained for its parameter count
    4. DGPT-3 had far too many training tokens for its size
  30. 31. GPT-3 had 175 billion parameters and was trained on about 300 billion tokens. Roughly how many training tokens per parameter is that?

    1. AAbout 2
    2. BAbout 20
    3. CAbout 300
    4. DAbout 100
  31. 32. Which statement about the Kaplan (2020) and Chinchilla (2022) scaling laws is NOT accurate?

    1. AChinchilla's 70B-parameter model outperformed the much larger 175B-parameter GPT-3
    2. BKaplan's laws described how loss falls predictably as models scale up
    3. CChinchilla revised Kaplan's laws two years after they appeared
    4. DChinchilla concluded GPT-3 had too few parameters for its training data
  32. 33. In what year was the Kaplan scaling-laws paper published?

    1. A2022
    2. B2018
    3. C2019
    4. D2020
  33. 34. The scaling-laws paper establishing that model loss falls predictably with scale appeared in which year?

    1. A2020
    2. B2021
    3. C2017
    4. D2023
  34. 35. According to the Chinchilla result, how many training tokens per parameter is compute-optimal?

    1. A5
    2. B20
    3. C10
    4. D50
  35. 36. Chinchilla showed many earlier giant models were undertrained. Which token-per-parameter ratio did it identify as compute-optimal?

    1. A400 to 1
    2. B100 to 1
    3. C20 to 1
    4. D1 to 1

Go deeper

Answer key

  1. 1. B 117 millionGPT-1 had 117 million parameters. 1.5 billion is GPT-2 and 175 billion is GPT-3.
  2. 2. C 1.5 billionGPT-2 jumped to 1.5 billion parameters — about an order of magnitude over GPT-1's 117 million. 175 billion is GPT-3, a generation later.
  3. 3. A 175 billionGPT-3 had 175 billion parameters, the model that revealed in-context learning. 1.5 billion is GPT-2, a generation earlier.
  4. 4. D 300 billionGPT-3 was trained on roughly 300 billion tokens. 175 billion is its parameter count, not its data size — a tempting mix-up.
  5. 5. A GPT-2: 175 billion parametersGPT-2's largest version had 1.5 billion parameters, not 340 million (that is BERT-large's size). The other three pairings are accurate.
  6. 6. B 7 billion to 65 billionLLaMA launched with models from 7B to 65B parameters, trained solely on public data (Common Crawl, GitHub, Wikipedia, books).
  7. 7. C GPT-3 (175B)LLaMA-13B beat GPT-3 (175B) on most standard benchmarks, showing that a smaller model trained on more data can beat raw parameter count.
  8. 8. D Mistral 7B and LLaMA-7BBoth are nominally 7-billion-parameter models (Mistral 7B is about 7.3B, LLaMA-7B about 6.7B). GPT-1 (117M) and GPT-2 (1.5B) differ, InstructGPT (1.3B) is smaller than GPT-2, and LLaMA's largest (65B) is far below GPT-3 (175B).
  9. 9. D The original LLaMA topped out at 65 billion parametersThe original LLaMA family (February 2023) topped out at 65 billion parameters. GPT-2 had 1.5 billion, not 175 billion (that is GPT-3), and GPT-1 had 117 million, not 7 billion.
  10. 10. B GPT-3GPT-3's 175 billion parameters dwarf the original LLaMA's largest model (65B), Mistral 7B (7B), and GPT-1 (117 million) -- all figures taught earlier in the course.
  11. 11. C 7 billionMistral 7B, one of Mistral AI's flagship open-weight models, has 7 billion parameters.
  12. 12. A 7 billionMistral 7B carries about 7 billion parameters, matching LLaMA-7B in scale.
  13. 13. A GPT-3 (2020) → ChatGPT (2022) → GPT-4 (2023)The spine of the modern timeline: GPT-3 in 2020, ChatGPT on Nov 30, 2022, and GPT-4 on March 14, 2023 — the under-three-year stretch in which LLMs went from research artifact to mass technology.
  14. 14. B 2018BERT was released in 2018, the same year as GPT-1. GPT-2 followed in 2019 and GPT-3 in 2020.
  15. 15. D 2019GPT-2 arrived in 2019, between GPT-1 (2018) and GPT-3 (2020).
  16. 16. C 2023LLaMA was released by Meta in 2023, the same year as GPT-4. ChatGPT had launched in late 2022.
  17. 17. A BERTBERT (2018) came first, then GPT-2 (2019), GPT-3 (2020), and LLaMA (2023).
  18. 18. C 2018BERT was released in 2018, ahead of GPT-2 (2019) and GPT-3 (2020) in the field's timeline.
  19. 19. B 2018BERT came out in 2018, ahead of GPT-2 (2019) and GPT-3 (2020).
  20. 20. D 2019GPT-2 was released in 2019, jumping to 1.5 billion parameters.
  21. 21. D 2019GPT-2 launched in 2019, between GPT-1 (2018) and GPT-3 (2020).
  22. 22. C 2023GPT-4 was released on March 14, 2023, following ChatGPT's late-2022 launch.
  23. 23. B November 30, 2022ChatGPT launched November 30, 2022. March 14, 2023 is GPT-4's release date, a few months later.
  24. 24. A About 2 monthsUBS analysts, using Similarweb data, estimated ChatGPT reached roughly 100 million monthly users within two months, which they called the fastest ramp of any consumer internet app they had followed.
  25. 25. B 2023GPT-4 launched March 14, 2023. 2022 is the year ChatGPT debuted, on November 30.
  26. 26. A About three and a half monthsChatGPT launched November 30, 2022 and GPT-4 followed March 14, 2023 — about three and a half months. Two months is how long ChatGPT took to reach 100 million users.
  27. 27. C It took about a year to reach 100 million monthly usersThe climb to 100 million monthly users took roughly two months, not a year — which is exactly why it set the growth record.
  28. 28. D InstructGPT's 1.3-billion-parameter model was preferred by humans over the 175-billion GPT-3Chinchilla (more training data for a smaller model) and InstructGPT (alignment) showed that something other than raw size can win — a 1.3B InstructGPT model was preferred over 175B GPT-3. The other statements are true but don't undercut model size.
  29. 29. B Loss falls predictably as a power function of compute, data, and parametersKaplan's laws gave a quantitative power-law map of loss versus compute, data, and parameters, justifying ever-larger models. The 20-tokens-per-parameter rule is Chinchilla (2022); the other two contradict the smooth power-law finding.
  30. 30. C GPT-3 was significantly undertrained for its parameter countChinchilla showed GPT-3 was undertrained given its 175B parameters: the 70B-parameter Chinchilla, trained on about four times as much data as the 280B Gopher at the same compute budget, outperformed both Gopher and GPT-3. Chinchilla revised the power laws rather than rejecting them.
  31. 31. A About 2300 billion tokens spread across 175 billion parameters is only about 1.7 tokens per parameter. Chinchilla (2022) judged that far too little data for a model that size, which is why it called GPT-3 significantly undertrained.
  32. 32. D Chinchilla concluded GPT-3 had too few parameters for its training dataChinchilla found the opposite: GPT-3 was undertrained, with too little data for its parameter count, not too few parameters. The other three statements are accurate; the 70B Chinchilla outperformed GPT-3 despite being far smaller.
  33. 33. D 2020The Kaplan scaling-laws paper, showing loss falls predictably with scale, was published in 2020.
  34. 34. A 2020Kaplan's scaling-laws paper was published in 2020, two years before Chinchilla refined the picture.
  35. 35. B 20Chinchilla's rule is about 20 training tokens per parameter for compute-optimal training.
  36. 36. C 20 to 1Chinchilla's compute-optimal ratio is about 20 tokens per parameter, well above what earlier giant models used.

Sources

Download all 10 lessons: PDF · EPUB