LLM 101 study guide · Lesson 1 of 10

A Brief History of LLMs

Before the Transformer · The GPT era · ChatGPT and the modern wave

This is the lesson's study guide from the FlashFeed app, plus its key facts and a self-test.

What you'll be able to do

  • Trace the pre-Transformer progression from RNNs/LSTMs through the attention mechanism to the 2017 Transformer.
  • Contrast the Kaplan (2020) and Chinchilla (2022) scaling laws and what each revised about optimal model training.
  • Explain how BERT's bidirectional (masked) pretraining differed from GPT and identify the GPT-1 to GPT-3 progression.
  • Describe how RLHF/InstructGPT and the ChatGPT and GPT-4 launches turned capable models into useful products.
  • Recognize the modern open-weight wave (LLaMA, Mistral) and the frontier assistants (Claude, Gemini, GPT-4).

Before the Transformer

The history of language AI is a series of progressively bolder approximations of how humans use words. Each generation hit a ceiling that the next paradigm broke through.

Pre-Transformer NLP: n-grams

N-gram models estimated the probability of a word by counting how often specific short sequences appeared in a training corpus. A trigram model, for example, guesses the next word from the prior two. They were computationally cheap and dominated commercial NLP for decades, but the architecture was fundamentally local: no amount of text could teach an n-gram model that a pronoun at the end of a paragraph referred to a noun at the top. Long-range meaning was simply invisible to them.

RNNs and LSTMs

Recurrent Neural Networks introduced an internal hidden state that carried information forward through a sequence, giving models a form of memory. LSTMs (Long Short-Term Memory networks) refined this with gating mechanisms that regulated what to remember and what to forget, partially addressing the notorious vanishing gradient problem that caused earlier RNNs to lose signal over long sequences. LSTM-based models dominated NLP benchmarks roughly from 2014 to 2017, but sequential processing remained a hard constraint: each token had to wait for the previous one, and training could not be parallelized across a sequence.

word2vec (2013)

Mikolov's word2vec trained a shallow neural network to predict words from context, producing dense embedding vectors that encoded semantic relationships geometrically. The classic demo — king − man + woman ≈ queen — showed that analogy relationships lived in the geometry of the space. word2vec did not model sequences; it modeled meaning. That distinction mattered: every major NLP model since uses some form of learned embeddings descended from this idea.

seq2seq and attention (2014–2015)

Sequence-to-sequence (seq2seq) models split translation into an encoder (compress source into a fixed vector) and a decoder (generate target from that vector). The fixed-vector bottleneck was the problem: a single state had to carry everything. Bahdanau's attention mechanism fixed it by letting the decoder query all encoder hidden states at each decoding step, weighting them by relevance. Quality jumped measurably. Attention was the decisive conceptual move; the 2017 Transformer paper ("Attention Is All You Need") proved you could discard the recurrence entirely and build a model from attention alone.

The GPT era

Between 2018 and 2020 the field ran the pretraining experiment at increasing scale and learned that the recipe generalized far beyond anyone's initial expectations.

BERT (2018): bidirectional pretraining

Google's BERT (Bidirectional Encoder Representations from Transformers) flipped the training objective: instead of predicting the next token from the left, it masked random words and predicted them from context on both sides simultaneously. This masked language modeling task forced the model to build genuinely bidirectional representations. Introduced in a paper in October 2018 (code and pretrained models followed in November) and quickly fine-tuned, BERT set new state-of-the-art results on eleven NLP benchmarks at once — a signal that the pretraining paradigm was not a GPT-specific quirk but a general method.

GPT-1 (2018): the first GPT

OpenAI's GPT-1 (117 million parameters, trained on BooksCorpus) was the original demonstration that a generatively pretrained Transformer could be fine-tuned on downstream tasks and match models purpose-built for those tasks. The contribution was less the architecture than the paradigm: pretrain once on text, adapt cheaply. Every successor GPT model, and most subsequent LLMs, inherits this two-phase recipe.

GPT-2 (2019): scaling surprises

GPT-2 scaled to 1.5 billion parameters trained on 40 GB of web text (WebText). Its outputs were coherent enough across paragraphs that OpenAI staged a phased release, initially withholding the full model over concerns about misuse for disinformation. The episode was one of the field's first high-profile public debates about capability risk. GPT-2 also began accumulating the intuition that scale alone, applied to the same objective, kept improving fluency — an intuition the scaling-law papers would soon formalize.

GPT-3 (2020): the few-shot giant

GPT-3 (175 billion parameters, trained on ~300 billion tokens) was more than 100× larger than GPT-2. Its defining discovery was in-context learning: given a handful of examples in the prompt alone — no gradient updates — the model performed competitively on tasks it was never explicitly trained on. The same year, Kaplan et al.'s scaling laws (2020) described how loss falls predictably as a power function of compute, data, and parameters, giving researchers a quantitative map for future runs. Two years later, Chinchilla (2022) revised those laws: GPT-3 was significantly undertrained given its parameter count, and DeepMind's 70B-parameter Chinchilla, trained on about four times as much data as DeepMind's earlier 280B-parameter Gopher at the same compute budget, outperformed both Gopher and GPT-3. A rule of thumb drawn from these results is roughly 20 training tokens per parameter; GPT-3's ratio was under 2.

ChatGPT and the modern wave

The 2022–2023 period compressed what might have been a decade of AI product development into about eighteen months, reshaping how the public and industry thought about language models.

InstructGPT and RLHF (2022)

Raw GPT-3 was capable but not reliably useful: it completed prompts in ways that were technically coherent but often unhelpful or harmful. Reinforcement Learning from Human Feedback (RLHF) fixed the alignment between capability and intent. Human raters ranked model outputs; those rankings trained a reward model; the policy was then fine-tuned against it. The striking finding: a 1.3B-parameter InstructGPT was preferred by raters over the untuned 175B GPT-3. Alignment training, not raw scale, determined practical usefulness.

ChatGPT launches (Nov 30, 2022)

OpenAI released ChatGPT on November 30, 2022, as a free public research preview built on a GPT-3.5-class RLHF model. It reached 1 million users in five days and an estimated 100 million monthly active users within two months (UBS analysts, citing Similarweb data) (an analyst estimate), making it the fastest consumer-product adoption curve recorded to that point. The public interface — a simple chat box — made capability legible to non-technical users for the first time and triggered a competitive response across the industry within weeks.

GPT-4 (March 2023)

GPT-4 launched on March 14, 2023, with two advances: multimodality (accepting image and text input) and substantially higher reasoning quality, demonstrated by above-human-average scores on bar exams, medical licensing tests, and other professional benchmarks. The initial context window was 8,000 tokens, with a 32K variant following. OpenAI provided no public parameter count, beginning an era of less transparent frontier model releases.

The open-weight wave: LLaMA (2023)

Meta released LLaMA in February 2023 — a family of models from 7B to 65B parameters trained exclusively on publicly available data (Common Crawl, GitHub, Wikipedia, books). The headline result: LLaMA-13B outperformed GPT-3 (175B) on most standard benchmarks, demonstrating that careful data curation and training compute could compensate for parameter count. Meta shared the weights with approved researchers under a non-commercial license; after a copy leaked online in March 2023, the community quickly produced fine-tuned variants such as Alpaca and Vicuna, igniting an open-source ecosystem.

Mistral AI (2023)

Former DeepMind and Meta researchers founded Mistral AI in Paris in 2023. Mistral 7B (September 2023), released under the permissive Apache 2.0 license, outperformed the larger Llama 2 13B on the benchmarks Mistral reported. Mixtral 8×7B matched GPT-3.5 performance using a Mixture-of-Experts (MoE) architecture — only a subset of the model's parameters activate per token, so inference cost stays well below the nominal parameter count.

Claude and Gemini

Anthropic released Claude in 2023, built with Constitutional AI and RLAIF (reinforcement learning from AI feedback) — techniques designed to reduce reliance on human labelers while encoding explicit values. Google released Gemini in December 2023, developed by Google DeepMind, as its multimodal flagship. It powered Bard, which Google renamed Gemini in February 2024. Together Claude and Gemini established that the frontier would be contested by multiple organizations rather than a single lab.

Key concepts

GPT-1

  • OpenAI's 2018 debut in what became a famous numbered family of models.
  • With just 117 million parameters, trained on BooksCorpus, it proved one pretrained model could be cheaply adapted to match purpose-built systems.
  • The original and smallest of its line, it established the 'pretrain once, adapt cheaply' recipe every successor inherited.

GPT-2

  • A 2019 OpenAI model with 1.5 billion parameters, trained on 40 GB of web text called WebText.
  • Its paragraphs were coherent enough that OpenAI staged a phased release rather than publishing it all at once.
  • It was initially withheld over fears it could mass-produce disinformation — one of the field's first big public capability-risk debates.

GPT-4

  • Launched March 14, 2023, this OpenAI model added multimodality, accepting image as well as text input.
  • It scored above the human average on bar exams and medical licensing tests.
  • OpenAI published no parameter count for it, and its context window began at 8,000 tokens.

LLaMA

  • A family of models Meta released in February 2023, ranging from 7B to 65B parameters.
  • It was trained exclusively on publicly available data like Common Crawl, GitHub, and Wikipedia.
  • Its 13B version outperformed a 175B model, and a leaked copy ignited an open-source fine-tuning ecosystem.

Mistral 7B

  • A September 2023 open model released under the permissive Apache 2.0 license.
  • It outperformed the larger Llama 2 13B on the benchmarks Mistral reported.
  • The first release from a Paris startup founded by former DeepMind and Meta researchers.

Key facts

RNNs and LSTMs

Recurrent Neural Networks (RNNs) and their improved variant, Long Short-Term Memory (LSTM) networks, processed text sequentially, keeping a hidden 'memory' of earlier words. They dominated NLP from roughly 2014–2017 but struggled with long sequences due to vanishing gradients.

Four points, king, queen, man and woman, joined by equal green arrows marked Same step: man to woman and king to queen. King minus man plus woman is about queen.
word2vec (2013)FlashFeed original diagram · CC-BY-4.0

word2vec (2013)

Mikolov and colleagues introduced word2vec, which learned dense vector representations of words from large text. It famously showed that vector arithmetic captured meaning (king − man + woman ≈ queen), making it a foundational embedding technique for all later NLP.

Translating the black cat into le chat noir: at the word noir, lines look back at every source word, thickest to black. A thicker line means more attention.
seq2seq and attention (2014–2015)FlashFeed original diagram · CC-BY-4.0

seq2seq and attention (2014–2015)

Sutskever's sequence-to-sequence model used an encoder–decoder LSTM to translate sentences. Bahdanau then added an attention mechanism that let the decoder 'look back' at all encoder states, sharply improving translation — and previewing the self-attention at the heart of the Transformer.

Attention replaces recurrence
Attention replaces recurrenceAI-generated (gpt-image-1.5)

Attention replaces recurrence

The 2017 Transformer paper by Vaswani et al. proved that recurrence could be discarded entirely — a model built from attention alone was enough.

BERT (2018): bidirectional pretraining

Devlin and colleagues at Google introduced BERT in a paper in October 2018. Unlike left-to-right models, BERT was pretrained to predict masked words using context from both sides of a sentence, setting new state-of-the-art results on eleven NLP benchmarks.

GPT-1 (2018): the first GPT

OpenAI released GPT-1 in 2018 with 117 million parameters, trained on BooksCorpus. It showed that a generative pretrained Transformer, fine-tuned on downstream tasks, could match task-specific models — validating the pretraining paradigm that everything since has built on.

GPT-2 (2019): scaling surprises
GPT-2 (2019): scaling surprisesAI-generated (gpt-image-1.5)

GPT-2 (2019): scaling surprises

GPT-2 scaled to 1.5 billion parameters, trained on 40 GB of web text. Its fluent, coherent generation was striking enough that OpenAI initially withheld the full model over misuse concerns — an early signal of the field's growing awareness of capability risks.

Kaplan's scaling laws
Kaplan's scaling lawsAI-generated (gpt-image-1.5)

Kaplan's scaling laws

Kaplan et al.'s 2020 scaling laws showed that loss falls predictably as a power function of compute, data, and parameters — a quantitative map that justified ever-larger models.

InstructGPT and RLHF (2022)

OpenAI showed that fine-tuning GPT-3 with Reinforcement Learning from Human Feedback (RLHF) sharply improved helpfulness and reduced harmful output. Remarkably, a 1.3B InstructGPT was preferred by human raters over the raw 175B GPT-3 — bigger is not automatically better.

GPT-4 (March 2023)

OpenAI released GPT-4 on March 14, 2023. It was multimodal (accepting image and text input), scored at or above human level on many professional exams, and launched with an 8,000-token context window, with a 32K variant following soon after.

Mistral AI (2023)

Former DeepMind and Meta researchers founded Mistral AI in Paris in 2023. Its Mistral 7B (September 2023), released under Apache 2.0, outperformed the larger Llama 2 13B on the benchmarks Mistral reported, and Mixtral 8×7B matched GPT-3.5 using a Mixture-of-Experts design.

Claude and Gemini

Anthropic released Claude (2023), built with Constitutional AI and RLAIF alignment, while Google released Gemini (December 2023) as its multimodal flagship, developed by Google DeepMind (Bard was renamed Gemini in February 2024). Both are major non-OpenAI frontier model families.

By the numbers

Year the Transformer paper 'Attention Is All You Need' was published
2017 year

Matched pairs

Each item on the left goes with the one on the right.

“Attention Is All You Need” (2017)
Vaswani et al. (Google) — introduced the Transformer, replacing recurrent layers entirely with self-attention; the architectural foundation of most modern LLMs
Kaplan scaling laws (2020)
Showed that LLM loss falls predictably as a power-law with model size, dataset size, and compute — justifying the strategy of building ever-larger models
Chinchilla scaling laws (2022)
DeepMind showed prior large models were undertrained; compute-optimal training needs about 20 tokens of data per parameter — shifting the field toward smaller models trained on far more data

Self-test

Answer each question from memory, then check your work against the answer key.

  1. 1. In sequence-to-sequence (seq2seq) translation models, what problem did Bahdanau's attention mechanism solve?

    1. AThe single fixed-length vector between encoder and decoder had to carry all the source's information and lost detail on long inputs
    2. BThe inability to train the model in parallel across a sequence on GPUs
    3. CThe vanishing gradient that caused RNNs to lose signal over long sequences
    4. DN-gram models' blindness to any word more than two positions back
  1. 2. LSTMs added gating mechanisms that regulated what to remember and what to forget. Which problem did this partially address?

    1. AThe sparsity of n-gram counts for rare word sequences
    2. BThe fixed-length vector bottleneck between a seq2seq encoder and decoder
    3. CThe inability to parallelize training across the tokens of a sequence
    4. DThe vanishing gradient that caused RNNs to lose signal over long sequences
  2. 3. Which statement about LSTM-based NLP models (roughly 2014–2017) is NOT accurate?

    1. AGating mechanisms regulated what the model remembered and forgot
    2. BAn internal hidden state carried information forward through the sequence
    3. CTraining could be parallelized across every token in a sequence at once
    4. DThey dominated NLP benchmarks for several years before the Transformer
  3. 4. What did the 2017 Transformer paper from Vaswani et al. prove about model architecture?

    1. AAttention layers worked best when bolted onto an LSTM encoder–decoder
    2. BRecurrence could be discarded entirely and a model built from attention alone
    3. CPredicting masked words beat predicting the next word left-to-right
    4. DLearned embeddings could replace sequence modeling altogether
  4. 5. Which sequence puts these pre-Transformer milestones in correct chronological order?

    1. Aword2vec, then seq2seq with attention, then the Transformer paper
    2. BThe Transformer paper, then word2vec, then seq2seq with attention
    3. Cword2vec, then the Transformer paper, then seq2seq with attention
    4. Dseq2seq with attention, then word2vec, then the Transformer paper
  5. 6. What did Kaplan et al.'s 2020 scaling laws describe?

    1. ALoss stops falling with added compute beyond about 10 billion parameters
    2. BLoss falls predictably as a power function of compute, data, and parameters
    3. CCompute-optimal training needs roughly 20 tokens of data per parameter
    4. DLoss improves only in discrete jumps once a model crosses size thresholds
  6. 7. What did the Chinchilla (2022) result revise about GPT-3?

    1. AGPT-3's loss did not follow a power law at all
    2. BGPT-3 had far too many training tokens for its size
    3. CGPT-3 was significantly undertrained for its parameter count
    4. DGPT-3's architecture, not its data, capped its performance
  7. 8. GPT-3 had 175 billion parameters and was trained on about 300 billion tokens. Roughly how many training tokens per parameter is that?

    1. AAbout 100
    2. BAbout 300
    3. CAbout 20
    4. DAbout 2
  8. 9. Which statement about the Kaplan (2020) and Chinchilla (2022) scaling laws is NOT accurate?

    1. AKaplan's laws described how loss falls predictably as models scale up
    2. BChinchilla concluded GPT-3 had too few parameters for its training data
    3. CChinchilla's 70B-parameter model outperformed the much larger 175B-parameter GPT-3
    4. DChinchilla revised Kaplan's laws two years after they appeared
  9. 10. A lab has a fixed compute budget for one training run. What does the Chinchilla finding recommend?

    1. ATrain a smaller model on substantially more data
    2. BSplit the budget across many tiny models and ensemble them
    3. CAdd parameters and cut the token count to fit the budget
    4. DTrain the largest model the budget allows on the same data
  10. 11. How did BERT's (2018) training objective differ from GPT's?

    1. AIt predicted the next token strictly left-to-right, one at a time
    2. BIt learned new tasks from a few examples in the prompt, with no weight updates
    3. CIt ranked human-preferred outputs to train a reward model
    4. DIt masked random words and predicted them using context from both directions (masked language modeling)
  11. 12. Which statement about BERT (2018) is NOT accurate?

    1. AIt predicted randomly masked words using context from both sides at once
    2. BIt set new state-of-the-art results on eleven NLP benchmarks
    3. CIt was pretrained to predict each next token using only left-side context
    4. DGoogle introduced it in a paper in October 2018
  12. 13. What were GPT-1's parameter count and training corpus?

    1. A1.5 billion parameters, trained on WebText
    2. B117 million parameters, trained on BooksCorpus
    3. C340 million parameters, trained on Wikipedia
    4. D175 billion parameters, trained on ~300 billion tokens
  13. 14. What was GPT-3's defining discovery?

    1. AMultimodality — the model accepted both image and text input
    2. BMasked language modeling produced bidirectional representations
    3. CHuman preference rankings could be distilled into a reward model
    4. DIn-context learning — a few prompt examples sufficed, with no gradient updates
  14. 15. Which option lists GPT-1, GPT-2, and GPT-3 with their correct parameter counts in order?

    1. A117 million, then 1.5 billion, then 17.5 billion
    2. B1.5 billion, then 117 million, then 175 billion
    3. C117 million, then 1.5 billion, then 175 billion
    4. D117 million, then 175 billion, then 1.5 billion
  15. 16. Which sequence correctly describes the RLHF pipeline used for InstructGPT?

    1. AHuman raters rank outputs, the rankings train a reward model, the policy is fine-tuned against it
    2. BThe policy generates outputs, a reward model scores them with no human input, and raters only review the final model
    3. CRaters write ideal answers, the model memorizes them, no reward model is used
    4. DA reward model is trained first, then the policy is fine-tuned against it, and only afterward do raters rank outputs
  16. 17. What was the striking finding when InstructGPT was compared against raw GPT-3?

    1. ARaters preferred GPT-3 for helpfulness but InstructGPT for safety
    2. BA 13B InstructGPT tied with the 175B GPT-3 on rater preference
    3. CInstructGPT won only after being scaled past 175B parameters
    4. DA 1.3B InstructGPT was preferred by raters over the 175B GPT-3
  17. 18. On what date did OpenAI release ChatGPT as a free public research preview?

    1. AMarch 14, 2023
    2. BNovember 30, 2022
    3. CDecember 6, 2023
    4. DFebruary 24, 2023
  18. 19. How fast did ChatGPT reach its early adoption milestones?

    1. A1 million users in five days and about 100 million monthly users within two months
    2. B100 million users in five days and 1 billion monthly users within two months
    3. C1 million users in two months and about 100 million monthly users within a year
    4. D1 million users in five weeks and about 10 million monthly users within two months
  19. 20. Which statement about the GPT-4 launch is NOT accurate?

    1. AGPT-4 was multimodal, accepting image and text input
    2. BOpenAI reported GPT-4 scoring above the human average on bar exams and medical licensing tests
    3. COpenAI published GPT-4's parameter count alongside the release
    4. DGPT-4 launched with an 8,000-token context window, with a 32K variant following
  20. 21. What did Meta's LLaMA (2023) demonstrate about model size versus performance?

    1. ALLaMA needed roughly the same 175B parameters as GPT-3 to reach parity
    2. BLLaMA's 65B model was the smallest that could beat GPT-3
    3. CLLaMA matched GPT-3 only after fine-tuning on private medical and legal data
    4. DA 13B-parameter LLaMA outperformed the 175B-parameter GPT-3 on most standard benchmarks
  21. 22. What was Gemini, which Google released in December 2023?

    1. AA multimodal flagship model family developed by Google DeepMind
    2. BA chat preview aligned with Constitutional AI and RLAIF
    3. CA Mixture-of-Experts model that matched GPT-3.5 at low inference cost
    4. DAn open-weight family from 7B to 65B trained only on public data
  22. 23. Under which license did Mistral AI release Mistral 7B in September 2023?

    1. AA research-only community license
    2. BApache 2.0
    3. CGPL-3.0
    4. DCreative Commons BY-NC 4.0
  23. 24. How does the Mixture-of-Experts design in Mixtral 8x7B keep inference cost below its nominal parameter count?

    1. AParameters are compressed to 4 bits before every forward pass
    2. BThe model discards its earliest layers once a prompt exceeds 8K tokens
    3. COnly a subset of the model's parameters activate for each token
    4. DEight separate 7B models vote and the majority answer is returned
  24. 25. Anthropic built Claude with Constitutional AI plus which technique, designed to reduce reliance on human labelers?

    1. ARLAIF — reinforcement learning from AI feedback
    2. BMixture-of-Experts routing
    3. CRLHF — reinforcement learning from human feedback
    4. DMasked language modeling

Go deeper

Answer key

  1. 1. A The single fixed-length vector between encoder and decoder had to carry all the source's information and lost detail on long inputsSeq2seq squeezed the whole source sentence into one fixed vector — a bottleneck. Attention let the decoder look back at all encoder states each step, weighting them by relevance. LSTMs addressed vanishing gradients; the Transformer later added parallelism.
  2. 2. D The vanishing gradient that caused RNNs to lose signal over long sequencesLSTM gating partially fixed vanishing gradients, letting signal survive longer sequences. The fixed-vector bottleneck was seq2seq's problem (solved by attention), and parallel training only arrived with the Transformer.
  3. 3. C Training could be parallelized across every token in a sequence at onceSequential processing was a hard constraint: each token waited for the previous one, so training could not be parallelized across a sequence. The other three describe LSTMs accurately.
  4. 4. B Recurrence could be discarded entirely and a model built from attention aloneThe Transformer paper showed recurrence was unnecessary — attention alone sufficed. Attention-on-LSTM was the earlier Bahdanau step; embeddings were word2vec; masked prediction was BERT in 2018.
  5. 5. A word2vec, then seq2seq with attention, then the Transformer paperword2vec landed in 2013, seq2seq and Bahdanau's attention in 2014–2015, and the Transformer paper in 2017. Attention was the decisive conceptual move that made the Transformer possible.
  6. 6. B Loss falls predictably as a power function of compute, data, and parametersKaplan's laws gave a quantitative power-law map of loss versus compute, data, and parameters, justifying ever-larger models. The 20-tokens-per-parameter rule is Chinchilla (2022); the other two contradict the smooth power-law finding.
  7. 7. C GPT-3 was significantly undertrained for its parameter countChinchilla showed GPT-3 was undertrained given its 175B parameters: the 70B-parameter Chinchilla, trained on about four times as much data as the 280B Gopher at the same compute budget, outperformed both Gopher and GPT-3. Chinchilla revised the power laws rather than rejecting them.
  8. 8. D About 2300 billion tokens spread across 175 billion parameters is only about 1.7 tokens per parameter. Chinchilla (2022) judged that far too little data for a model that size, which is why it called GPT-3 significantly undertrained.
  9. 9. B Chinchilla concluded GPT-3 had too few parameters for its training dataChinchilla found the opposite: GPT-3 was undertrained, with too little data for its parameter count, not too few parameters. The other three statements are accurate; the 70B Chinchilla outperformed GPT-3 despite being far smaller.
  10. 10. A Train a smaller model on substantially more dataChinchilla's compute-optimal point trades parameters for tokens: a smaller, more thoroughly trained model wins at equal compute. Maximizing parameter count was the Kaplan-era instinct that Chinchilla corrected.
  11. 11. D It masked random words and predicted them using context from both directions (masked language modeling)BERT masks random tokens and predicts them from context on both sides at once — bidirectional — unlike GPT's left-to-right next-token objective. It hit state-of-the-art on 11 benchmarks. (In-context learning is GPT-3; reward models are RLHF.)
  12. 12. C It was pretrained to predict each next token using only left-side contextLeft-to-right next-token prediction was GPT's objective; BERT's whole departure was masked language modeling, which forced bidirectional representations. The release date and the eleven-benchmark result are accurate.
  13. 13. B 117 million parameters, trained on BooksCorpusGPT-1 had 117M parameters and trained on BooksCorpus. 1.5B on WebText is GPT-2; 175B on ~300B tokens is GPT-3. GPT-1's contribution was the paradigm — pretrain once, adapt cheaply — more than its scale.
  14. 14. D In-context learning — a few prompt examples sufficed, with no gradient updatesGPT-3 showed in-context (few-shot) learning: examples in the prompt alone let it handle tasks it was never trained on. Multimodality arrived with GPT-4, masked LM is BERT, and reward models belong to RLHF.
  15. 15. C 117 million, then 1.5 billion, then 175 billionThe progression ran 117M (GPT-1, 2018) to 1.5B (GPT-2, 2019) to 175B (GPT-3, 2020) — GPT-3 was more than 100x larger than GPT-2 while keeping the same pretraining objective.
  16. 16. A Human raters rank outputs, the rankings train a reward model, the policy is fine-tuned against itRLHF runs in that order: rank outputs, train a reward model on the rankings, fine-tune the policy against the reward model. The reward model is learned from human rankings, so it cannot come first or score outputs without human input.
  17. 17. D A 1.3B InstructGPT was preferred by raters over the 175B GPT-3Raters preferred the 1.3B InstructGPT over the untuned 175B GPT-3 — alignment training, not raw scale, determined practical usefulness. The 13B figure belongs to LLaMA's comparison with GPT-3, not InstructGPT's.
  18. 18. B November 30, 2022ChatGPT launched November 30, 2022 on a GPT-3.5-class RLHF model. March 14, 2023 is GPT-4's launch; February 2023 is LLaMA; December 2023 is Gemini.
  19. 19. A 1 million users in five days and about 100 million monthly users within two monthsChatGPT hit 1 million users in five days and roughly 100 million monthly active users within two months — the fastest consumer-product adoption curve recorded to that point.
  20. 20. C OpenAI published GPT-4's parameter count alongside the releaseOpenAI gave no public parameter count for GPT-4, beginning an era of less transparent frontier releases. The multimodality, 8K/32K context, and professional-exam results are all accurate.
  21. 21. D A 13B-parameter LLaMA outperformed the 175B-parameter GPT-3 on most standard benchmarksLLaMA-13B beat GPT-3 (175B) on most benchmarks, showing careful data curation and training compute can outweigh raw parameter count. LLaMA trained only on publicly available data, and its weights, shared with researchers and then leaked, sparked an open fine-tuning ecosystem.
  22. 22. A A multimodal flagship model family developed by Google DeepMindGemini was Google DeepMind's multimodal flagship model family (Bard was renamed Gemini in February 2024). The open 7B–65B family is Meta's LLaMA, the MoE model is Mixtral 8x7B, and Constitutional AI plus RLAIF describes Anthropic's Claude.
  23. 23. B Apache 2.0Mistral 7B shipped under the permissive Apache 2.0 license, which allows use without restrictions, including commercial use. A copyleft, non-commercial, or research-only license would have limited exactly that.
  24. 24. C Only a subset of the model's parameters activate for each tokenMoE routes each token to a subset of experts, so only some parameters activate per token and inference stays well below the nominal count. That is routing, not quantization, layer-dropping, or an ensemble vote.
  25. 25. A RLAIF — reinforcement learning from AI feedbackClaude pairs Constitutional AI with RLAIF, which draws feedback from AI rather than human labelers while encoding explicit values. RLHF is the human-labeler method RLAIF was meant to lean on less; the other two are unrelated architectures.

Image credits

  • word2vec (2013): FlashFeed original diagram · CC-BY-4.0
  • seq2seq and attention (2014–2015): FlashFeed original diagram · CC-BY-4.0
  • Attention replaces recurrence: AI-generated (gpt-image-1.5)
  • GPT-2 (2019): scaling surprises: AI-generated (gpt-image-1.5)
  • Kaplan's scaling laws: AI-generated (gpt-image-1.5)

Download all 10 lessons: PDF · EPUB