LLM 101 study guide · Lesson 10 of 10
Key Concepts
Key People · Key Things
This is the lesson's study guide from the FlashFeed app, plus its key facts and a self-test.
What you'll be able to do
- Recognize the three Turing-Award 'godfathers of deep learning' — Bengio, Hinton, and LeCun.
- Identify key researchers behind modern LLMs — Sutskever and Radford at OpenAI, and educator Andrew Ng.
- Recognize foundational techniques and architectures — word2vec, LSTM, and the Transformer.
- Recognize landmark models — BERT, GPT-3, and ChatGPT — including GPT-3's 175-billion-parameter scale.
A quick-recognition gallery of the key people, places, and things from LLM 101.
Key People
Yoshua Bengio — A Canadian computer scientist known for his work in deep learning and artificial intelligence. A pioneer of deep learning; his group introduced influential neural language models and attention mechanisms. He won the Turing Award in 2018 alongside Geoffrey Hinton and Yann LeCun for their work on deep learning.
Geoffrey Hinton — A British-Canadian cognitive psychologist and computer scientist recognized as a leading figure in AI. He is known for his research on neural networks and deep learning techniques that underpin modern LLMs. He received the Turing Award in 2018, sharing it with Yoshua Bengio and Yann LeCun for their contributions to AI.
Yann LeCun — A French-American computer scientist known for his work in machine learning and computer vision. He is one of the founding figures of convolutional neural networks, crucial for image processing. He was awarded the Turing Award in 2018 alongside Geoffrey Hinton and Yoshua Bengio for advancements in deep learning.
Ilya Sutskever — A co-founder and former chief scientist of OpenAI who later left to found Safe Superintelligence. He played a significant role in developing the sequence-to-sequence learning model. He is known for his contributions to generative models, including the GPT series of language models.
Andrew Ng — An influential computer scientist and entrepreneur in the field of AI and machine learning. He co-founded Google Brain and has significantly impacted online education through Coursera. He is known for his work on deep learning and has taught many foundational courses in AI.
Alec Radford — An AI researcher known for his work at OpenAI, particularly in natural language processing. He was a key figure in developing the original GPT model and its successors. He has contributed to advancements in generative models and their applications in AI.
Key Things
word2vec — A technique for representing words as vectors in a high-dimensional space. It captures semantic relationships through vector arithmetic. Introduced by a team led by Tomas Mikolov in 2013.
BERT — A model designed for understanding the context of words in a sentence. It uses bidirectional training to predict masked words. Released by Google in October 2018, it set new benchmarks in NLP.
GPT-3 — A large language model known for its few-shot learning capabilities. It has 175 billion parameters, making it one of the largest language models of its time. Released by OpenAI in 2020, it can pick up new tasks from a few examples in the prompt, without task-specific retraining.
Transformer — An architecture that processes sequences of data in parallel. It replaces recurrent layers with self-attention mechanisms. Introduced in the 2017 paper 'Attention Is All You Need' by Vaswani et al.
LSTM — A type of recurrent neural network designed to remember information over long sequences. It addresses the vanishing gradient problem of standard RNNs. Developed in the 1990s, it became popular for NLP tasks around 2014.
ChatGPT — An AI model designed for conversational interactions. It utilizes reinforcement learning from human feedback to improve responses. Launched by OpenAI in November 2022, it quickly gained widespread use.
Key concepts
Yoshua Bengio
- A Canadian computer scientist known for his work in deep learning and artificial intelligence.
- A pioneer of deep learning; his group introduced influential neural language models and attention mechanisms.
- He won the Turing Award in 2018 alongside Geoffrey Hinton and Yann LeCun for their work on deep learning.
Geoffrey Hinton
- A British-Canadian cognitive psychologist and computer scientist recognized as a leading figure in AI.
- He is known for his research on neural networks and deep learning techniques that underpin modern LLMs.
- He received the Turing Award in 2018, sharing it with Yoshua Bengio and Yann LeCun for their contributions to AI.
Yann LeCun
- A French-American computer scientist known for his work in machine learning and computer vision.
- He is one of the founding figures of convolutional neural networks, crucial for image processing.
- He was awarded the Turing Award in 2018 alongside Geoffrey Hinton and Yoshua Bengio for advancements in deep learning.
Ilya Sutskever
- A co-founder and former chief scientist of OpenAI who later left to found Safe Superintelligence.
- He played a significant role in developing the sequence-to-sequence learning model.
- He is known for his contributions to generative models, including the GPT series of language models.
Andrew Ng
- An influential computer scientist and entrepreneur in the field of AI and machine learning.
- He co-founded Google Brain and has significantly impacted online education through Coursera.
- He is known for his work on deep learning and has taught many foundational courses in AI.
Alec Radford
- An AI researcher known for his work at OpenAI, particularly in natural language processing.
- He was a key figure in developing the original GPT model and its successors.
- He has contributed to advancements in generative models and their applications in AI.
word2vec
- A technique for representing words as vectors in a high-dimensional space.
- It captures semantic relationships through vector arithmetic.
- Introduced by a team led by Tomas Mikolov in 2013.
BERT
- A model designed for understanding the context of words in a sentence.
- It uses bidirectional training to predict masked words.
- Released by Google in October 2018, it set new benchmarks in NLP.
GPT-3
- A large language model known for its few-shot learning capabilities.
- It has 175 billion parameters, making it one of the largest language models of its time.
- Released by OpenAI in 2020, it can pick up new tasks from a few examples in the prompt, without task-specific retraining.
Transformer
- An architecture that processes sequences of data in parallel.
- It replaces recurrent layers with self-attention mechanisms.
- Introduced in the 2017 paper 'Attention Is All You Need' by Vaswani et al.
LSTM
- A type of recurrent neural network designed to remember information over long sequences.
- It addresses the vanishing gradient problem of standard RNNs.
- Developed in the 1990s, it became popular for NLP tasks around 2014.
ChatGPT
- An AI model designed for conversational interactions.
- It utilizes reinforcement learning from human feedback to improve responses.
- Launched by OpenAI in November 2022, it quickly gained widespread use.
Self-test
Answer each question from memory, then check your work against the answer key.
1. Yoshua Bengio and Geoffrey Hinton shared the 2018 Turing Award for deep learning with which third researcher?
- AYann LeCun
- BAlec Radford
- CIlya Sutskever
- DAndrew Ng
2. Which of the three 2018 Turing Award winners is a British-Canadian cognitive psychologist and computer scientist?
- AYann LeCun
- BGeoffrey Hinton
- CIlya Sutskever
- DYoshua Bengio
3. Which 2018 Turing Award recipient is a founding figure of convolutional neural networks, crucial for image processing?
- AYoshua Bengio
- BGeoffrey Hinton
- CAndrew Ng
- DYann LeCun
4. Which Turing Award 'godfather' led the group that introduced influential neural language models and attention mechanisms?
- AGeoffrey Hinton
- BAlec Radford
- CYoshua Bengio
- DYann LeCun
5. Which researcher is NOT one of the three who shared the 2018 Turing Award for deep learning?
- AGeoffrey Hinton
- BYoshua Bengio
- CAndrew Ng
- DYann LeCun
6. Who co-founded OpenAI and served as its chief scientist before leaving to found Safe Superintelligence?
- AIlya Sutskever
- BAlec Radford
- CAndrew Ng
- DYann LeCun
7. Alec Radford is best known for which contribution?
- ACo-founding Google Brain
- BThe word2vec embedding technique
- CFounding Safe Superintelligence
- DThe original GPT model and its successors
8. Andrew Ng co-founded which AI research organization?
- ADeepMind
- BGoogle Brain
- CSafe Superintelligence
- DOpenAI
9. Andrew Ng significantly impacted online education through which platform?
- ACoursera
- BKhan Academy
- CUdacity
- DedX
10. Ilya Sutskever played a significant role in developing which of the following?
- AThe LSTM recurrent architecture
- BThe word2vec embedding technique
- CThe Transformer architecture
- DThe sequence-to-sequence learning model
11. Which OpenAI co-founder and former chief scientist left to found Safe Superintelligence?
- AGeoffrey Hinton
- BAlec Radford
- CIlya Sutskever
- DAndrew Ng
12. Who helped develop the sequence-to-sequence model and contributed to the GPT series?
- AYoshua Bengio
- BIlya Sutskever
- CAlec Radford
- DYann LeCun
13. Which AI researcher was a key figure in developing the original GPT model at OpenAI?
- AGeoffrey Hinton
- BAlec Radford
- CIlya Sutskever
- DAndrew Ng
14. Whose OpenAI research centered on NLP and generative models?
- AAndrew Ng
- BYoshua Bengio
- CAlec Radford
- DYann LeCun
15. Which computer scientist co-founded Google Brain and had a major impact on AI education via Coursera?
- AYann LeCun
- BAlec Radford
- CIlya Sutskever
- DAndrew Ng
16. Which technique represents words as vectors in a high-dimensional space, capturing semantic relationships through vector arithmetic?
- Aword2vec
- BBERT
- CTransformer
- DLSTM
17. word2vec was introduced by a team led by which researcher, and in what year?
- ATomas Mikolov, in 2013
- BVaswani et al., in 2017
- CGeoffrey Hinton, in 1990
- DAlec Radford, in 2018
18. The Transformer architecture replaces recurrent layers with what?
- AWord-vector arithmetic
- BLSTM memory gates
- CSelf-attention mechanisms
- DConvolutional layers
19. LSTM was designed to address which weakness of standard RNNs?
- AThe cost of self-attention
- BOverfitting on small datasets
- CThe curse of dimensionality
- DThe vanishing gradient problem
20. LSTM was developed in which decade, long before it became popular for NLP tasks around 2014?
- AThe 1970s
- BThe 1990s
- CThe 2010s
- DThe 2000s
21. Which technique represents words as vectors, capturing semantic relationships through vector arithmetic?
- ABERT
- BLSTM
- Cword2vec
- DTransformer
22. Word2vec was introduced in 2013 by a team led by whom?
- ATomas Mikolov
- BAndrew Ng
- CAlec Radford
- DAshish Vaswani
23. Which architecture is a recurrent neural network designed to remember information over long sequences?
- ATransformer
- Bword2vec
- CBERT
- DLSTM
24. LSTM was developed to address which problem in standard RNNs?
- ALack of parallel processing
- BThe vanishing gradient problem
- COverfitting on small datasets
- DInsufficient vector dimensionality
25. Which architecture replaces recurrent layers with self-attention to process sequences in parallel?
- ABERT
- BLSTM
- CTransformer
- Dword2vec
26. How many parameters does GPT-3 have?
- A175 million
- B175 billion
- C13 billion
- D1.75 billion
27. How does BERT learn to understand the context of words in a sentence?
- ABidirectional training to predict masked words
- BReinforcement learning from human feedback
- CLeft-to-right next-word prediction
- DVector arithmetic over word embeddings
28. Which capability is GPT-3 best known for?
- ABidirectional prediction of masked words
- BCapturing semantics through vector arithmetic
- CLearning from human feedback on its responses
- DFew-shot learning — performing tasks from a few prompt examples, without retraining
29. Which technique does ChatGPT use to improve its responses?
- AConvolutional filters over text
- BReinforcement learning from human feedback
- CBidirectional masked-word pretraining
- DWord-vector arithmetic
30. Which of these was released most recently?
- AGPT-3
- Bword2vec
- CBERT
- DChatGPT
31. Which model uses bidirectional training to predict masked words in a sentence?
- AGPT-3
- BLSTM
- CBERT
- DChatGPT
32. Which company released BERT in October 2018?
- AGoogle
- BOpenAI
- CMicrosoft
- DMeta
33. Which AI model uses reinforcement learning from human feedback to improve conversational responses?
- Aword2vec
- BChatGPT
- CBERT
- DGPT-3
34. In what month and year did OpenAI launch ChatGPT?
- AJanuary 2023
- BOctober 2018
- CNovember 2022
- DJuly 2020
Go deeper
Answer key
- 1. A Yann LeCunThe 2018 Turing Award went jointly to Bengio, Hinton, and Yann LeCun for their foundational deep-learning work. LeCun is also known as a founding figure of convolutional neural networks.
- 2. B Geoffrey HintonHinton is British-Canadian and trained as a cognitive psychologist; his neural-network research underpins modern LLMs. Bengio is Canadian and LeCun is French-American, and Sutskever was not a Turing Award recipient.
- 3. D Yann LeCunLeCun, a French-American computer scientist, is one of the founding figures of convolutional neural networks. Hinton and Bengio shared the same award but for other deep-learning contributions; Ng did not receive it.
- 4. C Yoshua BengioBengio, a Canadian computer scientist, led the group that introduced influential neural language models and attention mechanisms. LeCun is associated with convolutional networks, and Radford was never a Turing laureate.
- 5. C Andrew NgThe 2018 Turing Award went jointly to Bengio, Hinton, and LeCun. Andrew Ng is influential in AI as a Google Brain co-founder and educator, but he was not among the three recipients.
- 6. A Ilya SutskeverSutskever co-founded OpenAI, was its chief scientist, and later left to found Safe Superintelligence. Radford also worked at OpenAI but is known for the original GPT model rather than co-founding the lab.
- 7. D The original GPT model and its successorsRadford, an AI researcher at OpenAI focused on natural language processing, was a key figure behind the original GPT model and its successors. Google Brain traces to Ng, word2vec to Mikolov's team, and Safe Superintelligence to Sutskever.
- 8. B Google BrainNg co-founded Google Brain. OpenAI was co-founded by Sutskever among others, and Safe Superintelligence is the lab Sutskever founded after leaving OpenAI.
- 9. A CourseraNg co-founded Coursera and taught many foundational AI courses there. The other platforms are real online-education services but are not the one credited to him.
- 10. D The sequence-to-sequence learning modelSutskever helped develop sequence-to-sequence learning, plus generative models like the GPT series. The Transformer came from Vaswani et al., LSTM from Hochreiter and Schmidhuber, and word2vec from Mikolov's team.
- 11. C Ilya SutskeverIlya Sutskever co-founded and served as chief scientist of OpenAI before leaving to found Safe Superintelligence.
- 12. B Ilya SutskeverIlya Sutskever is credited with the sequence-to-sequence model and contributions to the GPT series.
- 13. B Alec RadfordAlec Radford is a key figure behind the original GPT model and its successors at OpenAI.
- 14. C Alec RadfordAlec Radford's work at OpenAI focused on natural language processing and generative models.
- 15. D Andrew NgAndrew Ng is credited with co-founding Google Brain and shaping online AI education through Coursera.
- 16. A word2vecword2vec represents words as high-dimensional vectors whose arithmetic captures semantic relationships. LSTM is a recurrent architecture and the Transformer is a parallel sequence architecture, not embedding techniques.
- 17. A Tomas Mikolov, in 2013word2vec was introduced in 2013 by a team led by Tomas Mikolov. Vaswani et al. in 2017 introduced the Transformer instead, a different milestone.
- 18. C Self-attention mechanismsIntroduced in the 2017 paper 'Attention Is All You Need', the Transformer swaps recurrent layers for self-attention. LSTM gates are exactly the recurrent machinery it replaces, not what replaced it.
- 19. D The vanishing gradient problemLSTMs remember information over long sequences by addressing the vanishing gradient problem of standard RNNs. Self-attention cost belongs to Transformers, an architecture that came much later.
- 20. B The 1990sLSTM was developed in the 1990s and only became popular for NLP tasks around 2014. The 2010s is tempting because of that popularity date, but the architecture itself is far older.
- 21. C word2vecword2vec represents words as vectors in a high-dimensional space, capturing semantic relationships via vector arithmetic.
- 22. A Tomas MikolovWord2vec was introduced by a team led by Tomas Mikolov in 2013.
- 23. D LSTMLSTM is a type of recurrent neural network built to remember information across long sequences.
- 24. B The vanishing gradient problemLSTM addresses the vanishing gradient problem found in standard RNNs.
- 25. C TransformerThe Transformer architecture processes sequences in parallel using self-attention instead of recurrent layers.
- 26. B 175 billionGPT-3, released by OpenAI in 2020, has 175 billion parameters — one of the largest models of its time — and is known for strong few-shot learning, performing tasks from a few prompt examples without task-specific retraining.
- 27. A Bidirectional training to predict masked wordsBERT, released by Google in October 2018, uses bidirectional training to predict masked words, which is how it captures sentence context. Reinforcement learning from human feedback is ChatGPT's technique, not BERT's.
- 28. D Few-shot learning — performing tasks from a few prompt examples, without retrainingGPT-3, released by OpenAI in 2020, is known for few-shot learning: given a few examples in the prompt, it can perform new tasks without task-specific retraining. Masked-word prediction is BERT's approach and human-feedback learning is ChatGPT's.
- 29. B Reinforcement learning from human feedbackChatGPT, launched by OpenAI in November 2022, uses reinforcement learning from human feedback to improve responses. Bidirectional masked-word training is BERT's approach instead.
- 30. D ChatGPTChatGPT launched in November 2022, after GPT-3 (2020), BERT (October 2018), and word2vec (2013). GPT-3 is the closest tempting answer but predates ChatGPT by two years.
- 31. C BERTBERT uses bidirectional training to predict masked words and understand word context.
- 32. A GoogleBERT was released by Google in October 2018 and set new NLP benchmarks.
- 33. B ChatGPTChatGPT uses reinforcement learning from human feedback (RLHF) to improve its conversational responses.
- 34. C November 2022OpenAI launched ChatGPT in November 2022, and it quickly gained widespread use.