The Infinity Machine (book) study guide · Lesson 6 of 12

Atari to AlphaGo

Atari and the DQN · AlphaGo and Move 37

Unofficial study guide for The Infinity Machine: Demis Hassabis, DeepMind, and the Quest for Superintelligence by Sebastian Mallaby. FlashFeed is not affiliated with the author or publisher.

This is the lesson's study guide from the FlashFeed app, plus its key facts and a self-test.

What you'll be able to do

  • Define reinforcement learning and recall the DQN milestones — published in Nature (2015), human-level or better on 29 of 49 Atari games.
  • Define AlphaGo as the DeepMind program that beat Lee Sedol in Seoul in March 2016, after a January 2016 Nature paper reported its methods and its Fan Hui win.
  • Identify the people around AlphaGo — engineer Aja Huang, opponents Fan Hui and Lee Sedol, and Sergey Brin who backed the project
  • Explain Move 37 — the shoulder-hit AlphaGo's policy network rated a roughly 1-in-10,000 human move against Lee Sedol
  • Identify OpenAI's founders — CEO Sam Altman and first chief scientist Ilya Sutskever — formed as a counterweight to Google's DeepMind

Atari and the DQN

DeepMind's Atari breakthrough — a single neural network learning to play dozens of games from raw pixels — was the moment the lab's approach moved from promising to historic. The engine behind it, reinforcement learning, would soon be aimed at a far older game.

Forty-nine games, one network

The Deep Q-Network (DQN), published in Nature in February 2015, used a single architecture — same code, same hyperparameters — retrained from scratch on 49 different Atari 2600 games. It reached human-level or better performance on 29 of them without any hand-engineered, game-specific features. The input was raw screen pixels and a score signal; nothing more. Before this, game-playing AI was typically hand-tuned per title. DQN showed that one general algorithm could master many distinct environments, which was the conceptual leap the field had been waiting for. DeepMind had actually demonstrated early Atari results at NeurIPS in December 2013, months before the Google acquisition closed.

Sergey Brin and the Go problem

Not long after the Google acquisition, on a hallway walk in May 2014, Sergey Brin asked Hassabis what hard problem DeepMind would tackle next. Hassabis said: "Go." Brin was sceptical — Go has roughly 10^170 possible board positions, a search space that had defeated every serious AI approach for decades — but he provided the resources. The AlphaGo project started that summer. Go mattered because its complexity made brute-force search impossible; cracking it would require the kind of intuition-like pattern recognition that neural networks are built for.

Fan Hui, 5–0, October 2015

In October 2015, AlphaGo played Fan Hui, the European Go champion, in a closed-door five-game match. AlphaGo won 5–0. The result was kept under embargo until the January 2016 Nature cover paper — the same issue that revealed AlphaGo's architecture. Fan Hui was sufficiently moved by the experience that he joined DeepMind as an analyst to help prepare for the next challenge. AlphaGo's first prototype was built by Aja Huang, whose work on the tree-search component proved critical to making the system competitive at professional level.

Lee Sedol, Seoul, March 2016

In March 2016, AlphaGo travelled to Seoul to face Lee Sedol, widely regarded as the greatest Go player of his generation. Over five games, broadcast live and drawing tens of millions of viewers across Asia, AlphaGo won 4–1. Lee's single victory — game 4 — came from a move so unusual that AlphaGo had not adequately evaluated it. The match made artificial intelligence a front-page story worldwide, compressing years of public awareness into one week.

Move 37

In game two, move 37 — a shoulder-hit on the fifth line — stopped the broadcast commentators cold. Human grandmasters rated it a mistake; AlphaGo's policy network estimated a human would choose that move roughly one time in ten thousand. After twelve minutes away from the board, Lee Sedol returned visibly shaken. AlphaGo's reasoning was correct: the move won territory in a way human intuition would not reach. "Move 37" became shorthand for the idea that trained networks can discover strategic concepts outside the corpus of human play.

AlphaGo and Move 37

AlphaGo's Seoul victory in 2016 didn't just settle a benchmark — it rattled the wider AI world. Within months, a rival lab had materialised in San Francisco, and a talent war was under way that would reshape the industry.

OpenAI is born

In December 2015, Elon Musk and Sam Altman announced OpenAI: a non-profit pledging $1 billion in funding commitments, framed explicitly as a counterweight to Google's ownership of DeepMind. Musk had concluded that concentrating frontier AI capability inside a single commercial giant was dangerous, and DeepMind under Google was his primary concern. OpenAI's founding came about two months after AlphaGo's closed-door victory over Fan Hui, while that result was still confidential — two events close together in time. Sam Altman was co-chair alongside Musk; he became OpenAI's CEO in 2019. The January 2016 Nature cover that announced AlphaGo's Fan Hui result retroactively validated the very fears about Google's AI lead that had already driven OpenAI's December 2015 founding.

The poaching war begins

OpenAI's first hires targeted the DeepMind and Google Brain alumni networks with deliberate precision. Ilya Sutskever — a Geoffrey Hinton student from Toronto who had been working at Google — became chief scientist, the role that shaped OpenAI's early research agenda. Pieter Abbeel and others followed. DeepMind, constrained by Google's standard pay bands, responded with equity-style retention packages unusual for an arms-length subsidiary. The contest for talent between the two labs would intensify over the following decade, driving compensation to levels that reshaped hiring norms across the entire field.

Key terms

Reinforcement learning
Reinforcement learning is a training method in which an agent learns by trial and error, receiving rewards for good actions and penalties for bad ones. DeepMind used reinforcement learning to master Atari games directly from raw pixels.
AlphaGo
AlphaGo is the DeepMind program that defeated world champion Lee Sedol at the board game Go in 2016. AlphaGo combined deep neural networks with Monte Carlo tree search.

Key concepts

Fan Hui

  • A reigning European champion of an ancient Asian board game, chosen for AI's first serious professional test.
  • In a closed-door five-game match in October 2015, he was beaten without winning a single game.
  • Moved by the loss, he joined DeepMind as an analyst; the 5-0 result stayed under embargo until a January 2016 Nature cover.

Lee Sedol

  • Widely regarded as the greatest player of his generation at an ancient board game, he was AI's next challenge after a European champion fell.
  • In a March 2016 Seoul match watched by tens of millions across Asia, he lost four games to one.
  • His lone victory, game four, exploited a move the machine had failed to evaluate; earlier he had left the board shaken by its move 37.

Aja Huang

  • A DeepMind engineer who built the very first prototype of the lab's Go-playing system.
  • His work on the tree-search component proved critical to making that system competitive at professional level.

Sergey Brin

  • A Google co-founder who, soon after the DeepMind acquisition, asked the lab's leader what hard problem it would tackle next.
  • On a May 2014 hallway walk he heard the answer 'Go' and was sceptical of a game with about 10^170 positions.
  • Despite his doubts he provided the resources, and the project he funded launched that summer.

Key facts

Portrait of Google co-founder Sergey Brin
Sergey Brin and the Go problemSteve Jurvetson · CC BY 2.0

Sergey Brin and the Go problem

On a hallway walk at Google in May 2014, Sergey Brin asked Hassabis what hard problem DeepMind would tackle next. Hassabis answered: "Go." Brin was sceptical — the game's complexity had defeated AI for decades — but he provided the resources. The AlphaGo project began that summer.

A Go board covered with black and white stones mid-game
Fan Hui, 5–0, October 2015Goban1 · Public domain

Fan Hui, 5–0, October 2015

In October 2015, AlphaGo played the European Go champion Fan Hui in a closed-door match. The result, 5–0 to AlphaGo, was kept under embargo until the January 2016 Nature cover paper. Fan Hui was so impressed by the experience that he joined DeepMind as an analyst to help prepare for Lee Sedol.

Move 37

In game two, on move 37, AlphaGo played a shoulder-hit at the fifth line that human commentators called a mistake. After 12 minutes of human deliberation, Lee Sedol returned to the board visibly shaken. Move 37 became shorthand for AI creativity — a play no human grandmaster would have chosen, and one that the network calculated had a one-in-ten-thousand chance of being played by a human.

OpenAI is born
OpenAI is bornWikimedia Commons · David Shankbone · CC BY 3.0

OpenAI is born

In December 2015, Elon Musk and Sam Altman announced OpenAI: a non-profit AI lab pledging $1 billion in commitments, explicitly framed as a counterweight to Google DeepMind. Mallaby treats the move as a direct reaction to AlphaGo's progress — Musk in particular had decided that DeepMind "under Google" was the most dangerous concentration of AI talent in the world.

The poaching war begins

OpenAI's first hires deliberately targeted the DeepMind alumni network. Ilya Sutskever, a Toronto-trained Hinton student who had been at Google, became chief scientist. Pieter Abbeel and others followed. DeepMind, under Google's pay scales, fought back by offering equity-style retention packages that were unusual for an arms-length subsidiary.

AlphaGo's Nature cover

AlphaGo's first major research paper appeared on the cover of the journal Nature in January 2016.

Deep neural networks plus Monte Carlo tree search lead to AlphaGo; below, a branching tree with one gold path stands for Go’s enormous search space, where brute-force search failed.
Neural nets plus tree searchFlashFeed original diagram · CC-BY-4.0

Neural nets plus tree search

AlphaGo paired deep neural networks with Monte Carlo tree search — a combination that succeeded where brute-force search had failed for decades against Go's enormous search space.

The game record of game 4 between Lee Sedol and AlphaGo
Seoul, March 2016: 4-1Wesalius · CC BY-SA 4.0

Seoul, March 2016: 4-1

AlphaGo defeated Lee Sedol 4–1 in Seoul in March 2016, a very different result from the 5–0 sweep of Fan Hui in the earlier closed-door match.

By the numbers

Year AlexNet won the ImageNet competition
2012 year
Year DQN was published in Nature
2015 year
Number of Atari 2600 games the single DQN architecture was trained and evaluated on
49 games
Number of Atari games on which DQN reached human-level or better
29 games
Year AlphaGo defeated Lee Sedol in Seoul
2016 year
Year AlphaGo beat Lee Sedol in Seoul
2016 year

Matched pairs

Each item on the left goes with the one on the right.

Engineer who built AlphaGo's first prototype
Aja Huang
OpenAI cofounder who became CEO
Sam Altman
OpenAI's first chief scientist
Ilya Sutskever

Self-test

Answer each question from memory, then check your work against the answer key.

  1. 1. Which description best defines reinforcement learning as DeepMind applied it to Atari?

    1. AGroups are found in unlabelled data by measuring distance between points
    2. BAn agent learns by trial and error, receiving rewards for good actions and penalties for bad ones
    3. CA network compresses inputs and reconstructs them to find hidden structure
    4. DA network is shown labelled examples and corrected toward the right label
  1. 2. In what year was the Deep Q-Network (DQN) published in Nature?

    1. A2015
    2. B2016
    3. C2013
    4. D2012
  2. 3. How many Atari 2600 games was the single DQN architecture retrained on, and on how many did it reach human-level or better?

    1. A49 games, human-level or better on 49
    2. B37 games, human-level or better on 10
    3. C49 games, human-level or better on 29
    4. D29 games, human-level or better on 49
  3. 4. Which of the following was NOT part of what made DQN's Atari result a milestone?

    1. AOne general algorithm mastering many distinct environments
    2. BThe same code and hyperparameters used across every game
    3. CRaw screen pixels and a score signal as the only input
    4. DHand-engineered, game-specific features tuned for each title
  4. 5. Where and when did DeepMind first demonstrate early Atari results, before the Google acquisition closed?

    1. AThe Nature cover, January 2016
    2. BNeurIPS, December 2013
    3. CNeurIPS, December 2015
    4. DImageNet competition, 2012
  5. 6. What journal made AlphaGo's first major paper a cover story in January 2016?

    1. AScience
    2. BCell
    3. CPNAS
    4. DNature
  6. 7. AlphaGo is best defined as which of the following?

    1. AThe DeepMind program that defeated world champion Lee Sedol at Go in 2016
    2. BThe DeepMind network that learned 49 Atari games from raw pixels
    3. CThe OpenAI system announced as a counterweight to Google DeepMind
    4. DThe Toronto network that won the ImageNet competition in 2012
  7. 8. What made AlphaGo's March 2016 victory over Lee Sedol significant beyond the 4-1 scoreline?

    1. AIt showed that Lee Sedol was not among the strongest players of his generation
    2. BIt proved brute-force search could finally solve Go
    3. CIt made artificial intelligence a front-page story worldwide
    4. DIt was the first time AlphaGo had beaten a professional Go player
  8. 9. In which city and year did AlphaGo defeat Lee Sedol, and by what score?

    1. ASeoul, January 2016, 3–2
    2. BLondon, October 2015, 5–0
    3. CSeoul, March 2015, 5–0
    4. DSeoul, March 2016, 4–1
  9. 10. Which statement about AlphaGo is NOT accurate?

    1. AIts architecture was revealed in a January 2016 Nature cover paper
    2. BIt combined deep neural networks with Monte Carlo tree search
    3. CIt learned Go end to end from raw screen pixels alone
    4. DIt was built by DeepMind and defeated Lee Sedol in 2016
  10. 11. AlphaGo's 5-0 win over Fan Hui stayed secret until it appeared as a cover story in which publication?

    1. AScience
    2. BNature
    3. CMIT Technology Review
    4. DWired
  11. 12. In what month and year did Nature run the cover paper announcing AlphaGo?

    1. AJanuary 2016
    2. BOctober 2015
    3. CFebruary 2015
    4. DMarch 2016
  12. 13. Which engineer built AlphaGo's first prototype, contributing the critical tree-search component?

    1. AFan Hui
    2. BPieter Abbeel
    3. CAja Huang
    4. DIlya Sutskever
  13. 14. Who asked Hassabis on a May 2014 hallway walk what hard problem DeepMind would tackle next, then funded the answer despite being sceptical?

    1. ASergey Brin
    2. BSam Altman
    3. CElon Musk
    4. DLarry Page
  14. 15. Fan Hui held which title when he played AlphaGo in October 2015?

    1. AChinese Go champion
    2. BKorean national Go champion
    3. CWorld Go champion
    4. DEuropean Go champion
  15. 16. Lee Sedol took one game off AlphaGo in Seoul. Which game was it, and what explains his win?

    1. AGame one — AlphaGo's tree search ran out of thinking time
    2. BGame four — he found a move AlphaGo had not adequately evaluated
    3. CGame five — AlphaGo misread the final scoring
    4. DGame two — AlphaGo's shoulder-hit on the fifth line backfired
  16. 17. Which pairing of person and role is INCORRECT?

    1. AAja Huang — engineer whose tree-search work proved critical
    2. BFan Hui — European champion who later joined DeepMind as an analyst
    3. CSergey Brin — Google cofounder who funded the Go project despite scepticism
    4. DLee Sedol — European champion who lost 5–0 in a closed-door match
  17. 18. AlphaGo's Move 37 against Lee Sedol was so unusual that its policy network estimated a human would play it about how often?

    1. AOnce in 100 times
    2. BOnce in 1,000 times
    3. COnce in 10,000 times
    4. DOnce in a million times
  18. 19. In which game of the Seoul match, and at what board location, did AlphaGo play the move that became known as Move 37?

    1. AGame one, a shoulder-hit on the fourth line
    2. BGame two, a shoulder-hit on the fifth line
    3. CGame four, a shoulder-hit on the fifth line
    4. DGame two, a corner invasion on the third line
  19. 20. How did human grandmasters and commentators initially judge Move 37?

    1. AAs a mistake
    2. BAs a standard professional opening
    3. CAs an obvious winning move
    4. DAs a deliberate stalling tactic
  20. 21. How did Lee Sedol respond immediately after Move 37 was played?

    1. AHe replied instantly with a move that won the game
    2. BHe resigned the game on the spot
    3. CHe requested a review of AlphaGo's legality
    4. DHe left the board for about twelve minutes and returned visibly shaken
  21. 22. What has "Move 37" come to stand for?

    1. AThat AI systems fail in ways humans can easily predict
    2. BThat trained networks can discover strategy outside the corpus of human play
    3. CThat neural networks reliably reproduce expert human judgement
    4. DThat tree search alone can outperform human intuition
  22. 23. Which OpenAI cofounder became the organisation's CEO?

    1. ASam Altman
    2. BIlya Sutskever
    3. CPieter Abbeel
    4. DElon Musk
  23. 24. Who became OpenAI's first chief scientist, shaping its early research agenda?

    1. ASam Altman
    2. BPieter Abbeel
    3. CIlya Sutskever
    4. DGeoffrey Hinton
  24. 25. When was OpenAI announced, and how much funding was pledged?

    1. AMarch 2016, $1 billion in commitments
    2. BDecember 2015, $1 billion in commitments
    3. CDecember 2015, $100 million in commitments
    4. DMay 2014, $10 billion in commitments
  25. 26. What was the stated rationale for founding OpenAI as a non-profit?

    1. ATo act as a counterweight to Google's ownership of DeepMind
    2. BTo commercialise DeepMind's Atari research more quickly
    3. CTo build a Go program that could surpass AlphaGo
    4. DTo fund academic AI departments left behind by industry
  26. 27. Which of the following was NOT part of OpenAI's founding?

    1. AAn explicit framing as a counterweight to Google's DeepMind
    2. BA December 2015 announcement by Elon Musk and Sam Altman
    3. CA non-profit structure with $1 billion in pledged commitments
    4. DIlya Sutskever taking the CEO role while Sam Altman led research
  27. 28. How did Mallaby frame the December 2015 founding of OpenAI?

    1. AAs a merger with DeepMind
    2. BAs a spinoff of DeepMind's safety team
    3. CAs a counterweight to Google's DeepMind
    4. DAs a government-run AI lab
  28. 29. Who concluded that DeepMind 'under Google' was too dangerous a concentration of AI power, driving him to help found OpenAI?

    1. AElon Musk
    2. BSam Altman
    3. CSergey Brin
    4. DLarry Page

Go deeper

These are affiliate links: if you buy a book through one, FlashFeed may earn a small commission at no extra cost to you. The commission helps pay for FlashFeed's servers.

As an Amazon Associate we earn from qualifying purchases.

Answer key

  1. 1. B An agent learns by trial and error, receiving rewards for good actions and penalties for bad onesReinforcement learning is trial-and-error training driven by rewards and penalties — that is how DQN learned Atari from raw pixels. Learning from labelled examples describes supervised learning, not RL.
  2. 2. A 2015The DQN paper appeared in Nature in February 2015. 2013 is tempting — DeepMind showed early Atari results at NeurIPS that December — but the Nature publication came in 2015.
  3. 3. C 49 games, human-level or better on 29One architecture was retrained from scratch on 49 games and hit human-level or better on 29 of them. The 29/49 figures are often flipped — the smaller number is the successes.
  4. 4. D Hand-engineered, game-specific features tuned for each titleThe whole point was the absence of hand-engineered per-game features — earlier game-playing AI was hand-tuned per title. DQN used only raw pixels and score, with identical code and hyperparameters throughout.
  5. 5. B NeurIPS, December 2013DeepMind showed early Atari results at NeurIPS in December 2013, months ahead of the acquisition. 2012 was AlexNet's ImageNet win, and January 2016 was the AlphaGo Nature cover.
  6. 6. D Nature
  7. 7. A The DeepMind program that defeated world champion Lee Sedol at Go in 2016AlphaGo is DeepMind's Go program that beat Lee Sedol in 2016. The Atari-from-pixels system was DQN, a separate DeepMind project.
  8. 8. C It made artificial intelligence a front-page story worldwideThe Seoul match's global significance was cultural, not just technical: broadcast live to tens of millions across Asia, it compressed years of public AI awareness into one week and made AI a front-page story worldwide.
  9. 9. D Seoul, March 2016, 4–1AlphaGo beat Lee Sedol 4–1 in Seoul in March 2016. The 5–0 sweep was the earlier closed-door match against Fan Hui in October 2015.
  10. 10. C It learned Go end to end from raw screen pixels aloneLearning from raw pixels with no search describes DQN on Atari, not AlphaGo — AlphaGo's networks worked together with Monte Carlo tree search, and the January 2016 Nature cover revealed that architecture.
  11. 11. B NatureThe Fan Hui result was embargoed until the January 2016 Nature cover paper that announced AlphaGo.
  12. 12. A January 2016AlphaGo's Fan Hui result was embargoed until Nature's January 2016 cover paper, which also revealed AlphaGo's architecture.
  13. 13. C Aja HuangAja Huang built AlphaGo's first prototype; his tree-search work made the system competitive at professional level. Fan Hui joined DeepMind later, as an analyst rather than an engineer.
  14. 14. A Sergey BrinSergey Brin posed the question, heard "Go," doubted it was winnable, and provided the resources anyway. Musk and Altman were on the other side of this story — they founded OpenAI as a counterweight.
  15. 15. D European Go championFan Hui was the European Go champion and lost 5–0 in the closed-door match. Lee Sedol, not Fan Hui, was the world-class player widely called the greatest of his generation.
  16. 16. B Game four — he found a move AlphaGo had not adequately evaluatedLee Sedol won game four with a move AlphaGo had not foreseen — his lone victory in a 4–1 defeat. Game two belonged to AlphaGo; that was the game containing Move 37, which worked exactly as intended.
  17. 17. D Lee Sedol — European champion who lost 5–0 in a closed-door matchThe 5–0 closed-door loss was Fan Hui's, in October 2015. Lee Sedol was the Korean champion who lost 4–1 in Seoul in March 2016 — a public match, not a closed-door one.
  18. 18. C Once in 10,000 timesMove 37 — a shoulder-hit on the fifth line — was called a mistake by human commentators, but AlphaGo's policy network put the odds a human would choose it at roughly one in ten thousand. It won territory human intuition would not reach.
  19. 19. B Game two, a shoulder-hit on the fifth lineMove 37 was a shoulder-hit on the fifth line in game two. Game four was the one Lee Sedol won, with a move AlphaGo had not adequately evaluated.
  20. 20. A As a mistakeCommentators were stopped cold and rated it a mistake. AlphaGo's reasoning turned out to be correct — the move won territory in a way human intuition would not reach.
  21. 21. D He left the board for about twelve minutes and returned visibly shakenLee Sedol spent roughly twelve minutes away from the board and came back visibly shaken. He did win game four later, but game two was AlphaGo's.
  22. 22. B That trained networks can discover strategy outside the corpus of human playMove 37 is shorthand for networks finding strategic concepts beyond human play. Reproducing expert judgement is the opposite of the point — no human grandmaster would have chosen it.
  23. 23. A Sam AltmanSam Altman became CEO. Musk cofounded and announced OpenAI alongside him, and Sutskever took the chief scientist role rather than the top job.
  24. 24. C Ilya SutskeverIlya Sutskever became OpenAI's first chief scientist. Hinton was his doctoral adviser in Toronto but did not take the role; Abbeel was among the hires who followed later.
  25. 25. B December 2015, $1 billion in commitmentsMusk and Altman announced OpenAI in December 2015 with $1 billion in funding commitments — timing that coincided almost exactly with AlphaGo's closed-door win over Fan Hui.
  26. 26. A To act as a counterweight to Google's ownership of DeepMindOpenAI was framed explicitly as a counterweight to Google's ownership of DeepMind — Musk saw frontier capability concentrated in one commercial giant as dangerous.
  27. 27. D Ilya Sutskever taking the CEO role while Sam Altman led researchThe roles are reversed: Sam Altman was CEO and Ilya Sutskever was chief scientist. The other three describe the founding accurately.
  28. 28. C As a counterweight to Google's DeepMindMallaby frames OpenAI's founding as a non-profit counterweight to Google's ownership of DeepMind.
  29. 29. A Elon MuskMusk decided a single commercial giant controlling frontier AI was dangerous, with DeepMind under Google his primary concern.

Image credits

Download all 12 lessons: PDF · EPUB