LLM 101 study guide · Lesson 2 of 10

How LLMs Work

Tokens, embeddings and attention · How text is generated

This is the lesson's study guide from the FlashFeed app, plus its key facts and a self-test.

What you'll be able to do

  • Explain how text becomes tokens and then embedding vectors a model can compute over.
  • Describe the Transformer, self-attention, context window, and parameters that make up a model's internals.
  • Describe next-token prediction and autoregressive generation as the core loop that produces text.
  • Explain how logits and the temperature knob shape which next token is sampled.
  • Explain why LLMs hallucinate and recognize emergent abilities that appear only at scale.

Underneath the conversational polish, a large language model is doing something surprisingly mechanical: turning text into numbers, weighing how those numbers relate, and predicting what comes next, one piece at a time. You don't need the math to grasp the machine — just a handful of core concepts. This lesson builds that mental model.

Tokens, embeddings and attention

Everything starts with the token, the basic unit of text a model processes — on average about three-quarters of a word. A common word like "the" is a single token, while a rarer word like "ChatGPT" splits into several. The process of cutting text into these pieces is tokenization, usually done with an algorithm like Byte-Pair Encoding against a fixed vocabulary whose size varies by model, from roughly 32,000 tokens in older models to 200,000 or more in recent ones. This matters more than it sounds: context limits, API pricing, and speed are all measured in tokens, not words.

Each token is then turned into an embedding — a long list of numbers (a vector) that places the token's meaning in a high-dimensional space, where similar meanings sit close together. Embeddings are how a model converts discrete symbols into continuous math it can actually compute over. Those embeddings flow through the Transformer, an architecture that processes all tokens in a sequence at once, in parallel, rather than one at a time like an RNN — which is exactly what makes it fast enough to train at massive scale on GPUs. The Transformer's key trick is self-attention: every token looks at the other tokens in the context (in a text-generating model, the ones before it) and learns how much to "attend to" each one. That is how a model connects a pronoun to the noun it refers to, or a question word to its answer, regardless of how far apart they sit. Two more concepts round out the picture: the context window, the maximum number of tokens the model can see at once (its working memory — anything outside it is invisible), and the parameters or weights, the billions of learned numbers inside the model that encode everything it "knows."

How text is generated

With that machinery in place, generation itself is simple to describe. The model's core skill, learned in training, is next-token prediction: given all previous tokens, estimate how likely each possible next token is, then pick one. Generation is just this done repeatedly — autoregressive generation, where each new token is chosen based on all the tokens before it, including the ones the model just produced. A crucial consequence: the model can't go back and edit. It commits to each token in sequence, which is why a small early mistake can compound into a larger one.

How does it choose? For every possible next token the model computes raw scores called logits, which are converted into probabilities. The temperature knob then shapes that choice: low temperature makes the model pick the safe, high-probability token (focused, deterministic output), while high temperature flattens the odds and invites surprise (creative, varied output). Temperature is the main dial for controlling how predictable a model's responses are.

This same process explains one of the technology's best-known flaws. A hallucination is fluent, confident text that is simply wrong or fabricated. It happens because the model is built to produce plausible text, not to retrieve verified facts — it has no internal fact-checker, so it fills in convincing-but-wrong details exactly the way it fills in correct ones. Grounding the model with real sources (the job of RAG) is the main mitigation. Finally, scale appears to bring surprises: emergent abilities like multi-step arithmetic or step-by-step reasoning have been reported in large models but not in smaller ones — capability jumps that researchers didn't predict, though some argue the jumps partly reflect how abilities are measured.

Key terms

token
The basic unit of text an LLM processes — in English, on average about three-quarters of a word; a common word like 'the' is one token, while a rarer word like 'ChatGPT' splits into several. It matters because everything — context limits, pricing, and speed — is measured in tokens, not words or characters.
tokenization
The process of splitting raw text into tokens before feeding it to a model, using algorithms such as Byte-Pair Encoding (BPE). It matters because the vocabulary (from about 32K tokens in older models to 200K or more in recent ones) determines what the model 'sees' and how it handles rare or foreign words.
embedding
A list of numbers (a vector) that represents a token's meaning in a high-dimensional space, where tokens with similar meanings sit close together. It matters because embeddings are how a model turns discrete text symbols into continuous math it can compute over.
Transformer
A neural network architecture that processes all tokens in a sequence at once, in parallel, using self-attention layers, rather than one token at a time like an RNN. It matters because that parallelism makes Transformers far faster to train on GPUs, which is what enabled today's scale.
self-attention
A mechanism where each token looks at the other tokens in the context (in a text-generating model, the ones before it) and learns how much to attend to (weight) each one when forming its own representation. It matters because it lets the model link a pronoun to its noun or a question to its answer, no matter how far apart they sit in the text.
context window
The maximum number of tokens a model can see at once — its working memory for a single conversation or document. It matters because anything outside the window is invisible to the model; larger windows allow longer documents, conversations, and multi-step tasks.
parameters (weights)
The millions or billions of numerical values inside a model's layers, learned during training, that encode everything the model 'knows.' It matters because parameter count is the main proxy for a model's capacity — more parameters generally means more capability, up to a point.
next-token prediction
The core objective: given all previous tokens, predict how likely each possible next token is, then pick one — repeated over and over to generate text. It matters because this deceptively simple task, applied at massive scale, is what gives LLMs their broad language ability.
autoregressive generation
Producing text one token at a time, where each new token is chosen based on all prior tokens — including the ones the model just generated. It matters because the model can't go back and edit; it commits to each token in sequence, which is why small errors can compound.
logits
The raw, unnormalized scores a model assigns to every possible next token, converted into a probability distribution by the softmax function. It matters because understanding logits clarifies what 'temperature' and other sampling knobs are actually adjusting.
temperature
A sampling knob (often 0–1 or 0–2, depending on the provider) that scales the logits: low temperature makes output more deterministic and focused; high temperature makes it more random and creative. It matters because it is the primary dial for controlling how predictable or varied a model's output is.
hallucination
When an LLM generates fluent, confident-sounding text that is factually wrong or entirely fabricated, grounded in no real source. It matters because users often can't tell a hallucination from an accurate answer, making verification essential for high-stakes use.
emergent abilities
Capabilities that appear in large models but are absent in smaller ones of the same type — not predictable by smoothly extrapolating from smaller scale. Examples include multi-step arithmetic and chain-of-thought reasoning. It matters because emergence means capability jumps can surprise researchers, though there is debate over whether it is a true phase transition or an artifact of how abilities are measured.

Key concepts

Embedding

  • The step that converts a discrete symbol into continuous math a model can actually compute over.
  • It's a long list of numbers — a vector — placing a token's meaning in a high-dimensional space where similar meanings sit close together.
  • Nearly every neural language model since word2vec (2013) relies on some learned form of these to represent meaning geometrically.

Self-attention

  • The Transformer's key trick, and the mechanism that let the architecture drop recurrence entirely.
  • Each token in a sequence looks at the other tokens in its context and learns how much to weight each one.
  • It's how a model links a pronoun to the noun it refers to, no matter how far apart they sit.

Context window

  • A model's working memory: anything outside it is simply invisible to the model.
  • It's the maximum number of tokens the model can see at once.
  • For GPT-4 this began at 8,000 tokens; exceed it and the earliest text drops out of view.

Temperature

  • The main dial for controlling how predictable a model's responses are.
  • Set it low for focused, deterministic output; set it high to flatten the odds and invite surprise.
  • It reshapes the probabilities derived from the model's logits before the next token is chosen.

Hallucination

  • One of the technology's best-known flaws.
  • Fluent, confident text that happens to be simply wrong or fabricated.
  • It occurs because the model produces plausible text, not verified facts; grounding it with real sources via RAG is the main fix.

Key facts

Learned patterns produce two answer sheets that look exactly alike, one marked Correct with a check and one Wrong with a cross: both sound equally plausible.
Why hallucinations happenFlashFeed original diagram · CC-BY-4.0

Why hallucinations happen

LLMs are trained to produce plausible text, not to retrieve verified facts. With no internal fact-checker, they interpolate from patterns in their training data — so they 'fill in' convincing but wrong details the same way they fill in correct ones. Grounding techniques like RAG are the main mitigation.

Five tokens (The, cat, sat, on, the) enter a large language model (LLM), which ranks the next token by likelihood: mat, then sofa, then roof, as bars with no values.
Next-token predictionFlashFeed original diagram · CC-BY-4.0

Next-token prediction

The core skill an LLM learns in training is next-token prediction: given all previous tokens, estimate how likely each possible next token is. Its broader language ability follows from that one skill.

A chain of eight tokens committed in order: a small slip in the third builds into the later tokens, tinted red, and there is no going back to edit.
Why early errors compoundFlashFeed original diagram · CC-BY-4.0

Why early errors compound

A model can't go back and edit its own output — it commits to each token in sequence, so a small factual slip early in an answer compounds as later tokens build on the mistake.

What logits are
What logits areAI-generated (gpt-image-1.5)

What logits are

Logits are the raw, unnormalized scores an LLM assigns to every possible next token, before those scores are converted into probabilities.

One prompt, two settings: low temperature favors the top token and gives three identical answers; higher temperature samples more evenly and gives three different answers.
Low temperature, focused outputFlashFeed original diagram · CC-BY-4.0

Low temperature, focused output

Setting an LLM's temperature low makes it favor the safe, high-probability token, producing output that is focused and as deterministic as possible.

Self-test

Answer each question from memory, then check your work against the answer key.

  1. 1. On average, roughly how much of a word does a single token represent?

    1. AExactly one word
    2. BAbout one-third of a word
    3. CAbout two words
    4. DAbout three-quarters of a word
  1. 2. Which algorithm is a common way to split raw text into tokens?

    1. ASoftmax normalization
    2. BByte-Pair Encoding (BPE)
    3. CBackpropagation
    4. DGradient descent
  2. 3. A tokenizer works against a fixed vocabulary. Roughly how large is that vocabulary in today's models?

    1. AAbout 500,000 to 1,000,000 tokens
    2. BAbout 3,000 to 5,000 tokens
    3. CAbout 32,000 to 200,000+ tokens
    4. DAbout 10 million tokens
  3. 4. What is an embedding, and what does it accomplish for the model?

    1. AA vector of numbers placing a token's meaning in a space where similar meanings sit close together
    2. BA compressed copy of the training text stored for later lookup
    3. CA limit on how many tokens the model can read at once
    4. DA raw score assigned to each candidate next token
  4. 5. Two tokens' embedding vectors sit very close together in the embedding space. What does that closeness indicate?

    1. AThey appear next to each other in the sentence being processed
    2. BThey are equally likely to be chosen as the next token
    3. CTheir meanings are similar
    4. DThey are spelled with a similar number of characters
  5. 6. What distinguishes the Transformer from an RNN, and why does that matter?

    1. AIt removes the need for a fixed context window, letting it read unlimited text
    2. BIt processes tokens strictly one at a time, which improves accuracy on long documents
    3. CIt stores training text verbatim, so it never needs to compute embeddings
    4. DIt processes all tokens in a sequence at once in parallel, making it fast enough to train at massive scale on GPUs
  6. 7. Self-attention lets a model connect a pronoun to the noun it refers to. How does the mechanism achieve this?

    1. ATokens are re-sorted so related words end up adjacent in the sequence
    2. BEach token looks at the other tokens in its context and learns how much to attend to each one
    3. CEach token is compared only with the token immediately before it
    4. DA separate lookup table maps every pronoun to its most common noun
  7. 8. A document is longer than the model's context window. What happens to the portion that doesn't fit?

    1. AIt is invisible to the model — the window is the maximum it can see at once
    2. BIt is processed first, then the in-window text is discarded
    3. CIt is automatically converted into extra embeddings that bypass the limit
    4. DIt is stored in the parameters and recalled as needed
  8. 9. In an LLM, what are the parameters (weights)?

    1. AThe vocabulary entries the tokenizer matches text against
    2. BThe tokens currently held in the model's working memory
    3. CThe billions of learned numbers inside the model that encode everything it 'knows'
    4. DThe settings a user adjusts at request time, such as temperature
  9. 10. Which pairing of a model internal with its role is INCORRECT?

    1. AParameters — the maximum number of tokens the model can see at once
    2. BSelf-attention — each token weighs how much to attend to the other tokens in its context
    3. CTransformer — an architecture that processes a sequence in parallel
    4. DContext window — the model's working memory for a single conversation
  10. 11. What is the core skill an LLM learns in training, from which its broad language ability follows?

    1. ARanking whole candidate answers and returning the highest-scoring one
    2. BRetrieving the closest matching passage from a database of verified facts
    3. CTranslating the prompt into a formal logical representation before answering
    4. DNext-token prediction — given all previous tokens, predict how likely each possible next token is
  11. 12. In autoregressive generation, what does the model condition each new token on?

    1. AA draft of the full response written before any token is emitted
    2. BAll the tokens before it, including the ones the model itself just produced
    3. COnly the user's original prompt, ignoring its own output so far
    4. DOnly the single token immediately preceding it
  12. 13. A model makes a small factual slip early in a long answer. Why can that slip grow into a bigger problem?

    1. AThe mistake pushes the earlier correct text outside the context window
    2. BIt can't go back and edit — it commits to each token in sequence, so later tokens build on the mistake
    3. CEach mistake is written into the model's parameters and reused later
    4. DThe error raises the temperature setting for the rest of the response
  13. 14. A model is midway through a sentence and has produced three tokens. When is the fourth token decided?

    1. AAt training time, when the phrase was memorized from the data
    2. BAt the same moment as the others, since the Transformer works in parallel
    3. COnly once the first three exist — it is chosen conditioned on them
    4. DBefore generation started, when the model planned out the whole sentence
  14. 15. How do next-token prediction and autoregressive generation relate to each other?

    1. AAutoregressive generation predicts the whole sequence at once, then next-token prediction checks it
    2. BThey are competing architectures; a model uses one or the other, never both
    3. CNext-token prediction runs only at training time; generation uses a different objective
    4. DAutoregressive generation is next-token prediction applied repeatedly, feeding each new token back in
  15. 16. What are logits in an LLM?

    1. AThe raw, unnormalized scores assigned to every possible next token
    2. BThe learned weights stored inside the model's layers
    3. CThe vectors that encode each token's meaning in high-dimensional space
    4. DThe normalized probabilities the model finally assigns to each candidate token
  16. 17. Before the next token is chosen, what happens to the logits?

    1. AAppended to the context window
    2. BConverted into probabilities
    3. CSaved as parameters
    4. DStored as embeddings for later lookup
  17. 18. You need output that is as focused and deterministic as possible. How should you set the temperature?

    1. AHigh — it sharpens the odds toward the single best token
    2. BIt makes no difference; temperature only affects generation speed
    3. CLow — it makes the model pick the safe, high-probability token
    4. DHigh — it widens the search so the model finds the correct answer
  18. 19. Where in the generation process does the temperature setting take effect?

    1. AIt updates a subset of the model's parameters for the current request
    2. BIt edits the embeddings of the prompt tokens before they enter the model
    3. CIt expands the context window so more candidates are considered
    4. DIt scales the logits before the next token is sampled
  19. 20. Which statement about the temperature knob is NOT true?

    1. ASetting it to 0 guarantees the answer will be factually correct
    2. BIt is the main dial for controlling how predictable a model's responses are
    3. CAt low settings it favors the safe, high-probability token
    4. DHigh values flatten the odds and invite surprising word choices
  20. 21. Why do large language models 'hallucinate' — state false things with confidence?

    1. AThey are built to produce plausible-sounding text, not to retrieve verified facts, and have no internal fact-checker
    2. BTheir training data is always out of date, so every answer is stale
    3. CThey deliberately invent details to fill up the context window
    4. DThe temperature setting is always too high, forcing random errors
  21. 22. What is the defining description of a hallucination?

    1. AText that exceeds the model's context window and gets truncated
    2. BOutput that is obviously garbled and easy for a user to spot
    3. CAny answer the model refuses to give
    4. DFluent, confident text that is factually wrong or entirely fabricated
  22. 23. What is the main mitigation for hallucination?

    1. AEnlarging the tokenizer's vocabulary
    2. BReducing the model's parameter count
    3. CGrounding the model with real sources — the job of RAG
    4. DLowering the temperature to zero on every request
  23. 24. What makes an ability 'emergent'?

    1. AIt fades away as a model is scaled up
    2. BIt appears in large models but is absent in smaller ones of the same type
    3. CIt shows up only when temperature is set high
    4. DIt was explicitly programmed in by researchers before training
  24. 25. Which pair of capabilities are examples of emergent abilities?

    1. ASelf-attention and parallel processing
    2. BMulti-step arithmetic and step-by-step (chain-of-thought) reasoning
    3. CSoftmax normalization and logit scaling
    4. DTokenization and embedding lookup

Go deeper

Answer key

  1. 1. D About three-quarters of a wordA token is the basic unit an LLM processes — in English, on average about three-quarters of a word. Common words like 'the' are one token, while rarer strings like 'ChatGPT' split into several pieces. Tokens (not words) are what context limits, pricing, and speed are measured in.
  2. 2. B Byte-Pair Encoding (BPE)Tokenization splits raw text into tokens using algorithms such as Byte-Pair Encoding. Softmax converts raw next-token scores into probabilities, and gradient descent and backpropagation are training procedures — none of them cut text into pieces.
  3. 3. C About 32,000 to 200,000+ tokensVocabularies run from about 32K tokens in older models to about 100K (GPT-4) and 200K or more in recent ones. That size determines what the model 'sees' and how it handles rare or foreign words; a few thousand entries would be far too few, and half a million or more far too many.
  4. 4. A A vector of numbers placing a token's meaning in a space where similar meanings sit close togetherAn embedding is a vector representing a token's meaning in high-dimensional space, turning discrete symbols into continuous math. The limit on tokens read at once is the context window; raw next-token scores are logits.
  5. 5. C Their meanings are similarAn embedding places a token's meaning in a high-dimensional space where tokens with similar meanings sit close together. Closeness encodes meaning — not spelling, not position in the sentence, and not next-token probability.
  6. 6. D It processes all tokens in a sequence at once in parallel, making it fast enough to train at massive scale on GPUsThe Transformer's parallel processing of the whole sequence — unlike an RNN's one-at-a-time approach — is what makes large-scale GPU training feasible. It still has a context window, and one-at-a-time is the RNN, not the Transformer.
  7. 7. B Each token looks at the other tokens in its context and learns how much to attend to each oneIn self-attention each token weighs the other tokens in its context, learning how much to attend to each, which is why distance in the sequence doesn't break the link. Comparing only with the previous token or reordering the sequence is not what happens.
  8. 8. A It is invisible to the model — the window is the maximum it can see at onceThe context window is the maximum number of tokens the model can see at once — its working memory — and anything outside it is simply invisible. Parameters encode learned knowledge from training, not text you paste in now.
  9. 9. C The billions of learned numbers inside the model that encode everything it 'knows'Parameters are the billions of numerical values learned during training that encode the model's knowledge, and their count is the main proxy for capacity. Temperature is a sampling knob; the working memory is the context window.
  10. 10. A Parameters — the maximum number of tokens the model can see at onceThe maximum number of tokens visible at once is the context window, not the parameters. Parameters are the learned weights that encode what the model knows. The other three pairings are correct.
  11. 11. D Next-token prediction — given all previous tokens, predict how likely each possible next token isThe model's core learned objective is next-token prediction: given all previous tokens, estimate how likely each possible next token is. Retrieval from a verified database is what grounding/RAG adds — it is not the base skill.
  12. 12. B All the tokens before it, including the ones the model itself just producedAutoregressive generation chooses each new token based on all prior tokens — the prompt plus the model's own just-generated output. It never drafts the full answer first, and it uses far more than just the previous token.
  13. 13. B It can't go back and edit — it commits to each token in sequence, so later tokens build on the mistakeBecause generation is autoregressive and committed token by token, the model can't revise; subsequent tokens condition on the error, so it compounds. Parameters are fixed after training, and errors don't change temperature.
  14. 14. C Only once the first three exist — it is chosen conditioned on themGeneration is autoregressive: tokens come out one at a time, each conditioned on everything before it. The Transformer reads a sequence in parallel, but it still emits tokens sequentially and never plans the full sentence up front.
  15. 15. D Autoregressive generation is next-token prediction applied repeatedly, feeding each new token back inGeneration is simply next-token prediction done repeatedly, each new token conditioned on everything before it. The same learned objective drives both training and generation — they aren't rival mechanisms.
  16. 16. A The raw, unnormalized scores assigned to every possible next tokenLogits are the raw, unnormalized scores for every candidate next token; they only become probabilities after normalization, so the finished probability values are not the logits. Meaning vectors are embeddings.
  17. 17. B Converted into probabilitiesThe model's raw next-token scores (logits) are converted into probabilities, and temperature shapes which token is chosen from them. Logits are not tokens added to the context, not saved as parameters, and not stored as embeddings for later lookup.
  18. 18. C Low — it makes the model pick the safe, high-probability tokenLow temperature makes the model favor the safe, high-probability token, giving focused, deterministic output. High temperature does the opposite — it flattens the odds and invites surprise.
  19. 19. D It scales the logits before the next token is sampledTemperature is a sampling knob: it scales the logits — the raw scores — so the token draw is more or less random. It never touches the embeddings, the trained parameters, or the size of the context window.
  20. 20. A Setting it to 0 guarantees the answer will be factually correctTemperature shapes how random the sampling is, not whether the content is true — at 0 the model just reliably picks its highest-scoring token, right or wrong. The other three statements describe it accurately.
  21. 21. A They are built to produce plausible-sounding text, not to retrieve verified facts, and have no internal fact-checkerA hallucination is fluent, confident text that's wrong. It happens because the model fills in plausible continuations the same way whether they're true or false — it has no fact-checker. Grounding it with real sources (RAG) is the main fix.
  22. 22. D Fluent, confident text that is factually wrong or entirely fabricatedA hallucination is fluent, confident-sounding text that is simply wrong or fabricated. The danger is precisely that it is NOT obviously garbled — users often can't tell it from an accurate answer.
  23. 23. C Grounding the model with real sources — the job of RAGGrounding the model in real sources, which is what RAG does, is the main mitigation. Temperature shapes randomness, not truth, and vocabulary or parameter count don't add a fact-checker.
  24. 24. B It appears in large models but is absent in smaller ones of the same typeEmergent abilities appear in large models but are absent in smaller ones — capability jumps researchers don't fully predict. They arise from scale, not from explicit programming or a sampling setting.
  25. 25. B Multi-step arithmetic and step-by-step (chain-of-thought) reasoningMulti-step arithmetic and chain-of-thought reasoning are the commonly cited examples of abilities that appear only at scale (though how sharp those jumps really are is debated). Tokenization, self-attention, and softmax are built-in mechanics present at any model size.

Image credits

  • Why hallucinations happen: FlashFeed original diagram · CC-BY-4.0
  • Next-token prediction: FlashFeed original diagram · CC-BY-4.0
  • Why early errors compound: FlashFeed original diagram · CC-BY-4.0
  • What logits are: AI-generated (gpt-image-1.5)
  • Low temperature, focused output: FlashFeed original diagram · CC-BY-4.0

Download all 10 lessons: PDF · EPUB