LLM 101 study guide · Lesson 4 of 10

Prompting, RAG and Using LLMs

Prompting techniques · RAG, tools and agents

This is the lesson's study guide from the FlashFeed app, plus its key facts and a self-test.

What you'll be able to do

  • Describe what a prompt and a system prompt contain, and why prompting, though free, can't give a model knowledge it wasn't trained on.
  • Distinguish zero-shot, few-shot and chain-of-thought prompting, explain in-context learning, and recognize step-by-step reasoning as emergent.
  • Explain RAG and its supporting components — vector databases and chunking — for grounding a model in external knowledge.
  • Describe function calling and LLM agents, and how an agent's planning loop differs from a single-prompt interaction.
  • Contrast fine-tuning with RAG as ways to add knowledge or behavior, one baking into weights and the other into context.

Prompting techniques

A prompt is everything a model receives as input on a call: the developer's instructions, any earlier turns of the conversation, and the user's current message. Because inference leaves the weights frozen, the prompt is the main way to steer what a trained model does. OpenAI's API documentation describes prompt engineering as the process of writing effective instructions so that a model consistently produces what you need.

System prompts

Chat APIs label each message by role. The system prompt (OpenAI's current API calls it the developer message) is the block of instructions an application's developer supplies ahead of the conversation to set the model's role, tone, rules and task; the user's requests arrive separately as user messages. OpenAI's documentation says developer instructions are prioritized ahead of user messages. Anthropic's prompting guide (2026) adds that setting a role in the system prompt focuses the model's behavior and tone, that instructions should be clear and direct about the desired output format and constraints, and that prompt engineering starts from clear success criteria, a way to test against them empirically, and a first draft to improve.

Zero-shot, one-shot and few-shot

The GPT-3 paper (Brown et al., 2020) named three ways of posing a task through the prompt alone. In zero-shot prompting the model gets only a natural-language instruction describing the task, with no examples. One-shot prompting adds a single worked example, and few-shot prompting adds a few, which the paper calls demonstrations. In every setting no weight updates are allowed: the model adapts through in-context learning, and nothing it picks up persists once the text leaves the context window. The paper found that scaling up models greatly improved few-shot performance. Provider guides still lean on examples: Anthropic's 2026 guide says a few well-crafted examples improve accuracy and consistency, and suggests three to five.

Chain-of-thought prompting

Chain-of-thought (CoT) prompting gets the model to write out a series of intermediate reasoning steps before its final answer. Wei et al. (2022) did this with a few worked examples whose answers spelled out the steps, and it improved performance on arithmetic, commonsense and symbolic reasoning tasks. They reported that this reasoning ability emerges in sufficiently large models, making it one of the emergent abilities described in How LLMs Work. Kojima et al. (2022) then showed zero-shot chain-of-thought: adding "Let's think step by step" before the answer, with no examples at all, also improved reasoning.

RAG, tools and agents

Prompting alone can't give a model knowledge it wasn't trained on. Three patterns fill that gap without retraining: RAG, function calling, and agents.

Prompting vs. RAG vs. fine-tuning

Prompting costs nothing but is bounded by context and existing weights. RAG (Retrieval-Augmented Generation), introduced by Lewis et al. (2020), retrieves documents from an external store and inserts them as context before the model answers — ideal for fresh or proprietary knowledge. It typically relies on a vector database that stores document embeddings and runs fast similarity search, and on chunking (splitting documents into focused segments before embedding) so retrieved text fits the context window. Chunk size is a key tuning choice: chunks that are too large waste context, while chunks that are too small lose coherence, because they lack the surrounding context needed to answer a query. Fine-tuning bakes persistent behavior into the weights: expensive but durable across calls. For real-world capabilities, function calling (tool use) lets the model emit a structured call to an external API; the host runs it and feeds the result back. An LLM agent wraps this in a planning loop — the model acts as a controller that plans, selects tools, takes actions, and observes results until a goal is met.

Key terms

prompt
The input sent to an LLM, typically a system message (behavior instructions), zero or more prior turns, and the current user message. It matters because the weights are frozen at inference, so the prompt is the main way to steer the model, which makes prompt engineering a high-leverage skill.
system prompt
The block of instructions an application's developer supplies ahead of the conversation to set the model's role, tone, rules and task context. It matters because it is where developers configure a model's behavior for their specific application.
zero-shot prompting
Asking a model to perform a task with only a natural-language instruction and no examples, such as 'Translate this to French.' It matters because it is the baseline setting that the GPT-3 paper compared against one-shot and few-shot prompting.
in-context learning
An LLM's ability to adapt its behavior from examples or instructions in the prompt, without changing any weights. It matters because it blurs the line between 'prompting' and 'training' in people's minds — but no learning persists once the context window ends.
chain-of-thought (CoT) prompting
Prompting the model to reason step by step (for example, 'Let's think step by step') before giving a final answer, via instruction or worked examples. It matters because chain-of-thought sharply improves performance on arithmetic and multi-step reasoning, especially in larger models.
few-shot prompting
Putting a few worked examples of the task in the prompt before asking the model to handle a new case. It matters because examples steer the output's format and improve accuracy and consistency with no weight updates, teaching the task inside the context window.
RAG (Retrieval-Augmented Generation)
A pattern in which relevant documents are retrieved from an external store and inserted into the prompt as context before the model answers. Introduced by Lewis et al. (2020). It matters because it gives a model access to fresh, proprietary, or specific knowledge without retraining it.
vector database
A specialized database that stores document embeddings and supports fast similarity search, returning the most semantically relevant chunks for a query. Examples include Pinecone, Weaviate, and Chroma. It matters because it is the retrieval backend that makes RAG systems work at scale.
chunking
Splitting source documents into smaller segments before embedding them for RAG, so retrieved passages fit the context window and stay semantically focused. It matters because chunk size and overlap are key knobs: too large wastes context, too small loses coherence.
function calling / tool use
A structured way to let an LLM invoke external tools — APIs, calculators, databases — by emitting a well-formed call the host application runs, then feeding the result back to the model. It matters because it grounds the model in real-time data and gives it abilities, like exact math or search, that pure text generation lacks.
LLM agent
A system in which an LLM acts as a controller that plans, selects tools, takes actions, and observes results in a loop until a goal is met, rather than answering a single prompt. It matters because agents extend LLMs from one-shot answerers into multi-step problem solvers that can browse, write and run code, and manage files.

Key concepts

RAG

  • A pattern that gives a model knowledge it wasn't trained on, without any retraining.
  • It fetches relevant documents from an external store and inserts them as context before the model answers.
  • Ideal for fresh or proprietary knowledge, it typically relies on a vector database and on chunking documents to fit the context window.

Function calling

  • One of three patterns for real-world capability, also known simply as tool use.
  • The model emits a structured request to an external API.
  • The host executes that request and feeds the result back into the model.

LLM agent

  • The most autonomous of the three patterns for extending a model.
  • It wraps tool use in a planning loop.
  • The model becomes a controller that plans, selects tools, takes actions, and observes results until a goal is met.

Vector database

  • The storage engine RAG typically relies on for retrieval.
  • It holds document embeddings and runs fast similarity search over them.
  • RAG queries it to find the most relevant chunks before the model answers.

Fine-tuning

  • The one approach here that actually changes a model's weights rather than just its context.
  • It bakes persistent behavior into the model — expensive, but durable across calls.
  • Unlike prompting or RAG, its effect survives from one call to the next because it lives in the weights themselves.

Key facts

A field of dots: the query contract renewals sits near three blue document dots in a dashed circle, nearest by meaning even without that phrase; gray groups far away differ in meaning.
Vector databases, explainedFlashFeed original diagram · CC-BY-4.0

Vector databases, explained

A vector database stores document embeddings and finds relevant material through fast similarity search — matching by meaning, not exact keyword matching.

A tall source document split into six chunks; the two retrieved chunks, in gold, fit inside a shorter context window with room to spare, so they stay focused.
Why RAG chunks documentsFlashFeed original diagram · CC-BY-4.0

Why RAG chunks documents

Source documents are split into smaller segments before being embedded for RAG so the retrieved passages fit the context window and stay semantically focused.

A loop: Goal, select a tool, take action, observe the result, then check Goal met? Not yet returns to select a tool; Yes stops.
The agent planning loopFlashFeed original diagram · CC-BY-4.0

The agent planning loop

An LLM agent differs from a single-prompt interaction by running a planning loop: it acts as a controller that selects tools, takes action, and observes results until the goal is met.

What function calling produces

In function calling, the model itself produces a structured, well-formed call — it's the host application that actually runs that call against the real API or tool.

Step by step

Engineer a robust system prompt

  1. State the assistant's role in the system prompt; even one sentence focuses its behavior and tone.
  2. Give clear, direct instructions for the task and the rules it must follow.
  3. Specify the desired output format and constraints explicitly.
  4. Include a few worked examples (Anthropic suggests three to five) showing the desired behavior.
  5. For multi-step reasoning, ask the model to think step by step before answering.
  6. Test the prompt against clear success criteria and revise the draft.

Build a basic RAG system

  1. Chunk your source documents into focused passages.
  2. Embed each chunk with an embedding model.
  3. Store the embeddings in a vector database.
  4. At query time, embed the user's question.
  5. Retrieve the top-k most similar chunks.
  6. Insert the retrieved chunks plus the question into the LLM prompt.
  7. Have the model generate an answer grounded in those chunks.

Build a retrieval-augmented generation (RAG) pipeline

  1. Chunk your source documents into focused passages with some overlap.
  2. Embed every chunk using a sentence-embedding model into a vector database.
  3. At query time, embed the user's question and run k-nearest-neighbor search.
  4. Concatenate the top-k passages into a context block for the LLM prompt.
  5. Send the augmented prompt to the LLM and stream the response back to the user.
  6. Optionally re-rank retrieved passages with a cross-encoder for higher accuracy.

Self-test

Answer each question from memory, then check your work against the answer key.

  1. 1. What is the key limitation of prompting alone?

    1. AIt is limited to knowledge the model already has
    2. BIt requires a vector database to function
    3. CIt permanently rewrites the model's weights
    4. DIt costs far more than fine-tuning
  1. 2. What does prompting cost, compared to fine-tuning?

    1. AA recurring vector-database fee
    2. BThe same as fine-tuning, since both update the model
    3. CNothing, but it's bounded by context and the model's existing weights
    4. DNothing, and it has no limitations at all
  2. 3. What can prompting alone never give a model that RAG and fine-tuning both can?

    1. AA vector database
    2. BThe ability to generate text at all
    3. CA context window
    4. DKnowledge the model wasn't trained on
  3. 4. Which of these does fine-tuning do that prompting does not?

    1. ACosts nothing to use
    2. BBakes persistent behavior into the model's weights
    3. CStays bounded by the context window
    4. DRetrieves documents from an external store
  4. 5. Which approach to adding knowledge typically relies on a vector database?

    1. ARAG
    2. BFine-tuning
    3. CPrompting
    4. DAll three equally
  5. 6. In a chat API call, what is the system prompt?

    1. AThe user's latest message in the conversation
    2. BThe documents a RAG pipeline retrieves for the query
    3. CThe model's own reply from the previous turn
    4. DThe developer's instructions that set the model's role, tone and rules
  6. 7. GPT-3's defining discovery was adapting to a task from examples in the prompt alone, with no weight updates. What is this called?

    1. AFine-tuning
    2. BIn-context learning
    3. CChain-of-thought prompting
    4. DRLHF
  7. 8. A developer includes three worked examples in the prompt before asking the model to handle a new case. Which technique is this?

    1. AZero-shot prompting
    2. BFine-tuning
    3. CFew-shot prompting
    4. DChain-of-thought prompting
  8. 9. Multi-step arithmetic and step-by-step reasoning are described as what kind of ability in large models?

    1. AAn emergent ability that appears in large models but not small ones
    2. BAn ability that appears only once retrieval-augmented generation is added
    3. CAn ability that appears only after fine-tuning on labeled data
    4. DA built-in ability of every model, present at every size
  9. 10. Which statement about in-context learning is NOT true?

    1. AIt was GPT-3's defining discovery
    2. BIt requires no weight updates
    3. CThe learning it produces persists after the context window ends
    4. DIt lets a model adapt its behavior from examples or instructions in the prompt
  10. 11. What does in-context learning NOT require, unlike ordinary model training?

    1. AA context window
    2. BA prompt
    3. CA pretrained model to start from
    4. DGradient updates to the model's weights
  11. 12. A user adds "Let's think step by step" to a math question, gives no worked examples, and the model reasons more accurately. Which technique is this?

    1. ARetrieval-augmented generation
    2. BZero-shot chain-of-thought
    3. CFew-shot prompting
    4. DSupervised fine-tuning (SFT)
  12. 13. Which pattern retrieves relevant documents from an external store and inserts them into the prompt as context before the model answers?

    1. AFunction calling
    2. BFine-tuning
    3. CChain-of-thought prompting
    4. DRAG (Retrieval-Augmented Generation)
  13. 14. What does a vector database store, and how does it find the material RAG needs?

    1. AModel weights, found by gradient lookup
    2. BDocument embeddings, found by fast similarity search
    3. CTokenizer vocabularies, found by prefix matching
    4. DChat transcripts, found by exact keyword match only
  14. 15. Why are source documents split into smaller segments before being embedded for RAG?

    1. ASo the vector database can store documents without embeddings
    2. BSo similarity search only has to match exact keywords
    3. CSo retrieved passages fit the context window and stay semantically focused
    4. DSo the model can memorize each segment permanently
  15. 16. Which two supporting components does RAG typically rely on?

    1. AA vector database and chunking
    2. BZero-shot and few-shot prompting
    3. CA system prompt and an agent loop
    4. DFine-tuning and function calling
  16. 17. Chunk size is a key RAG knob. What goes wrong when chunks are too large?

    1. ATheir embeddings can no longer be searched by similarity
    2. BThey waste context
    3. CRetrieved passages become too short to carry meaning
    4. DThey overwrite the model's weights
  17. 18. What makes an LLM agent different from a standard single-prompt interaction?

    1. AIt permanently updates the model's weights after each task
    2. BIt splits documents into chunks before embedding them
    3. CIt runs a planning loop — selecting tools, acting, and observing results until a goal is met
    4. DIt inserts retrieved documents into the prompt before answering
  18. 19. In function calling, what does the model itself actually produce?

    1. AA structured, well-formed call that the host application runs
    2. BAn embedding to be stored in a vector database
    3. CA new set of fine-tuned weights for the task
    4. DThe API's response data, computed directly by the model
  19. 20. Once the host application has executed a tool the model requested, what happens next?

    1. AThe result is chunked and embedded permanently
    2. BThe interaction ends with the raw tool output as the answer
    3. CThe model's weights are updated with the result
    4. DThe result is fed back to the model
  20. 21. An assistant must answer with today's live exchange rate from a currency API. Which mechanism gives it that ability?

    1. AZero-shot prompting
    2. BFine-tuning
    3. CFunction calling
    4. DChunking
  21. 22. Which of these is NOT part of the agent loop?

    1. ARetraining the model's weights after each observation
    2. BSelecting which tool to use
    3. CPlanning toward a goal
    4. DObserving the results of an action
  22. 23. Fine-tuning and RAG differ in where the new knowledge or behavior lands. Which pairing is correct?

    1. ABoth → the prompt context
    2. BFine-tuning → the model's weights; RAG → the prompt context
    3. CBoth → the model's weights
    4. DFine-tuning → the prompt context; RAG → the model's weights
  23. 24. How would you characterize fine-tuning's cost and staying power?

    1. AFree, but bounded by context size
    2. BFree, and permanent
    3. CCheap, and temporary within one conversation
    4. DExpensive, but durable across calls
  24. 25. A team needs a bot to answer questions over internal documents that are rewritten every week. Which approach fits best?

    1. AZero-shot prompting
    2. BRAG
    3. CFine-tuning
    4. DChain-of-thought prompting
  25. 26. A company wants a house style and domain behavior the model keeps on every call, without adding instructions or examples to each prompt. Which approach fits best?

    1. AFine-tuning
    2. BFunction calling
    3. CFew-shot prompting
    4. DRAG
  26. 27. Which statement about RAG is NOT true?

    1. AIt inserts retrieved text into the prompt as context before the model answers
    2. BIt suits fresh or proprietary knowledge
    3. CIt retrieves documents from an external store
    4. DIt permanently changes the model's weights

Go deeper

Answer key

  1. 1. A It is limited to knowledge the model already hasPrompting costs nothing but is bounded by context and the model's existing weights, so it can't supply knowledge the model never learned. That gap is exactly why RAG, function calling, and agents exist.
  2. 2. C Nothing, but it's bounded by context and the model's existing weightsPrompting costs nothing but is bounded by context and existing weights — that tradeoff is exactly why RAG and fine-tuning exist.
  3. 3. D Knowledge the model wasn't trained onPrompting alone can't give a model knowledge it wasn't trained on — that gap is exactly why RAG and fine-tuning exist.
  4. 4. B Bakes persistent behavior into the model's weightsFine-tuning bakes persistent behavior into the weights — expensive but durable across calls — unlike prompting, which is free but limited by context.
  5. 5. A RAGRAG typically relies on a vector database that stores document embeddings and runs fast similarity search; prompting and fine-tuning don't need one.
  6. 6. D The developer's instructions that set the model's role, tone and rulesThe system prompt is the block of instructions a developer supplies ahead of the conversation to set the model's role, tone and rules. The user's requests arrive separately as user messages.
  7. 7. B In-context learningGPT-3 is credited with in-context learning: performing a task competitively from examples in the prompt alone, with no gradient updates.
  8. 8. C Few-shot promptingFew-shot prompting puts a few worked examples in the prompt, teaching the task inside the context window with no weight updates. Zero-shot gives no examples, chain-of-thought asks for step-by-step reasoning, and fine-tuning changes weights.
  9. 9. A An emergent ability that appears in large models but not small onesMulti-step arithmetic and step-by-step reasoning are called emergent abilities, present in large models and absent in smaller ones. Chain-of-thought prompting can draw that reasoning out, but Wei et al. (2022) found it appears in sufficiently large models.
  10. 10. C The learning it produces persists after the context window endsNo learning persists once the context window ends: during a chat the weights never change, and any apparent memory is just text in the current context window. The other three are true of GPT-3's in-context learning.
  11. 11. D Gradient updates to the model's weightsBy definition, in-context learning is task performance from prompt examples alone, with no gradient updates — contrasting with how models are normally trained.
  12. 12. B Zero-shot chain-of-thoughtKojima et al. (2022) showed that adding "Let's think step by step", with no examples, improves reasoning: zero-shot chain-of-thought. Few-shot prompting needs worked examples, SFT changes weights, and RAG retrieves documents.
  13. 13. D RAG (Retrieval-Augmented Generation)RAG, introduced by Lewis et al. (2020), retrieves documents and inserts them as context. Fine-tuning changes weights instead, and function calling emits a tool call rather than retrieving documents.
  14. 14. B Document embeddings, found by fast similarity searchA vector database stores document embeddings and runs fast similarity search to return the most semantically relevant chunks. Semantic similarity — not exact keyword matching — is the point.
  15. 15. C So retrieved passages fit the context window and stay semantically focusedChunking keeps retrieved text small enough for the context window and semantically focused. Retrieval does not make the model memorize anything, a vector database stores embeddings rather than raw documents, and its similarity search matches by meaning, not exact keywords.
  16. 16. A A vector database and chunkingRAG typically relies on a vector database (embeddings + similarity search) and on chunking (splitting documents so retrieved text fits the context window).
  17. 17. B They waste contextChunks that are too large waste context window space; chunks that are too small lose coherence, which is why chunk size is a key knob. Chunk size never breaks similarity search or touches the weights.
  18. 18. C It runs a planning loop — selecting tools, acting, and observing results until a goal is metAn agent wraps tool use in a planning loop: the model acts as a controller that plans, picks tools, acts, and observes until the goal is met. Weight updates are fine-tuning, inserting retrieved docs is RAG, and splitting docs is chunking.
  19. 19. A A structured, well-formed call that the host application runsThe model emits a structured call; the host application executes it against the real API or tool. The model does not run the tool or compute the result itself.
  20. 20. D The result is fed back to the modelThe host runs the call and feeds the result back so the model can use it — that round trip is what grounds the model in real-time data. No weight update or embedding step is involved.
  21. 21. C Function callingFunction calling lets the model emit a structured call to an external API, grounding it in real-time data. Fine-tuning bakes in static behavior, chunking is a retrieval step, and zero-shot is just a bare instruction.
  22. 22. A Retraining the model's weights after each observationThe loop is plan, select tools, act, observe — repeated until the goal is met. Nothing in it retrains the weights; that would be fine-tuning.
  23. 23. B Fine-tuning → the model's weights; RAG → the prompt contextFine-tuning bakes behavior into the weights; RAG inserts retrieved documents into the context before the model answers. The two operate at different layers, which is the whole contrast.
  24. 24. D Expensive, but durable across callsFine-tuning is expensive but persists across calls because it changes the weights. 'Free but bounded by context' describes prompting, not fine-tuning.
  25. 25. B RAGRAG is ideal for fresh or proprietary knowledge: retrieve the current documents and insert them as context. Fine-tuning would have to be redone as the documents change, and prompting alone adds no new knowledge.
  26. 26. A Fine-tuningFine-tuning bakes persistent style and behavior into the weights, so it holds across calls. Few-shot would work but must be re-supplied in every prompt, and RAG adds knowledge rather than behavior.
  27. 27. D It permanently changes the model's weightsRAG never touches the weights — that is fine-tuning's job. RAG works entirely by retrieving external documents and placing them into the context.

Sources

Image credits

  • Vector databases, explained: FlashFeed original diagram · CC-BY-4.0
  • Why RAG chunks documents: FlashFeed original diagram · CC-BY-4.0
  • The agent planning loop: FlashFeed original diagram · CC-BY-4.0

Download all 10 lessons: PDF · EPUB