# Retrieval-augmented generation (RAG): giving models your data
Ask ChatGPT "How many vacation days do I get after three years here?" and it has no idea. It never read your company handbook. It might confidently invent an answer, which is worse than saying nothing.
RAG fixes this. It's the technique behind almost every "chat with your documents" product you've seen: customer support bots, internal HR assistants, legal research tools. By the end of this lesson you'll understand exactly how it works, and you'll see the core of it in about 20 lines of Python.
A large language modellarge language modelA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.View full definition → (LLMLLM) like GPT-5 or Claude knows what was in its training data. It does
You could try pasting your entire 200-page handbook into the chat. But models have a context windowcontext windowThe context window is the maximum amount of text (measured in tokens) a language model can process at once, including both the input prompt and the generated output.View full definition →: a limit on how much text they can read at once. Even with today's large windows, dumping everything in is slow, expensive, and makes the model less accurate (it gets distracted by irrelevant text).
RAG's insight: don't give the model everything. Find the few relevant paragraphs and give it only those.
Think of RAG as a smart librarian. You ask a question, the librarian runs to find the three most relevant pages, and hands them to the model along with your question.
1. Retrieval: Search your documents for the chunks most relevant to the question.
2. Augmentation: Paste those chunks into the prompt.
3. Generation: The model answers using that pasted text.
The magic is in step 1. How does a computer know which paragraphs are "relevant"? This is where embeddings come in.
An embeddingembeddingAn embedding is a numerical vector that represents data (text, images, or items) in a way that captures meaning, so similar items sit close together in space.View full definition → is a list of numbers that represents the *meaning* of a piece of text. Similar meanings get similar numbers.
"How much paid leave do I get?" and "What is the vacation policy?" use almost no words in common. But their embeddings land close together, because they mean nearly the same thing. That's the key: embeddings let you search by meaning, not by keyword.
You create embeddings by calling an embeddingembeddingAn embedding is a numerical vector that represents data (text, images, or items) in a way that captures meaning, so similar items sit close together in space.View full definition → model (OpenAI, Google, and Cohere all offer them cheaply). You get back a long list of numbers, often 1,536 of them, for each chunk of text.
Want a visual intuition for this? Google's embeddingembeddingAn embedding is a numerical vector that represents data (text, images, or items) in a way that captures meaning, so similar items sit close together in space.View full definition → projector lets you fly through real word embeddings in 3D: projector.tensorflow.org.
Let's build it conceptually, then in code.
You can't embed a whole 200-page document as one blob. You split it into chunks, usually a few paragraphs each (say 500 words). Each chunk becomes one searchable unit.
So our handbook becomes maybe 400 chunks: one about parental leave, one about expense reports, one about vacation accrual, and so on.
You run each chunk through the embeddingembeddingAn embedding is a numerical vector that represents data (text, images, or items) in a way that captures meaning, so similar items sit close together in space.View full definition → model and store the resulting numbers in a vector databasevector databaseA vector database stores data as high-dimensional numeric vectors (embeddings) and retrieves items by similarity rather than exact matches, powering semantic search and AI applications.View full definition → (a database built to search by similarity). Popular ones in 2026 include Chroma, Pinecone, and pgvector. For small projects you don't even need one: a plain list in memory works.
This step happens once, ahead of time. It's called "indexing."
When a user asks something:
1. Embed their question.
2. Find the chunks whose embeddings are closest to the question's embeddingembeddingAn embedding is a numerical vector that represents data (text, images, or items) in a way that captures meaning, so similar items sit close together in space.View full definition →.
3. Paste those chunks into the prompt.
4. Send it to the model.
Here's the entire pattern in runnable Python, using OpenAI. Don't worry if you're not a coder; read the comments and you'll follow the logic.
from openai import OpenAI
import numpy as np
client = OpenAI() # uses your API key
# 1. Our "handbook," split into chunks
chunks = [
"Employees accrue 15 vacation days per year. After 3 years, this increases to 20 days.",
"Expense reports must be submitted within 30 days using the Concur portal.",
"Parental leave is 16 weeks of paid time off for all new parents.",
]
def embed(text):
# Turn text into a list of numbers (its embedding)
resp = client.embeddings.create(
model="text-embedding-3-small", input=text
)
return np.array(resp.data[0].embedding)
# 2. Embed every chunk once (the "index")
chunk_vectors = [embed(c) for c in chunks]
# 3. A user asks a question
question = "How many vacation days do I get after three years?"
q_vector = embed(question)
# 4. Find the most similar chunk (cosine similarity)
def similarity(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
scores = [similarity(q_vector, cv) for cv in chunk_vectors]
best_chunk = chunks[int(np.argmax(scores))]
# 5. Stuff the best chunk into the prompt
prompt = f"""Answer using ONLY the context below.
If the answer isn't there, say you don't know.
Context: {best_chunk}
Question: {question}"""
answer = client.chat.completions.create(
model="gpt-5",
messages=[{"role": "user", "content": prompt}],
)
print(answer.choices[0].message.content)What happens: the question's embeddingembeddingAn embedding is a numerical vector that represents data (text, images, or items) in a way that captures meaning, so similar items sit close together in space.View full definition → is closest to the *vacation* chunk (not expenses, not parental leave). We paste only that chunk in, and the model answers "20 days" straight from your data.
Notice the prompt instruction: "Answer using ONLY the context below." This is what keeps the model honest. It stops it from inventing answers and grounds it in your text.
Knowledge check
1. According to the lesson, what is the core insight that makes RAG effective?
2. In the librarian analogy, which of the three RAG steps corresponds to the librarian handing the found pages to the model alongside your question?
3. The lesson mentions a 'context window.' What does this term refer to?
4. Select ALL correct answers about what a base LLM like GPT-5 or Claude does NOT inherently know.
Sélectionnez toutes les réponses correctes.
5. Select ALL correct answers about embeddings as described in the lesson.
Sélectionnez toutes les réponses correctes.
People often ask: why not just fine-tune the model on my data instead? (Fine-tuningFine-tuningFine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a smaller, targeted dataset, improving accuracy and style for that use case.View full definition → means retraining the model so your data lives inside it.)
RAG usually wins for company knowledge because:
Fine-tuningFine-tuningFine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a smaller, targeted dataset, improving accuracy and style for that use case.View full definition → is better for changing the model's *style or behavior* (always answer in legal language, always use our brand voice). RAG is better for giving it *facts*. Many real systems use both.
RAG isn't magic. The most common failures:
Bad chunking. If you split a sentence in half, retrieval breaks. Fix: chunk by paragraph or section, and let chunks overlap slightly so context isn't lost at the edges.
Retrieving the wrong chunks. If the question is vague, you might pull irrelevant text. Fix: retrieve the top 3 to 5 chunks instead of just one, giving the model more to work with.
The answer spans many chunks. "Compare our 2024 and 2025 travel policies" needs two sections. Fix: retrieve more chunks and let the model synthesize.
Stale index. You updated the handbook but forgot to re-embed. The bot gives old answers. Fix: re-index on a schedule.
In 2026, you can get RAG without writing the code above:
Use these for small or personal use. Build the pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → yourself when you need control, scale, private hosting, or integration into a product.