Skip to main content

RAG: giving the model an open book

BeginnerLesson 8 of 167 min readReviewed by Mayank Raj Parmar on

In short

RAG, short for retrieval-augmented generation, lets an AI answer from documents it was never trained on. Before the model answers, a search finds the most relevant passages and adds them to the prompt, so the answer is based on them and can point to its sources.

Ask a chatbot "What time do the gates of my college hostel close on Saturdays?" and it can't really know. It has never read your hostel's rulebook. It might say so, or it might guess a time that sounds right, as you saw in lesson 6. Now paste the right page of the rulebook into the chat and ask again. This time it reads the page and gets the answer right.

That's the whole idea behind RAG: before the model answers, find the right pages and hand them over. It's how chatbots answer from a company's help pages, a bank's policies or your own PDFs. In this lesson you'll see how RAG works step by step, what the model actually receives, why it's often better than retraining the model, and what can still go wrong.

What RAG means

RAG stands for retrieval-augmented generation. Each word is one step:

  • Retrieval: finding the passages most likely to hold the answer, usually with the semantic search from lesson 7.
  • Augmented: adding those passages to the prompt, the text the model receives.
  • Generation: the LLM writes its answer, now with the right information in front of it.

The name comes from a 2020 paper by researchers at Facebook AI Research (now part of Meta), University College London and New York University. Their system searched Wikipedia, cut into 21 million passages of 100 words each, and wrote answers from what it found. They reported that its answers were more specific and more factual than those of a similar model working from memory alone.

How RAG works, step by step

RAG has two halves. The documents are prepared once, ahead of time. Then every question goes through three steps, which give RAG its name:

Once, ahead of time

  1. Split. Cut every document into chunks, short passages a few hundred tokens long.
  2. Embed. Turn each chunk into an embedding, a list of numbers for its meaning.
  3. Store. Save the chunks and their embeddings in a vector database.

You get: A library of your documents that can be searched by meaning.

Every time someone asks

  1. Retrieve. Turn the question into an embedding and find the closest chunks.
  2. Augment. Add those chunks to the prompt, with instructions to answer only from them.
  3. Generate. The LLM writes the answer from the chunks and points to the ones it used.

You get: An answer based on your documents, with the passages you can check.

Figure 1: the two halves of RAG. The documents are prepared once; retrieve, augment and generate happen every time someone asks a question.

A chunk is a short passage cut from a longer document. Chunks keep the search precise and the prompt short: a whole rulebook would match almost any question, while the one paragraph about gate timings matches only the right ones. OpenAI's file search, for example, cuts files into chunks of up to 800 tokens by default, which is about 600 words, or two pages. Each chunk overlaps the one before it by 400 tokens, so an idea that gets cut off at the end of one chunk still appears whole in the next.

The retrieve step is the semantic search from lesson 7: the question becomes an embedding, and the chunks with the closest embeddings win. So a good system can find the gate-timings rule even when a student asks "Can I come back late on a Saturday?" without using the word "gate".

What the model actually sees

Here's a made-up example. A college puts its 40-page hostel rulebook into a RAG system, and a student asks: "Can I come back at 10:30 pm on a Saturday?" The search finds two chunks, and the app sends the model a prompt like this:

text
Answer the question using only the passages below.
If the answer isn't in them, say you don't know.
Mention the section you used.

[1] Section 4.2: Hostel gates close at 10:00 pm on weekdays
and at 11:00 pm on Saturdays and Sundays.

[2] Section 4.3: Students who return after the gates close
must sign the late-entry register at the security desk.

Question: Can I come back at 10:30 pm on a Saturday?

The model replies: "Yes. On Saturdays the gates close at 11:00 pm (Section 4.2), so at 10:30 pm you're back before closing time and don't need to sign the late-entry register (Section 4.3)."

Nothing about the model changed. It wasn't retrained on the rulebook. It was handed the right two paragraphs and told to stick to them, and the section numbers let the student check the answer.

Why not retrain the model?

You could train a model further on your own documents, which is called fine-tuning. For answering questions from documents, RAG has three big advantages:

  • It works with documents the model never saw. Your hostel rules, your company's leave policy and yesterday's circular were never in the model's training text. RAG brings them in at the moment they're needed.
  • Updating is easy. Change a document and the next answer uses the new version. In the 2020 paper, the team swapped their December 2018 copy of Wikipedia for one from December 2016, without retraining anything, and asked about 82 world leaders who had changed between those dates. With the Wikipedia from the right year, the system named about 70% of them correctly. With the wrong year's Wikipedia, it managed 12% or fewer. The answers followed the documents.
  • You can check the answer. Because the answer comes from specific chunks, the app can show which ones it used, like the section numbers above.

Why not paste in everything?

Lesson 5 showed that context windows are now huge, so why search at all? For a small collection, you often don't need to. Anthropic suggests that if all your material fits in under 200,000 tokens, about 500 pages, you can simply put all of it in the prompt.

But a bank's help centre or a library of court judgements is far bigger than any context window. And every token you send costs money and time (lesson 5), with every single question. Sending only the few chunks that matter is cheaper and faster.

What can still go wrong

RAG makes answers better grounded in real documents, but it doesn't make them guaranteed. Things go wrong in three places:

  1. The search misses. If the right chunk isn't found, the model can't use it, and it may guess instead. Chunks can also lose their meaning when they're cut out of a document. Anthropic's example is a chunk that says only "The company's revenue grew by 3% over the previous quarter", without saying which company or which quarter. Anthropic found that adding a line of context to each chunk before storing it, and searching by both meaning and exact keywords, cut the times the search missed the right chunk by 49%. Adding reranking, a second, more careful pass that re-sorts the chunks the search found, took that to 67%.
  2. The model still slips. It can misread a chunk, mix two together or add things that aren't there. In 2024, Stanford researchers tested legal research tools built on RAG, some of them advertised as "hallucination-free". On more than 200 legal questions, two of the tools gave incorrect information more than 17% of the time, and a third made things up more than 34% of the time.
  3. The documents are wrong or out of date. RAG faithfully repeats what's in its library. If the rulebook is last year's, so is the answer.

Key takeaways

  • RAG, retrieval-augmented generation, finds the passages most likely to hold the answer, adds them to the prompt, then lets the LLM write the answer from them.
  • Documents are cut into chunks, turned into embeddings and stored once. Each question is matched against them with semantic search.
  • The model isn't retrained. It's handed the right text for each question, so updating a document updates the answers.
  • If everything fits in the context window, you can paste it all in. RAG matters when there's too much, or to keep each question cheap and fast.
  • RAG reduces made-up answers but doesn't remove them: the search can miss, the model can slip, and documents can be wrong. Check the sources.

Check your understanding

Question 1 of 8
What does RAG do before the LLM writes its answer?

Sources and further reading

Report an error in this lesson