Generative AI & LLM Apps
Topic 7 · Building LLM apps · ⏱ about 4 h

Retrieval-Augmented Generation (RAG)

Make a model answer from your own documents: split them into chunks, embed and index them, retrieve the best chunks for each question, put them in the prompt, and make the answer cite its sources. Plus hybrid search, reranking, and how RAG fails.

Show:
In this topic
Beginner

Why RAG?

An LLM knows only what was in its training data, up to its cutoff date, and it does not know your clinic's policies, your company's wiki or last week's meeting notes. You could paste every document into every prompt, but that is slow, expensive and limited by the context window.

Retrieval-augmented generation (Lewis et al., 2020) solves this: for each question, retrieve the few most relevant passages from your documents and add them to the prompt, with an instruction to answer only from them and cite them. It is the most common pattern in real LLM apps, because it is cheap, keeps answers grounded and up to date (change a document and the next answer changes), and shows users where answers come from.

Beginner

The pipeline

Once, ahead of time (indexing):

  1. Load documents (PDF, web pages, Word files, database rows) and clean the text.
  2. Chunk: split them into passages. A common starting point is a few hundred tokens per chunk with some overlap between neighbours, splitting on headings and paragraphs rather than mid-sentence. Keep each chunk's source and title.
  3. Embed every chunk and store the vectors with the text and metadata (embeddings).

For every question (querying):

  1. Retrieve the top k chunks (say 3 to 8) most similar to the question.
  2. Augment: build a prompt with the chunks, each labelled with an id, plus the question and the rules ("Answer only from the sources. Cite them like [2]. If they do not contain the answer, say so.").
  3. Generate the answer, and show the cited sources to the user.
Try it

Try it: a tiny RAG system

The knowledge base is eight short passages from a made-up clinic handbook. Ask a question (or pick one), change k, and see which passages are retrieved and the exact prompt that would go to the model. Retrieval here uses keyword scoring (BM25), which runs instantly in the browser.

Interactive: this part runs in your browser.

Beginner

Making retrieval better

  • Hybrid search: combine keyword search (BM25, great for names, codes and rare words like "Medicare") with vector search (great for paraphrases), and merge the two rankings.
  • Reranking: retrieve a generous 20 to 50 candidates fast, then reorder them with a slower, more accurate cross-encoder reranker that reads the question and each chunk together, and keep the best few.
  • Metadata filters: only search documents the user is allowed to see, the right product, or recent dates. Access control must happen in retrieval, never by asking the model to keep secrets.
  • Query rewriting: turn a vague follow-up ("what about for kids?") into a standalone question using the chat history before searching.
  • Chunking matters: chunks too small lose context; too big dilute the match and waste tokens. Test a few sizes on your own questions.
Beginner

How RAG fails, and how to tell which part failed

SymptomLikely causeFix
Answer is wrong, and the right passage was not retrievedretrievalhybrid search, better chunking, reranking, query rewriting
The right passage was retrieved but the answer ignores or contradicts itgenerationclearer instructions, fewer and better chunks, a stronger model
Answer includes facts that are in no sourcehallucinationrequire citations, check that each claim is supported, allow "I don't know"
Answers are out of datestale indexre-index changed documents automatically

Always log the retrieved chunks with each answer: it is the only way to tell retrieval failures from generation failures. The evaluation topic shows how to measure both.

Beginner

Code: a minimal RAG pipeline

Chunking, retrieval and prompt building run here in the page. The final model call is left as a function you plug your provider into.

Python
import numpy as np, re

handbook = """# Appointments
You can book online, by phone or at reception. Standard appointments are 15 minutes; long appointments are 30 minutes.
# Cancellations
Please cancel at least 24 hours ahead. Late cancellations and missed appointments may be charged a $40 fee.
# Billing
Children under 16 and concession card holders are bulk-billed. Other patients pay at the time of the visit.
# Opening hours
Weekdays 8am to 6pm. Saturdays 9am to 12pm. Closed on Sundays and public holidays."""

# 1. chunk on headings, keeping the title with each chunk
chunks = []
for block in handbook.split("# ")[1:]:
    title, body = block.split("\n", 1)
    chunks.append({"id": len(chunks) + 1, "title": title.strip(), "text": body.strip()})

# 2. embed (stand-in: word counts; swap in a real embedding model)
tok = lambda t: re.findall(r"[a-z0-9]+", t.lower())
vocab = sorted({w for c in chunks for w in tok(c["title"] + " " + c["text"])})
def embed(t):
    v = np.array([tok(t).count(w) for w in vocab], float); n = np.linalg.norm(v)
    return v / n if n else v
M = np.stack([embed(c["title"] + " " + c["text"]) for c in chunks])

# 3. retrieve and 4. build the prompt
def build_prompt(question, k=2):
    best = np.argsort(-(M @ embed(question)))[:k]
    sources = "\n".join(f'<source id="{chunks[i]["id"]}" title="{chunks[i]["title"]}">{chunks[i]["text"]}</source>' for i in best)
    return f"""Answer the question using only the sources below.
Cite sources like [1]. If the sources do not contain the answer, say "I don't know".

{sources}

Question: {question}"""

print(build_prompt("Is there a fee if I cancel late?"))
# 5. answer = call_your_model(build_prompt(...))   <- your API call from the previous topic
Open in playground ↗
Check yourself

Quiz

1. What is the main purpose of RAG?

2. Users search for a product code like "XR-4471" and vector search misses it. What helps most?

3. The correct passage was retrieved, but the answer still contradicts it. Where is the failure?

4. Some documents are confidential to managers. How should a RAG app protect them?

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready