RAG: When LLMs Read External Data

The Problem

An LLM's trained parameters are frozen between model updates. Without an external data source, ask it about your company's refund policy, a document you uploaded last week, or today's stock price and it lacks that current or private information. It may say "I don't know" or confidently make something up (this is called hallucination).

Fine-tuning (retraining the model on your specific data) is expensive, slow, and still doesn't help with data that changes daily. There had to be a simpler way.

The Solution: RAG

RAG stands for Retrieval-Augmented Generation. Despite the fancy name, here's all it does:

1. User asks a question
2. Search your documents for chunks relevant to that question
3. Paste those chunks into the prompt alongside the question
4. Let the model answer using the provided context

At a high level, RAG is "retrieve + provide context + generate."

The retrieval layer may search a static index, a live database, or even the web. The application finds relevant material, places it in the context window, and the model reads it like any other text; the base model does not perform that retrieval by itself.

Here's what the context window actually looks like after retrieval - notice how the retrieved documents are just more text alongside everything else:

The Open-Book Exam Analogy

Think of RAG like an open-book exam.

The student (the LLM) doesn't memorize every textbook. Instead, before each question, you hand them the most relevant pages. They read the provided material and synthesize an answer.

The student didn't "know" the answer. They read it from the material you gave them. If you give them the wrong pages, they'll give a wrong answer. If you give them pages with misleading information, they'll be misled.

But How Do You Search?

OK, so RAG needs to find the right documents. But how? You've got thousands of pages. A user asks "How do I get my money back?" - how do you find the relevant chunk?

Plain keyword matching often misses it. The refund policy might say "reimbursement procedure" or "return process" - none of those words match "get my money back." If you're just Ctrl+F-ing for keywords, you'll miss it.

This is where embeddings come in.

An embedding converts text into a list of numbers whose learned geometry often places semantically related text close together. It is a useful similarity signal, not a perfect representation of "meaning."

So the system does this:

1. All your documents get split into small chunks (paragraphs, sections, etc.)
2. Each chunk gets converted into an embedding - a numerical fingerprint of its meaning
3. When the user asks a question, that question gets converted into an embedding too
4. The system finds the chunks whose embeddings are closest to the question's embedding

"How do I get my money back?" and a chunk titled "Reimbursement Procedures" have similar embeddings - because they're about the same thing. The search finds it, even though they share zero words.

What RAG Doesn't Do

Common misconceptions:

• RAG does not itself give the model internet access. A retrieval component can use stored documents or live sources, but the model reads only the material that component supplies.
• RAG doesn't make the model smarter. It gives the model more relevant context, but the model can still misinterpret or hallucinate.
• RAG doesn't guarantee accuracy. If the retriever finds the wrong chunks, the model may answer incorrectly, sometimes with unwarranted confidence.

RAG lets models read external data. But what if you want the model to take actions - send an email, look up a flight, or query a database in real time? That's where tools and function calling come in - and where text manipulation starts having real-world consequences.

Sources

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks - the foundational RAG paper
• Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection - security risks when retrieved content contains instructions