System Atlas
the atlas
AI · AI & LLMs7 minApplied

What is Retrieval-Augmented Generation (RAG)

Ground an LLM in your own data so it stops making things up.

the gist

RAG bolts a retrieval step onto generation. Offline, documents are chunked, embedded and stored in a vector database. Online, the user's query is embedded, a similarity search pulls the top-k relevant chunks, and those chunks plus the query go into the prompt so the LLM answers from real, citable sources.

ANALOGY // An open-book exam: instead of answering from memory, you first flip to the most relevant pages, then write your answer using what's actually on them.
living diagram01 / 05
indexUser queryEmbed queryto vectorVector DBsimilarity searchAugment promptchunks + queryLLMgenerateGrounded answerwith citationsDocumentschunk → embed
01

Index your documents (offline)

Ahead of time, split your source documents into chunks of a few hundred tokens each, then run every chunk through an embedding model to turn it into a vector. Those vectors, paired with the original text, are stored in a vector database built for fast similarity search. This indexing runs once (and re-runs when content changes), so nothing here is on the critical path of a live request.

02

Embed the query

At request time, the user's question is run through the exact same embedding model used to index the chunks. Using the same model is essential — it places the query in the identical vector space, so 'closeness' between query and chunk is meaningful. The output is a single query vector.

03

Retrieve top-k chunks

The vector database performs a similarity search, comparing the query vector against the stored chunk vectors (usually by cosine similarity) and returning the k nearest. At scale this uses approximate nearest-neighbor search to stay fast without checking every vector. The result is the handful of chunks most semantically relevant to the question.

04

Augment the prompt

The retrieved chunks are stitched into the prompt alongside the original question, typically under an instruction like 'answer using only the context below'. This is the 'augmented' in Retrieval-Augmented Generation — the model is handed the exact source text it should reason over. Good systems also pass along identifiers so the answer can cite where each fact came from.

05

Generate the answer

The LLM reads the augmented prompt and produces a response grounded in the supplied chunks rather than its parametric memory. Because it was given real source text, it can quote and cite specifics, and a well-instructed model will say it doesn't know when the context lacks the answer. The weights never changed — only what went into the prompt did.

the full picture
01

Why a raw LLM isn't enough

A large language model is a next-token predictor whose knowledge is baked into its weights during training. That design has two consequences that matter enormously in practice, and RAG exists to fix both.

The first is stale knowledge. A model's parameters are frozen at the moment training stops. It has never seen your company's internal wiki, last week's news, the ticket a customer filed this morning, or any document created after its training cutoff. Ask it about those and it is guessing from general patterns, not facts.

The second, and more dangerous, is hallucination. Because the model is optimized to produce fluent, plausible-sounding text, it will happily invent a confident answer when it doesn't actually know — a fabricated citation, a made-up API method, a wrong date delivered in the same authoritative tone as a correct one. There is no built-in mechanism that says "I'm unsure." For a demo this is a curiosity; for anything where correctness matters — legal, medical, financial, support over real product docs — it is disqualifying.

You could try to fix this by retraining the model on your data, but that is expensive, slow, has to be redone every time the data changes, and still bakes knowledge into weights where it can't be audited or cited. Retrieval-Augmented Generation takes a different route: leave the model's weights alone and, at the moment of each question, fetch the relevant real text and put it in front of the model. The core idea is a shift from "answer from memory" to "look it up, then answer" — an open-book exam instead of a closed one.

02

The offline half: indexing your knowledge

RAG has two phases that run at completely different times. The first, indexing, happens ahead of time and builds the searchable knowledge store. It has three steps.

Chunk. Whole documents are too big to retrieve and stuff into a prompt wholesale, and they mix many topics. So each document is split into smaller passages — often a few hundred tokens — sometimes along natural boundaries like paragraphs or headings, usually with a little overlap between neighbors so a sentence split across a boundary isn't lost. Each chunk becomes an independently retrievable unit.

Embed. Every chunk is run through an embedding model, a neural network that maps text to a fixed-length vector (say 768 or 1536 numbers) capturing its meaning. The crucial property is that passages about similar topics land near each other in this vector space, even when they share no exact words — "how do I reset my password" and "steps to recover account access" end up close. This is what lets retrieval work on meaning rather than keyword matching.

Store. The vectors, each paired with its original text and metadata, go into a vector database — Pinecone, Weaviate, pgvector, FAISS, and others. Beyond simple storage, these systems build an index (typically an approximate-nearest-neighbor structure like HNSW) so that finding the closest vectors among millions stays fast. Indexing is a batch job you run once and refresh whenever the underlying documents change, so none of this cost lands on a live user request.

03

The online half: retrieve, augment, generate

The second phase runs on every user request, and it is a tight three-step pipeline layered on top of a normal LLM call.

Embed the query. The incoming question is passed through the *same* embedding model used during indexing. This matters: only if the query and the chunks live in the same vector space is their distance meaningful. The result is a single query vector representing what the user is asking about.

Retrieve (similarity search). The vector database compares the query vector against the stored chunk vectors — usually via cosine similarity, which measures the angle between vectors — and returns the top-k most similar chunks (k is often 3 to 10). At scale it uses approximate nearest-neighbor search, trading a sliver of accuracy for a massive speed-up, so retrieval stays in the low milliseconds even over huge corpora.

Augment and generate. The retrieved chunks are inserted into the prompt alongside the question, wrapped in an instruction such as "Using only the context below, answer the question; if the answer isn't present, say so." The LLM then generates its response grounded in that supplied text. Because the model is reading real source material rather than recalling from its weights, the answer can be accurate, current, and — when chunk identifiers are passed through — footnoted with citations the user can verify. Nothing about the model was retrained; the "learning" happened entirely by controlling what went into the context window.

04

Where RAG gets hard: chunking and retrieval quality

RAG's failure mode has a blunt name: garbage in, garbage out. The generation step can only be as good as the chunks retrieved, so most of the engineering effort goes into retrieval quality, and the trade-offs are real.

Chunk size is a genuine dilemma. Small chunks are precise — a retrieved passage is tightly on-topic — but they can sever context, returning a sentence that only makes sense with the paragraph around it. Large chunks preserve context but dilute the embedding (one vector now averages several ideas) and burn more of the prompt budget, leaving less room and adding cost and latency. There is no universally correct size; it is tuned per corpus, and overlap between adjacent chunks is a common hedge.

Retrieval can simply miss. If the right chunk isn't in the top-k, the model never sees it and will either say it doesn't know or, worse, fall back to hallucinating. Semantic similarity also isn't the same as relevance — a chunk can be topically close yet unhelpful. Common remedies include retrieving more candidates and then re-ranking them with a more precise (and more expensive) model, hybrid search that blends vector similarity with old-fashioned keyword matching to catch exact terms like product codes, and query rewriting to sharpen vague questions before searching.

The context window is finite. You cannot retrieve everything, so k is a balance: too few chunks and you miss the answer, too many and you drown the real signal in noise (models are demonstrably worse at using information buried in the middle of a long context) while paying for every token. And retrieval adds a network hop plus an extra embedding call to every request — RAG trades a little latency and system complexity for grounding. Getting these knobs right is the difference between a demo and a dependable system.

05

When to use RAG, and where it fits

RAG has become the default architecture for putting LLMs to work on real, proprietary, or changing information, and knowing when it's the right tool is half the skill.

Reach for RAG when the model needs knowledge it doesn't have or that keeps changing: internal documentation and wikis, customer-support knowledge bases, product catalogs, legal and policy documents, or anything where answers must be current and, ideally, citable. Its standout advantages are that updating knowledge is as cheap as re-indexing a document (no retraining), answers can point to their sources so humans can verify them, and access can be scoped so the model only ever sees documents a given user is allowed to read.

It is *not* the right tool for everything. RAG changes what the model *knows*, not how it *behaves*. If you need the model to adopt a particular tone, follow a rigid output format, or master a specialized skill or style, that is a job for fine-tuning or LoRA — see the companion concept on fine-tuning vs RAG. The two are complementary and frequently combined in production: a lightly fine-tuned model that has the right voice and formatting, paired with RAG that feeds it current facts.

Architecturally, RAG is also a foundational building block for agents. An agent that can call a "search the knowledge base" tool is essentially performing retrieval on demand as one action among many in its reasoning loop. So RAG is worth understanding not just as a technique for grounded question-answering, but as the pattern by which language models are connected to the specific, living information of the world they operate in.

what to remember
01

RAG grounds an LLM in external data, sharply reducing hallucination.

02

Offline (indexing): documents are chunked, each chunk is embedded, and the vectors are stored in a vector database.

03

Online: the query is embedded and a similarity search retrieves the top-k most relevant chunks.

04

Those retrieved chunks are injected into the prompt alongside the query — this is the 'augment' step.

05

The LLM generates a grounded answer that can cite the sources it was given.

check yourself

Quick quiz

Q1

What is RAG's main benefit over a bare LLM?

Q2

What happens during the offline indexing phase?

follow the thread