A model's parametric knowledge is whatever it learned during training. It is frozen, incomplete, and out of date the moment training ends. Retrieval-augmented generation fixes this by giving the model access to a separate, queryable corpus at inference time.
The pipeline
- Ingest: split documents into chunks of 500-1500 tokens, ideally with overlap
- Embed: run each chunk through an embedding model — get a vector
- Store: put vectors in a vector database (Pinecone, Weaviate, Qdrant, pgvector, Chroma)
- Retrieve: at query time, embed the query, find the k nearest-neighbour chunks by cosine similarity
- Generate: include those chunks in the LLM prompt as context, ask the question, get a grounded answer
Chunking strategy
Cut documents at semantic boundaries (paragraphs, headings) where possible. Add overlap (100-200 tokens between adjacent chunks) so that an idea spanning a chunk boundary isn't split. For complex documents (legal, financial), domain-aware chunking (preserving sections, tables) beats generic chunking by a lot.
Embedding models
OpenAI text-embedding-3-large, Cohere embed-multilingual-v3, BGE / NV-Embed (open source), Gemini text-embedding-004 are all production-grade. Dimensions typically 768-3072. Costs are low (cents per million tokens). Test on your own data; benchmarks lie.
Retrieval evaluation
The most common RAG failure is retrieval: the relevant chunk isn't in the top-k. Diagnose this with retrieval recall: given a question whose answer is in chunk X, does the retriever return chunk X in the top 10? If recall is poor, no amount of clever prompting saves you.
RAG vs fine-tuning
When to RAG, when to fine-tune
RAG when the knowledge changes (news, policy documents, internal docs), when you need citations, when the corpus is large. Fine-tuning when you need a specific style or format the model is bad at by default, when you have lots of labelled task examples, or when you need low-latency single-shot inference without retrieval.
Hybrid retrieval
Combine semantic (vector) retrieval with keyword (BM25) retrieval, then re-rank with a cross-encoder. This is the production pattern at most serious deployments — pure vector search misses keyword-heavy queries, and pure BM25 misses paraphrases. Hybrid + rerank handles both.
Exercise
A Kenyan SACCO wants to deploy an LLM-powered assistant that answers member questions about loan products, interest rates, and account procedures. The relevant information lives in the SACCO's policy manual (300 pages, updated quarterly) and FAQ database (2,000 entries). Two engineers debate: one wants to fine-tune an LLM on the manual; the other wants to use RAG. Which is right, and why? Walk through the specific factors that drive the decision.