Retrieval-Augmented Generation (RAG)
Definition
RAG is an architecture that combines a retrieval system (fetching relevant documents from an index) with a generative model (producing text conditioned on those documents). The model generates answers grounded in retrieved evidence rather than solely in its training weights.
Core Ideas
How RAG Works
- Index — source documents are chunked and embedded into a vector store (e.g. Chroma, Pinecone, Weaviate)
- Retrieve — at query time, the query is embedded and nearest-neighbour search fetches the top-k relevant chunks
- Augment — retrieved chunks are inserted into the LLM prompt as context
- Generate — the LLM produces an answer grounded in the retrieved context
Why RAG
| Problem | RAG Solution |
|---|---|
| Hallucination | Ground answers in real retrieved documents |
| Knowledge cutoff | Connect to live or updated knowledge stores |
| Private data | Index internal documents not in training data |
| Citation | Retrieved chunks provide traceable sources |
RAG Variants
- Naive RAG — simple retrieve-then-generate pipeline
- Advanced RAG — query rewriting, re-ranking, multi-hop retrieval
- Modular RAG — pluggable components for retrieval, fusion, generation
- Graph RAG — entity/relation extraction into a knowledge graph; retrieve via graph traversal
Embedding Models
- OpenAI
text-embedding-3-small/large - Sentence Transformers (open source)
- Domain-specific fine-tuned embedders
Evaluation
- Faithfulness — is the answer supported by the retrieved context?
- Answer relevance — does the answer address the question?
- Context relevance — are the retrieved chunks actually useful?
- Tools: RAGAS framework, LLM-as-judge
Model-Routed Retrieval vs Vector RAG
Selecting the right knowledge is retrieval — and there are two ways to route to it:
| Vector RAG | Model-routed (skills) | |
|---|---|---|
| Routes by | Embedding similarity | The model reading descriptions and choosing |
| Infrastructure | Vector store + embeddings | None — just skill folders with descriptions |
| Best for | Hundreds of fine-grained, overlapping units | A curated set of nameable clusters (a few dozen) |
| Hardest failure mode | Misses relevant notes when words don’t match | Can reason that a growth-stock question needs the margin-of-safety principle |
For a curated knowledge set, model-routed retrieval (via MCP skills) is the better default: simpler, no embedding infrastructure, and stronger on semantic-mismatch retrieval. The two compose — when volume outgrows description-routing, add vector RAG underneath as a narrowing step: similarity pulls a candidate slice from the large pile, and the skill/model still does the final reasoning.
Signals to add vector RAG under skills: (a) hundreds of fine-grained notes whose descriptions no longer route cleanly, or (b) loading a whole skill body to use one sentence becomes wasteful. When you do, keep finance-style metadata (type, durability, asserted_on, conviction) so a time-bound opinion isn’t served with the authority of a timeless principle, and prefer metadata pre-filter → hybrid (BM25 + vector) → reranker over naive similarity search.
Relationships
- Machine Learning — embedding models are ML models
- AI & Machine Learning — RAG is a primary LLM application pattern
- Entrepreneurship & SaaS — RAG powers AI SaaS products (knowledge assistants, search)
- Cloud & AWS Infrastructure — vector stores and embedding inference run on cloud
- Model Context Protocol (MCP) — skills delivered over MCP are the model-routed alternative to vector retrieval
- Vector Databases and pgvector in Production — the retrieval layer’s production reality: index builds, filtered search, hybrid search