In short
Use CAG when the knowledge is small, stable and asked about constantly. Use RAG when it is large, changes often or needs per-user permissions. Many real systems end up with a bit of both.
Most "chat with your documents" projects start with the same question: how does the model get the right information at the moment it answers? Two patterns cover nearly every case we build. They trade off very differently, so it is worth choosing on purpose.
The two patterns in one sentence each
RAG (retrieval-augmented generation) searches your knowledge base for every question, picks the most relevant passages and puts only those in the prompt.
CAG (cache-augmented generation) puts the whole knowledge base in the prompt once, caches that long prefix with the provider's prompt caching, and reuses it for every question.
RAG sends the model a small, targeted slice. CAG sends the model everything, but while the cache is warm, later requests read that prefix faster and at a much lower price.
Where CAG wins
- Nothing to retrieve, so nothing to miss. The most common RAG failure is not the model, it is the search: the right paragraph never made it into the prompt. With CAG the model sees all of it.
- Simpler to build and run. No chunking strategy, no embeddings, no vector database to keep in sync. Fewer moving parts means fewer things to debug.
- Questions that span the whole corpus. "Which policies mention night shifts?" or "summarise every change since version two" are hard for retrieval and natural for a model that can read everything.
- Fast repeat answers. While the prefix is cached, follow-up questions skip re-processing the knowledge, which keeps latency and cost down for busy assistants. Caches expire after a while, so a quiet assistant pays full price again on its next question.
Where RAG wins
- Large knowledge bases. If the material is far bigger than a context window, or would make every request expensive even with caching, you have to choose what goes in.
- Content that changes all the time. Every change invalidates a CAG cache. RAG updates one document's chunks and carries on.
- Per-user permissions. When different people may see different documents, retrieval can filter by access rights before anything reaches the model. With one shared cached context, that is much harder to get right.
- Citations. Retrieved passages come with their source, so answers can link to the exact page or section.
Five questions we ask before choosing
- How big is the knowledge, and how fast is it growing? Small and steady points to CAG; large or growing quickly points to RAG.
- How often does it change? Daily edits make a cache stale all day. Monthly releases are easy to re-cache.
- Does everyone see the same content? Shared handbooks suit CAG. Customer records and permissioned files suit RAG.
- What do the questions look like? Lookups of one fact favour RAG. Comparing, summarising and "everything about X" favour CAG.
- How will you test it? For RAG, measure retrieval separately: is the right passage in the top results? For CAG, test long-context answers on realistic questions, including ones about the middle of long documents.
The hybrid we use most
In practice many systems combine them. A small, stable core, such as product rules, pricing logic or a style guide, is loaded and cached for every request. The large, changing or permissioned material sits behind retrieval and is fetched per question. The model gets the context it always needs plus the specific facts this question needs.
Where to start
Collect twenty real questions people ask today, with the answers an expert would give. Try the simplest option first: if the knowledge fits comfortably in the context, cache it and test against those questions. Add retrieval only where the tests show it is needed, and keep the question set as your regression test as the content grows.
