In this guide · 8 sections
Short answer
RAG splits your documents into small chunks, turns each into a numeric embedding, and at question time retrieves the closest chunks and gives them to the AI model with the question. The model answers from those passages. It is easier to update than training a model, but only as good as its search.
What RAG is, in one paragraph
Large language models know what was in their training data, which does not include your price list, your staff handbook or last week's policy change. Retrieval-augmented generation fixes this at question time. The OWASP GenAI project describes RAG as combining pre-trained language models with external knowledge sources. Think of it as an open-book exam: the model is not memorising your documents, it is being handed the right pages just before it answers.
That is why RAG is the method behind most business AI assistants: file upload in ChatGPT and Claude projects, Google's NotebookLM, Microsoft Copilot answering from SharePoint, and website chatbots that answer from your own content.
How RAG works, step by step
Anthropic's explainer on contextual retrieval lays out the standard pipeline. In plain terms:
1. Split documents into chunks
Each document is cut into small pieces, usually no more than a few hundred tokens (roughly a few paragraphs). Small chunks make search precise.
2. Turn each chunk into an embedding
An embedding model converts each chunk into a list of numbers that represents its meaning. Chunks about similar topics end up with similar numbers.
3. Store them in a vector database
The embeddings go into a searchable index. OpenAI calls these vector stores; other tools use names such as knowledge base or index.
4. Search when a question arrives
The question is also turned into an embedding, and the system finds the chunks closest in meaning. Many systems add keyword search too, to catch exact codes and names.
5. Answer from the retrieved passages
The top chunks are added to the prompt with instructions such as "answer only from these sources and cite them". The model writes the answer.
OpenAI's retrieval guide shows why meaning-based search matters: the most relevant result can share none of the words in the question. It also shows its limits: by default a search returns up to 10 results, so anything not in those chunks is invisible to the model.
Why RAG answers can still be wrong
RAG reduces made-up answers but does not remove them. The common failure points:
| Problem | What happens | Typical fix |
|---|---|---|
| Wrong chunk retrieved | The search picks a passage that looks similar but is about something else | Hybrid keyword and meaning search, better chunking, reranking |
| Chunk lacks context | A passage says "fees rose 3%" but not which product or year | Add titles and context to each chunk |
| Old and new versions both present | The model mixes last year's price with this year's | Remove superseded files; keep one source of truth |
| Answer not in the documents | The model fills the gap from general knowledge | Instruct it to say "I don't know" and test that it does |
| Tables and scans | PDF tables and scanned images are parsed badly | Convert key tables to text; use OCR |
Anthropic reports that adding context to chunks and combining embeddings with keyword (BM25) search cut failed retrievals by 49% in its tests, and by 67% with reranking. The point for a business is not the exact number: it is that retrieval quality, not the model, decides most answers.
Keeping the documents current
A RAG system is only as current as its index. Most problems in production come from content, not technology:
Knowledge base hygiene
If your whole knowledge base is small, you may not need RAG at all. Anthropic notes that if it is under about 200,000 tokens (around 500 pages), you can include all of it in the prompt, with prompt caching to keep it fast and cheap. For many small businesses, a well-written FAQ and policy document fits inside that.
Permission-aware retrieval: the data leak to avoid
The most serious RAG risk in a company is not hallucination, it is access. If every document goes into one shared index, a junior employee's question can retrieve passages from HR files, salary sheets or board papers. OWASP lists this under LLM08:2025 Vector and Embedding Weaknesses and recommends permission-aware vector stores with strict partitioning between groups of users.
- Built-in assistants such as Microsoft Copilot respect existing file permissions, so oversharing in SharePoint becomes oversharing in Copilot. See our Copilot oversharing guide.
- Connectors that link ChatGPT or Claude to Google Drive or SharePoint need the same review; our connectors security guide covers it.
- Custom builds should filter retrieval by the user's role before the model sees anything, not rely on the prompt to "not mention" restricted content.
- Public website assistants should index only content you would publish anyway.
Treat documents as untrusted input too. A file containing hidden instructions can try to steer the model, which OWASP covers as prompt injection.
RAG, fine-tuning or just a better prompt?
RAG adds knowledge. Fine-tuning changes behaviour, such as style or output format, and is far less often what a small business needs. A better prompt with clear instructions and a few examples fixes many problems on its own. Our fine-tuning vs RAG guide walks through the decision.
For ready-made options, our guide to NotebookLM for business covers Google's research-style RAG tool, and our website AI chatbot guide covers customer-facing assistants.
Do it yourself or get help?
For internal use, start with a built-in tool your business already pays for, such as a ChatGPT or Claude project, NotebookLM or Copilot, and fix sharing permissions first. Build a custom RAG system only when you need control over sources, permissions or integration with your own systems.
We build website assistants that answer only from your approved content, with sources shown, as an AI Website Assistant (S$790). Internal assistants with role-based retrieval are scoped as a Custom Tool or Dashboard (from S$1,990).
Frequently asked questions
No. RAG searches your documents each time a question is asked and passes the most relevant passages to the model inside the prompt. The model itself is unchanged. Whether the provider may use your data for training depends on the plan and its terms, not on RAG itself, so check the data settings.
An embedding is a list of numbers that represents the meaning of a piece of text. Texts with similar meanings get similar numbers, which lets a system find relevant passages even when they use different words from the question. Embeddings are created by a separate, smaller model from the one that writes answers.
Usually because it retrieved the wrong passage, the passage lacked context, old and new versions conflicted, or the answer was not in the documents at all. Fix the content first, show sources with each answer, add keyword search alongside meaning-based search, and instruct the assistant to say when it does not know.
It can be, if retrieval respects permissions. If all documents share one index, anyone using the assistant may retrieve passages they could not normally open. Use tools that honour existing file permissions, review sharing settings first, and for custom builds filter results by user role before the model sees them.
Not always. Anthropic notes that if your material is under about 200,000 tokens, roughly 500 pages, you can put it all in the prompt instead, using prompt caching to keep costs down. Many small businesses' FAQs, price lists and policies fit within that, which is simpler to build and maintain.