Ask a general AI assistant about your company’s returns policy, last quarter’s internal report or a document you wrote yesterday, and it has no way of knowing the answer. It was built from text collected before you asked, and it has never seen your files. So it either admits it does not know or, worse, produces something that sounds right.
Retrieval-augmented generation, usually shortened to RAG, is one of the most common ways of solving this. This guide explains what RAG is in plain language, how the pieces fit together, what it is good and bad at, how it differs from other approaches, and what questions to ask if a vendor says their product uses it. No coding knowledge is needed.
Key takeaways
- RAG means the system first looks up relevant information from a chosen source, then gives that information to the AI model so it can write an answer based on it.
- It helps AI answer questions about material the model was never trained on, such as your documents, policies or recent notes.
- RAG reduces invented answers but does not remove them: the quality depends on what is retrieved and how it is used.
- Most RAG problems are really content problems, such as messy documents, outdated files or poor organisation.
- Ask vendors what sources are used, how answers are checked, whether citations are shown and how your data is stored and protected.
The problem RAG solves
A language model learns patterns from a very large body of text during its building. That training has two limits that matter to everyday users. First, it stops at some point in time, so the model does not know about anything that happened or was written afterwards. Second, it never saw your private material. Our guide to what a large language model is covers the underlying idea.
Why not just paste your documents in?
For a short document, pasting it into the chat works well. But you can only paste so much, because the model can only consider a limited amount of text at once. If you have hundreds of policies, thousands of support articles or years of reports, they will not fit. You need a way to pick out only the pieces relevant to each question.
The open-book exam idea
A helpful comparison is an exam. A model answering from training alone is like a student answering from memory, with all the risk of misremembering. RAG turns it into an open-book exam: before answering, the system finds the right pages in the book and puts them in front of the student. The student still has to read, understand and write a good answer, but now has the facts at hand.
How RAG works, step by step
The name describes the process. Retrieval means finding information. Augmented means the model’s input is enlarged with that information. Generation means the model then writes the answer.
- You ask a question. For example: “How many days do customers have to return an item?”
- The system searches your sources. It looks through a prepared collection of documents for passages that seem related to the question.
- It selects the best matches. A small number of the most relevant passages are chosen.
- It builds a new prompt. Behind the scenes, the system combines your question, the selected passages and instructions such as “answer using only these passages”.
- The model writes the answer. It reads the combined prompt and produces a response based on the supplied text.
- The answer is shown, ideally with sources. Good systems show which passages were used so you can check them.
What you see and what you do not
From your side it looks like an ordinary chat. The retrieval step happens out of sight, which is why it can be hard to tell why an answer was good or bad. Tools that show their sources make this much more transparent.
Why the model needs help finding things
The model can only handle a limited amount of text per request, a limit explained in our article on the context window. Retrieval is the filter that decides which small slice of your large collection earns a place in that limited space.
The main parts explained
A RAG system has several moving parts. You do not need to build them, but knowing their names helps you read product descriptions and spot weak points.
| Part | Plain-English meaning | What can go wrong |
|---|---|---|
| Source documents | The files and pages the system can draw from | Outdated, duplicated or contradictory content |
| Chunking | Splitting documents into smaller passages | Passages that cut off mid-thought or lose context |
| Index or search store | An organised catalogue of the passages for fast lookup | Not refreshed when documents change |
| Retriever | The component that finds passages relevant to a question | Misses the right passage or returns near-misses |
| Prompt builder | Combines the question, passages and instructions | Weak instructions allow the model to ignore the passages |
| Language model | Writes the final answer | May still misread or overreach |
| Citations or sources | Shows where the answer came from | Missing, or pointing to the wrong place |
Chunking in more detail
Documents are usually cut into pieces, often a few paragraphs each, because retrieving a whole long document is wasteful. The way text is cut matters. If a rule is in one chunk and its exception is in the next, a retriever might fetch only the rule, and the answer would be misleadingly incomplete.
Search by meaning, not only keywords
Many RAG systems use a type of search that matches meaning rather than exact words. A question about “sending a product back” can match a passage titled “Returns and refunds”, even though the words differ. Some systems combine meaning-based search with ordinary keyword search, since exact terms such as product codes or names are better matched by keywords.
Instructions to the model
The prompt builder usually tells the model how to behave: use only the supplied passages, say when the answer is not there, and show sources. How well these instructions are written strongly affects how trustworthy the system is.
A worked example
Imagine a small online shop that wants a help assistant for its customers. The shop has a returns policy, a delivery guide and a set of product care pages.
Setting up
The owner uploads the three sets of documents. The system splits them into passages and builds a searchable catalogue.
A good question
A customer asks whether a gift can be returned without a receipt. The system finds the passage about proof of purchase, adds it to the prompt, and the model writes a short answer quoting the policy and linking to the source page. The customer can verify it in a click.
A question that exposes a gap
Another customer asks about a discontinued product that is not in any document. A well-built system retrieves nothing relevant and replies that it does not have that information, then suggests contacting the shop. A poorly built one may still produce a confident answer from the model’s general knowledge, which might be wrong for this shop.
A question that exposes outdated content
If an old policy page was never removed, the system may retrieve both old and new versions and blend them. The fault is not the AI; it is the untidy source material. This is why content housekeeping is half of the work. If you are planning a customer-facing assistant, see our guide to planning an AI customer support chatbot.
RAG compared with other approaches
RAG is not the only way to make an AI tool useful for your own information. Here is how the main options differ in broad terms.
| Approach | What it means | Good for | Limitations |
|---|---|---|---|
| Pasting text into the chat | You supply the material directly | One-off tasks with short documents | Does not scale; manual each time |
| RAG | The system retrieves relevant passages for each question | Question answering across many documents that change | Depends on retrieval quality and content tidiness |
| Fine-tuning | Further training a model on examples to adjust its behaviour | Teaching a consistent style or format | Poor at keeping facts current; needs technical work |
| Custom instructions | Standing rules the model sees every time | Tone, audience and preferences | Limited space; not a knowledge base |
| Web search features | The tool searches the internet for the question | Current public information | Quality depends on what is found; not your private data |
RAG and fine-tuning are not rivals
People often ask which is better. They solve different problems. Fine-tuning shapes how a model behaves, such as the style it writes in. RAG supplies facts at the moment of the question. For information that changes, RAG is usually the more practical route, because updating a document is far easier than retraining anything.
RAG and web search
Some assistants search the open web and then answer from the results. That is a form of retrieval, with the web as the source. The same strengths and weaknesses apply: answers are only as good as the pages found.
What RAG does well
- Keeps answers tied to your material. Instead of guessing, the model works from supplied text.
- Stays current. Update a document and the next answer can reflect the change, with no retraining.
- Shows sources. Citations let people check claims, which builds trust.
- Handles large collections. It picks out the relevant slice from a volume too big to paste.
- Reduces, though does not remove, invented answers. Grounding in text lowers the chance of fabrication.
Why grounding helps
When the model has the relevant passage in front of it, it has less reason to rely on vague memory. That matters because invented details are a core weakness of these tools, as we explain in our guide to AI hallucinations.
Where RAG goes wrong
RAG is not magic, and its failures can be subtle because the answers still read fluently.
| Failure | What it looks like | Typical cause |
|---|---|---|
| Missed retrieval | The answer says nothing is available when the document exists | The passage was phrased differently, or poorly chunked |
| Wrong retrieval | The answer uses a related but incorrect passage | Similar documents confuse the search |
| Incomplete retrieval | The answer gives a rule but omits the exception | Only part of the relevant text was fetched |
| Misreading | The model draws the wrong conclusion from a correct passage | Ambiguous wording, or the model overreaching |
| Stale content | The answer reflects an old version | Documents or the index were not updated |
| Blending | The answer mixes the documents with general knowledge | Weak instructions to stick to the sources |
Questions that need reasoning across documents
Simple lookups work well. Questions that need information combined from many places, such as “what are the common themes across all customer complaints this year”, are harder, because retrieval returns only a handful of passages. Such tasks may need a different design.
Confidence without correctness
A RAG answer with a citation can look authoritative even if the cited passage does not truly support it. Spot checks matter. Follow the routine in how to check AI output for mistakes, and click through the sources for anything important.
Preparing your content
The most useful thing a non-technical person can do for a RAG project is tidy the source material. Systems cannot fix what people have left in a mess.
A content preparation routine
- List what should be included. Decide which documents are authoritative for which topics.
- Remove duplicates and old versions. One current version of each document is far better than five.
- Give documents clear titles and headings. Headings give the system and your readers useful signposts.
- Write in plain, complete sentences. Avoid fragments that only make sense with the surrounding page.
- Put exceptions next to rules. Keep a rule and its exceptions in the same section.
- Add dates and owners. Record when each document was last reviewed and who maintains it.
- Convert scanned files properly. Image-only PDFs may not be readable by the system unless text is extracted.
Decide what must stay out
Not everything should be searchable by everyone. Drafts, personal data, confidential contracts and anything with access restrictions need deliberate handling. Do not feed material into a system without knowing who will be able to query it.
Plan for upkeep
Treat the knowledge base like a product. Assign someone to review it on a regular schedule, retire outdated items and add new ones. Many disappointing RAG projects fail because the content was left to rot.
Common uses for non-technical teams
Internal help desk
Staff ask questions about policies, procedures and tools, and get answers with links to the source document. Because policies change, retrieval beats memory-based answers.
Customer support assistance
Support agents or customers get answers drawn from help articles. Many teams start with agent-facing use, where a human checks the draft before it is sent.
Research and reading
People working with a set of reports or articles can ask questions across them and be pointed to the relevant passages, saving time on searching.
Onboarding
New starters can ask plain questions about how things work and get answers tied to the handbook. It does not replace a person, but it reduces repeated basic questions.
Sales and proposal support
Teams can retrieve past answers and standard wording for questionnaires and proposals. A human should review anything that makes a commitment.
If you want to join several steps together without code, our guide to building your first AI workflow without code shows how retrieval fits into a larger process.
Questions to ask a vendor
When a product claims to use RAG or to “chat with your data”, marketing language can hide big differences. These questions cut through it.
| Topic | Question to ask | What a good answer looks like |
|---|---|---|
| Sources | Which documents and formats can be used, and how are they updated? | Clear list, with an explanation of how changes are picked up |
| Citations | Does every answer show where it came from? | Yes, with links to the exact passage |
| Unknown answers | What happens if the answer is not in the sources? | It says so instead of guessing |
| Access control | Can answers respect who is allowed to see which documents? | Permissions are honoured, not bypassed |
| Data handling | Where are my documents stored, and are they used to train models? | A specific written answer |
| Testing | How can I test quality on my own questions? | A trial where you can use real documents |
| Exit | Can I export or delete my data and index? | Yes, with a described process |
Test with your own hard questions
Build a list of twenty questions, including a few where the answer is not in your documents and a few where an exception applies. Run them and check the results. Our guide to choosing an AI tool offers a framework for this kind of evaluation.
Privacy and security considerations
RAG systems store copies or representations of your documents so that they can be searched. That makes data handling central, not an afterthought.
- Find out where the indexed content is stored and who at the vendor can access it.
- Check whether the system respects document permissions, so staff cannot retrieve files they should not see.
- Keep especially sensitive material, such as personal records, out unless you have clear approval and clear terms.
- Know how to delete documents and confirm they are removed from the index.
- Write down who may add content, and review what has been added.
Our guide to protecting company data when using AI tools goes through these questions in more detail. If your documents contain personal data, rules such as data protection laws may apply to you. This is general information, not legal advice, so speak to a qualified adviser where the stakes are real.
Frequently asked questions
Does RAG stop AI from making things up?
It lowers the risk but cannot eliminate it. The model can still misread a passage, combine sources badly or fill gaps from general knowledge. Citations and spot checks are the practical safeguard.
Do I need to be technical to use RAG?
Not to use it. Many products offer it as a feature where you upload files and ask questions. Building a custom system needs technical skills, but preparing good content, which matters most, does not.
Is RAG the same as training the AI on my data?
No. In RAG, your documents are looked up at question time and supplied to the model as reference text. The model itself is not changed. Whether a vendor also uses your data for training is a separate question you should ask directly.
How is RAG different from a normal search engine?
A search engine gives you a list of documents to read yourself. A RAG system finds relevant passages and then writes a direct answer based on them. That is convenient, but it also adds a step where errors can creep in, so checking sources remains important.
How many documents can a RAG system handle?
It depends on the product and plan. Limits are usually stated in file counts, file sizes or total storage. Read the limits before you commit, and test with a realistic amount of your own content.
Why are the answers worse than I expected?
Look at your content first. Outdated, duplicated or badly structured documents are the most common cause. Then check whether the right passages are being retrieved by looking at the sources shown for poor answers.
Conclusion
Retrieval-augmented generation lets an AI model answer from your own information by looking up relevant passages first and writing from them. It is an open-book approach: more current, easier to check and usually more reliable than answering from memory, but still dependent on good content, good retrieval and sensible human review.
If you are considering a RAG-based product, start with your content and your hard questions, not the marketing claims. Tidy the sources, test with real examples and read the data terms. For related reading, see our explainer on AI agents, which often use retrieval as one of their tools, and our guide to checking AI output for mistakes for a routine you can apply to any answer.
