RAG (Retrieval-Augmented Generation) is a setup where the AI first finds the relevant fragments in your company’s documents and only then writes an answer strictly from them, with a link to the source. The model answers from your current price list, policies and knowledge base, and any answer can be checked in a couple of clicks. Below: what such a system is made of, where it loses accuracy and how to fix it.
Why AI without access to your documents gets things wrong
An out-of-the-box language model is trained on general text. It knows nothing about your rates, delivery times to a particular city or the fact that the returns policy changed last week. Ask it directly and it rarely says “I don’t know”; it is more likely to assemble a plausible answer from what it has seen before. In a customer conversation that is dangerous: a confident invention about price or warranty is worse than an honest “let me check”.
The second problem is verification. Even when the answer is right, an employee can’t see where it came from and has to double-check. RAG solves both: the model receives the retrieved document fragments as input, answers only from them and shows which document each part of the answer came from.
What RAG is made of: two loops
The system runs in two modes. Preparation runs once and then every time the documents change. Answering runs for every question from an employee or a customer.
Steps 1–3 are preparation, done once and on every document update. Steps 4–5 run for every question.
- Collecting sources. Documents, the knowledge base, wiki, support history, CRM data: everything the AI will rely on.
- Splitting into fragments. Each document is cut into meaningful pieces: a section of a policy, a price row with its conditions, an answer from the knowledge base.
- Index. The fragments go into a searchable base that searches by meaning and understands a question asked in different words.
- Retrieval. For each question the system finds a few of the most relevant fragments.
- Answer. The model gets the question and the retrieved fragments, writes an answer from them and attaches links to the documents.
This article is about the architecture and accuracy. If you need the business side, see our page on a corporate knowledge base with AI.
Preparing documents is half the job
The AI answers exactly as well as the documents underneath it. If a folder holds three versions of the price list, retrieval may find the old one. So before launch we go through the sources together with the client.
| What we check | Why |
|---|---|
| Duplicates and old versions | So that search doesn’t find an outdated policy next to the current one |
| Contradictions between documents | If two places state different terms, the AI must not pick one itself: the document owner decides |
| An owner for every source | Someone is responsible for keeping the price list and policies up to date |
| Access rights | Which documents sales reps see and which are for managers only |
| Format | Scans without a text layer and tables saved as images are hard to search; convert them to text |
You don’t need to rewrite documents for the AI. Usually it is enough to remove clutter and agree who updates each source and how often.
Where RAG loses accuracy
Answer quality depends most of all on retrieval. If the wrong fragments come back at this step, even a strong model answers off-target. The main failure points:
- Fragment size. Pieces that are too large blur the meaning; pieces that are too small lose context, so a condition ends up in one fragment and its exception in another. We split along the document’s own structure: sections, clauses, table rows.
- Exact terms. Search by meaning handles a rephrased question well but can miss a SKU, a tariff code or an acronym. So it is paired with exact-word search and the results are merged.
- Ranking. A separate step re-sorts the retrieved fragments by how well they answer this particular question. The model sees the most relevant ones first.
- Questions spanning documents. When an answer needs two sources, for example delivery terms and the price list, retrieval has to return both. Such questions deserve their own tests.
Retrieval owns accuracy. A strong model on the wrong fragment gives a confident wrong answer.
How the AI writes the answer
The model gets a strict instruction: answer only from the fragments provided, cite them and never fill in what is missing. If the documents don’t contain the answer, the right reply is “this isn’t in the documents”, and in a customer channel the question goes to an employee. This is configured explicitly and tested with questions that deliberately have no answer in the base.
The second part is format. A sales rep wants a short answer and a link to the policy clause; a customer needs plain language without internal jargon. We write different instructions for different channels on top of the same knowledge base.
Freshness, access and data
Three requirements separate a working system from a demo.
Freshness. The index updates when documents change. Otherwise the AI will confidently answer from last year’s price list. Who updates each source and how often is agreed before launch.
Access. Permission filtering is built into retrieval: an employee won’t get, through the chat, a fragment of a document they aren’t allowed to see. A customer in a messenger only gets answers from the public part of the base.
Data. We work under NDA. Which documents and fields may be passed to the model, and which stay closed, is fixed before the project starts.
How to check quality before launch
Before launch we collect a set of real questions: from support email, from chats, from sales reps. For each one the correct answer and the document it lives in are known in advance. We run the questions through the system and look at three things: was the right document found, is the answer correct, and does the system honestly say “not found” when there is no answer.
Errors almost always come down to one of two causes: a gap or contradiction in the documents, or poor splitting and retrieval. The document owner fixes the first, we fix the second. After launch the test set grows with every question the system got wrong in real use.
When RAG pays off and when it’s overkill
| Situation | Do you need RAG? |
|---|---|
| Hundreds of documents, the product or terms change often | Yes: without document search, answers go stale fast |
| Support answers complex questions from documentation | Yes: the AI drafts an answer with links, a person checks it |
| Sales reps look up answers in policies and price lists | Yes: an answer in seconds instead of digging through folders |
| A dozen standard questions whose answers never change | No: a simple script or an AI agent with a short instruction is enough |
| Hardly any documents, knowledge lives in people’s heads | Build the knowledge base first, then connect the AI |
An example from our work: the support team of a B2B SaaS company answers from a knowledge base with AI. Request handling became 48 times faster, 90% of requests are closed without a human, and answers go out 24/7. Details in the B2B SaaS case study.
Cost and how we implement it
A corporate knowledge base with AI is a separate module from $650; a full project starts from $3,750, and ongoing support from $400 a month. We start with a free one-hour express audit of your sources and tasks, prepare an implementation plan within 3 more hours, then run a pilot on your documents and launch from 7 days. Two weeks of support are included. If the same knowledge base should answer customers in messengers, see how an AI agent qualifies leads on top of it.
Frequently asked questions
How is RAG different from plain ChatGPT?
A public chatbot answers from the model’s general knowledge and can’t see your documents. With RAG the model first receives retrieved fragments of your base and answers only from them, with a link to the source.
Do we need to rewrite our documents for the AI?
No. Usually it is enough to remove duplicates and old versions, resolve contradictions and assign document owners. Scans without a text layer are better converted to text.
What happens if the answer isn’t in the documents?
The system says so directly, and in a customer channel it passes the question to an employee. This is configured and tested before launch.
How does the AI learn about price or policy changes?
The index updates when documents change. The update routine and the owner of each source are agreed before launch.
Where is the data stored?
We work under NDA. What may be passed to the model and what stays closed is fixed before the project starts.