What Is RAG? Making Company Knowledge Searchable with AI

"Where was it written down how we invoice warranty claims?" – In many companies the answer to a question like that exists somewhere: in the quality manual, in an old email, in a long-archived quote. A general-purpose AI model such as ChatGPT is only of limited help here, because it does not know your company. That gap is exactly what a method with the unwieldy name RAG closes.

Note: This article explains RAG from a decision-maker's perspective. For the technical side of setting such a system up, see my post on self-hosting Qdrant, a vector database for RAG. Last updated: August 2026.

What is RAG?

RAG (retrieval-augmented generation) is a method in which an AI language model first searches your own documents for the passages relevant to a question, then builds its answer on exactly those passages – rather than on whatever it happened to see during training.

Microsoft describes the flow in three steps: retrieve (find matching passages in your own content), augment (combine the question and the retrieved passages into a single prompt) and generate (produce an answer, with references back to the sources).

The important distinction is against retraining a model: with RAG the AI does not memorise your data, it is handed the data at the moment the question is asked – like a clerk who checks the file before giving an answer. A newly filed document is therefore available as soon as it has been added to the searchable collection. AWS names precisely this as one of the main reasons for RAG: new domain knowledge can be introduced without the expense of retraining a foundation model.

The term goes back to a research paper by Patrick Lewis and colleagues, published in 2020 and presented at the NeurIPS conference.

Sources: Microsoft Learn – Retrieval augmented generation (RAG), AWS – What is RAG?, Lewis et al. 2020 (arXiv)

Why does RAG reduce wrong answers?

Because the model does not have to reconstruct the answer from memory – it has the relevant original text in front of it and can hand back the source alongside the answer. Language models invent details when they need to bridge a gap in their knowledge – the technical term for this is "hallucination". Give them the right passage and the incentive to guess disappears.

The original 2020 paper already notes that RAG models generate "more specific, diverse and factual language" than comparable models without retrieval, and that they can provide provenance for their statements. For everyday business use, the second point matters almost more than the first: an answer with a citation can be verified by a person in seconds. An answer without one has to be taken on faith.

What RAG does not provide is a guarantee. Microsoft lists "hallucination despite grounding" explicitly as a known limitation ("grounding" is the technical term for supplying the model with those passages) – even with retrieved content, models can still generate inaccurate responses. The recommended countermeasures are enabling citations and giving the model clear instructions to stick to the retrieved content. And the old rule of data processing still applies: if the underlying documents are outdated or contradictory, RAG merely delivers the wrong answer faster.

Which company data is suitable?

The best candidates are written-down materials that are looked up frequently and are hard to find today. Typical examples in small and mid-sized businesses:

• Quality manuals, work instructions, standard operating procedures
• Quotes, service descriptions, reference projects
• Contracts, terms and conditions, maintenance and service agreements
• Support history, tickets, recurring customer questions
• Product and spare-part documentation, data sheets, manufacturer manuals
• Meeting minutes and acceptance reports

The file format is rarely the obstacle: PDF, Word, Excel, PowerPoint, HTML and database content can all be ingested, and scanned documents are made readable through text recognition. Scattered storage is no barrier either – Microsoft explicitly describes content spread across SharePoint, databases and file storage as the normal enterprise situation.

Less suitable are figures from live systems – stock levels, open items, revenue. For those, a report out of your ERP is the right tool, not a language model.

Source: Microsoft Learn – RAG in Azure AI Search

What about data protection and confidentiality?

The critical issue with RAG is not the language model but access control: the assistant may only show a person content that this person would be allowed to see without AI. Microsoft frames the problem with a very concrete example: finance data should only be accessible to the finance team, even when an executive asks the chatbot. And further: if you do not control access to the source content, answers backed by citations can leak sensitive information from the searchable collection.

Three practical consequences:

• Permissions belong in the retrieval step, not in the user interface. Filtering happens when searching, not when displaying.
• Retrieved document content must be treated like any other external input. Microsoft explicitly advises treating it as untrusted – documents can contain instructions designed to mislead the model.
• Where the model runs is a decision in its own right. Cloud service with a data processing agreement, or a model on your own infrastructure: I am not reopening that question here, it is covered in detail in Using ChatGPT in your business in compliance with GDPR.

For further reading: Germany's Federal Office for Information Security (BSI) has published "Generative KI-Modelle: Chancen und Risiken für Industrie und Behörden", a German-language guide aimed at companies and public authorities considering the use of generative AI.

Sources: Microsoft Learn – Security and privacy considerations, Microsoft Learn – RAG in Azure AI Search, BSI – Generative KI-Modelle

How does an entry-level project actually run?

1. Start with a question, not a technology. The starting point is one lookup that happens often and is annoying today – for example, "which maintenance intervals did we agree with customer X?". Without that reference point, there is no way to judge later whether the result is any good.
2. Narrow the content down. One storage location, one subject area, one team. Not "all our drives". Small, clean collections produce better results than large, neglected ones.
3. Prepare the documents. Long documents are split into meaningful sections, scanned pages are made readable through text recognition, and title, date and origin are carried along as metadata – the latter being the basis for citations later on.
4. Build the search index and map the permissions. This is where the searchable collection comes into being, and where it is decided who gets to see which sections at all.
5. Test with real questions. The people who handle the case every day put together a catalogue of genuine questions. What gets assessed is not just whether the answer sounds plausible, but whether the cited source actually supports it.
6. Pilot with a small group, then move to operations. A small group of users works with the system and reports back on poor hits. Only after that is it worth connecting further data sources and automating updates.

How much effort this takes depends mainly on the state of your documents. The technology is rarely the bottleneck; keeping the content in order almost always is.

Conclusion

RAG is not a new kind of artificial intelligence but a sober division of labour: search supplies the facts, the language model phrases the answer. For companies it is currently the most pragmatic way to make existing knowledge usable – with citations, and without retraining anything every time something changes.

The sensible first step is a small one: one frequent lookup, one clearly bounded set of documents, one test run with real questions.

Wondering whether your company knowledge is a fit for this? For businesses in Recklinghausen and the Ruhr region I am happy to take a look – matter-of-factly and without buzzwords. More under AI consulting or directly in a free initial consultation.

Publication note: This article was scheduled for the morning of 11 August 2026 and, because of a technical fault in our publishing automation, only went live later the same day. All information was checked again before publication to make sure it was up to date.