How RAG Works

Enterprise AI Systems, Explained · Part 3

Suppose an engineer asks an internal AI assistant:

What changed in the payment service before yesterday’s incident?

Assume the language model was not trained on yesterday’s deployment record, the latest runbook or the incident timeline. It cannot reliably supply those facts from what it learned during training.

A retrieval-augmented generation system searches approved company sources for relevant evidence, adds it to the model’s context—the information supplied for this request—and asks the model to answer from what was found.

That is RAG: retrieve first, augment the request with evidence, then generate an answer.

The basic idea is simple. The engineering work is in deciding what may be retrieved, which passages are relevant, how they are presented to the model, and how the resulting answer is checked.

In the diagram, the upper row prepares searchable knowledge; the lower row handles a user request. Query preparation is grouped into retrieval. Optional reranking, omitted from the overview, belongs after the access check and before context assembly. Permissions already constrain retrieval; the separate access check adds another safeguard.

What RAG is—and what it is not

Retrieval-augmented generation connects a language model to external information at request time. That information can come from documents, databases, search indexes, ticketing systems, product catalogs or other enterprise sources.

The 2020 paper that introduced the term described two sources of information: knowledge stored in the model’s learned weights, called parametric memory, and an external collection searched when needed, called non-parametric memory. In operational terms:

the model provides language understanding and generation;
the retrieval system provides evidence for the current request.

RAG does not retrain the model whenever a document changes. It does not permanently insert company facts into model weights. It supplies selected information in the request context, much like giving an engineer the relevant pages before asking for an explanation.

RAG can use different storage and search technologies. Vector search is common, but keyword search, SQL queries, knowledge graphs and external APIs can also supply information for the model request.

Two flows: preparing knowledge and answering questions

The architecture shown here contains two distinct flows:

Knowledge preparation runs as enterprise data changes.
Request processing runs every time a user asks a question.

The quality of the answer depends on both flows. If the knowledge pipeline is stale, incomplete or insecure, the model receives bad evidence even when the request is carefully written.

Flow one: preparing company knowledge

1. Connect approved sources

The knowledge pipeline reads from explicitly approved sources: document repositories, wikis, support systems, databases, object storage or internal APIs.

Each connector needs an owner, credentials, synchronization rules and a failure path. The system must detect new, changed and deleted records, as well as changes to access permissions. Those changes must reach the search index and any response caches, so an old copy cannot continue granting access.

2. Parse the source correctly

A PDF is not automatically clean text. Documents contain headings, tables, footnotes, diagrams, repeated headers and scanned pages. Parsing determines which of those structures survive into retrieval.

If a table loses its row labels or a procedure loses its section heading, the index may contain words without the context required to interpret them.

3. Divide content into useful passages

Large documents are normally split into smaller passages called chunks. Small chunks can match a precise question; larger chunks preserve more context. The correct boundary depends on the content.

A fixed character count is easy to implement, but structure-aware chunking is often more useful. A contract clause, troubleshooting step and source-code function should not be divided in the same way.

4. Attach metadata and access attributes

Each passage should retain metadata: fields describing its source, version and access rules. Include the source identifier, modification date and authorization attributes, plus fields such as owner, language or jurisdiction when the application needs them. In a shared service, a tenant identifies the organization or account whose data must remain isolated.

Metadata lets the request flow select current information, enforce exact filters and return citations to the original record.

5. Embed and index

When using vector search, the pipeline converts each passage into an embedding: a list of numbers that represents features of its content for comparison. It stores these vectors in a search index alongside the passage text or a reference to it. A text index can also support searches for specific words, identifiers and error messages.

At search time, the query is converted into an embedding compatible with the stored passage vectors. This often uses the same model, with query and document settings recommended by its provider. Comparing these vectors helps locate related content. RAG then uses the retrieved content to help generate an answer.

Flow two: answering a request

1. Receive the question with identity and context

The application sends the question together with the authenticated identity, tenant, conversation state and relevant application context.

Identity must arrive before retrieval. If the system searches globally and filters sensitive passages only after they reach the model, the security boundary is already in the wrong place.

2. Prepare the query

A user’s wording is not always a good search query. The orchestrator, the application component that coordinates these steps, may correct spelling, expand an abbreviation, identify an exact product code or split a complex question into smaller searches.

Query rewriting must remain traceable. Operators should be able to see the original question, the rewritten queries and the sources each query returned.

3. Retrieve a broad candidate set

The retriever searches one or more eligible sources and returns candidate passages: results that might help answer the question. Tenant and access constraints restrict this search; the later access check adds a second safeguard. A common pattern is hybrid retrieval, which combines vector and keyword search:

vector search finds passages with similar meaning;
keyword search finds exact names, codes and phrases;
metadata filters restrict the search space.

The goal of this stage is recall: avoid missing evidence that may answer the question. The first result list can therefore contain passages that are related but not precise enough.

4. Enforce permissions

The platform verifies that the requesting identity may use every candidate under the current permissions. Unauthorized content must be removed before it is returned to the user or sent to another service, including an external ranking service or generation model. Implementations can restrict the search in advance, use separate indexes for different access groups, or filter results using access attributes. If the platform cannot verify access, it must stop rather than pass the content onward.

A relevant document is not necessarily an authorized document. This boundary is examined in Enterprise RAG Is an Authorization Problem Before It Is a Search Problem.

5. Rerank for precision

Vector similarity measures general relatedness. Keyword scores measure term matches. Neither necessarily tells us whether a passage answers this exact question.

A reranker evaluates the query and candidate passages together, then places the most useful evidence first. The aim is precision: a larger share of the selected passages should help answer the question. Microsoft describes this balance between recall and precision. AWS also supports reranking before generation.

Reranking adds latency—the time a user waits for a response—and cost. Measure whether the improvement justifies that overhead for the workload.

6. Assemble the context

The orchestrator selects the final passages and builds the model request, or prompt. A typical RAG prompt contains:

system instructions;
the user’s question;
the retrieved passages with source identifiers;
rules for conflicting or missing evidence;
the required answer and citation format.

Irrelevant passages dilute the useful evidence and use part of the model’s limited input space. Context assembly should select enough relevant material to answer the question while preserving any conditions needed to interpret it.

7. Generate the answer

The language model receives the assembled prompt and produces an answer. The instructions should tell it to use the supplied evidence, identify uncertainty and refuse to invent missing details.

The model is still generating text probabilistically. RAG improves the information available to it; RAG does not turn generation into a deterministic database lookup.

8. Return citations and trace the operation

The application maps statements or answer sections back to the supplied source passages. A citation should identify evidence supplied to the model, not merely attach a plausible-looking document after generation.

Check both that a citation points to a supplied passage and that the passage supports the associated claim. A valid source identifier alone does not make the claim correct.

Correlate the request identity, source identifiers, applied filters, ranking decisions, prompt and model versions, timing and outcome. Do not log full documents, prompts or answers by default. Capture content only when permitted, with redaction, restricted access and defined retention; source versions or protected references can support investigation without duplicating sensitive data in ordinary logs.

A small worked example

Return to the payment-service question. This is a fictional teaching example; the records and values below are invented.

Suppose the engineer is allowed to read two records:

Deployment record D1: “At 14:05, the payment service’s request timeout changed from 2 seconds to 5 seconds.”
Incident timeline I1: “At 14:12, the payment service’s error rate began to rise.”

Retrieval finds these passages. The application checks access and supplies their text and source identifiers to the model. A supported answer would be:

The request timeout increased from 2 seconds to 5 seconds at 14:05 [D1], seven minutes before the recorded rise in errors [I1]. These records establish the sequence of events, but do not establish that the timeout change caused the incident.

The labels refer to the invented records above. If the deployment record is unavailable, the assistant should say that it cannot identify the change from the available evidence.

Retrieved documents are data, not instructions

A document can be authorized for a user and still contain malicious instructions. A retrieved page might tell the assistant to ignore its rules, reveal other context or call a tool. This is indirect prompt injection.

Treat retrieved text as untrusted evidence, not as authority to change the workflow. Separate instructions from source content, test adversarial documents, and enforce tool permissions and input validation outside the model. A prompt alone is not a security boundary. High-impact actions may also need explicit approval. See the OWASP RAG Security guidance.

Why RAG can still produce a wrong answer

A hallucination is an incorrect or unsupported claim presented as fact. RAG gives the model better evidence and makes answers more traceable, but it cannot eliminate these errors.

The final result can still fail at several stages:

the required document was never indexed;
parsing removed important structure;
chunking separated a statement from its conditions;
retrieval missed the correct passage;
permission filters removed too many candidates—or too few;
reranking selected a related but incorrect source;
the prompt did not explain how to handle conflicting evidence;
the model ignored or misread the supplied context;
citations were attached to claims they do not support.

“The model hallucinated” is therefore not a sufficient incident diagnosis. The failure may have occurred before the model was called.

Standard RAG and agentic RAG

Standard RAG follows a designed sequence: accept the question, run a known search, assemble context and call the model. This works well when a request maps to a predictable source and a single retrieval step.

Agentic RAG treats retrieval as a tool. An agent uses a model to choose its next step: which source to search, whether to break a question into smaller searches, or whether to retrieve again after inspecting intermediate results.

That flexibility is useful for complex tasks, but it also adds variable cost, latency and behavior. A fixed pipeline is often the better starting point. Agentic retrieval should be introduced when the workload requires dynamic decisions, not because it sounds more advanced.

Evaluate retrieval and generation separately

End-to-end answer quality alone does not show where the system is improving or failing.

Retrieval evaluation asks:

Did the system find the required passage?
How much of the returned context was relevant?
Did filters preserve all authorized evidence?

Generation evaluation asks:

Is the answer correct and complete?
Is it faithful to the retrieved context?
Do the citations support the claims?
Does the system refuse when evidence is insufficient?

AWS evaluates retrieval and generation separately. Context relevance asks whether the returned passages relate to the question; context coverage asks whether they include the information in a prepared reference answer. Faithfulness checks whether the generated answer stays within the retrieved evidence. Citation precision checks whether citations are used correctly, while citation coverage checks whether the answer has the citations it needs. Context coverage requires reference information in the test dataset. These measures help diagnose different failures; none alone proves an answer is correct.

A practical production checklist

Before calling a RAG system production-ready, verify that the team can answer these questions:

Which sources are approved, and who owns them?
How are content updates, deletions and permission changes propagated?
How are tables, images and document structure parsed?
How are chunk boundaries chosen and versioned?
Which identity and policy restrict each search?
When are keyword, vector or structured queries used?
How are candidates merged and reranked?
What happens when sources conflict or provide no answer?
Can each claim be traced to evidence the model received?
Can operators reconstruct the entire request?
Which test set measures retrieval and answer quality?
What latency, cost and failure budgets apply?

Where RAG fits in the enterprise AI system

RAG is not the whole AI platform. It is the evidence path between enterprise knowledge and model generation.

The surrounding platform still owns identity, model access, policy, secrets, monitoring, evaluation, cost attribution and release control.

The essential runtime sequence is:

User and identity → query preparation → retrieval within access boundaries → current permission check → optional reranking → context assembly → model generation → answer and validated citations.

RAG supplies selected evidence for a request without changing the model’s learned weights. The reliability of the answer depends on the engineering of that entire evidence path.

Continue with How AI Agents and Tools Work, Part 4 of the series.