Enterprise AI Systems, Explained · Part 1

Open the enterprise AI diagram at full size.

When people talk about enterprise AI, the discussion often jumps straight to models: GPT, Claude, Gemini or a model running inside the company.

But the model is only one part of the system.

A large language model, or LLM, can interpret a request and generate text. By itself, it has no reliable access to the company’s current documents and cannot enforce user permissions or safely operate business systems.

To use AI in a real organization, engineers have to build a complete system around the model.

Take a simple question: “Which termination conditions are included in our supplier contract?” Something must find the right contract, check that the employee may read it, and give the relevant passages to the model. We will follow that path, then look at what changes when the assistant also needs to perform an action.

This is a reference design for a knowledge assistant that can also use business tools, not a mandatory checklist for every AI application. Classification, extraction or a small internal assistant may need only some of these components. A separate gateway, vector index and agent loop are architectural choices, not prerequisites.

What is an enterprise AI system?

In this series, an enterprise AI system is an application built around a language model, with the data access, permissions and operational controls needed for use inside an organization.

For this knowledge-and-tools reference design, the components include:

  • user-facing applications;
  • identity and access control;
  • search over company data;
  • RAG;
  • an LLM gateway;
  • one or more language models;
  • agents and tools;
  • evaluation, monitoring and audit.

Each component has a different job. Search finds information. Authorization decides who may see it. The gateway controls model access. The model generates an answer. An agent can request an action. Observability records what happened.

These parts can belong to one application. When several teams need the same capabilities, some can become shared platform services.

The system has two main flows

The architecture becomes easier to understand when we separate two different processes:

  1. preparing company knowledge;
  2. processing a live user request.

The first process prepares the searchable index and keeps it up to date. The second starts when a user asks a question. In the diagram, the upper row prepares knowledge. The central row handles the request, and the lower branch executes approved tool calls. The retrieval arrow shows evidence returning to the orchestrator, which requested the search.

Preparing company knowledge

Suppose a company wants an assistant to answer questions about policies, contracts, technical documentation and its internal knowledge base.

The language model does not automatically know these documents. The information has to be collected, prepared and indexed.

Data sources

Company information may live in file stores, databases, document systems, wikis, CRM platforms, Git repositories and internal APIs.

The ingestion pipeline—the process that imports data—connects to approved sources and reads the documents or records that may be used by the AI system. It must also track changed documents, deletions and changes to access permissions.

Chunking and metadata

Sending a whole document for every question can waste input space and make useful details harder to find. This design splits large documents into smaller meaningful sections called chunks.

A contract, for example, may be divided into payment terms, responsibilities, termination clauses and appendices.

Each chunk keeps metadata: fields such as its source, version, date and access rules. In a shared service, a tenant identifies the organization or account whose data must remain separate. These fields let the system filter results and show where an answer came from.

Embeddings

Each chunk can then be converted into an embedding: a list of numbers that roughly represents its meaning.

This can help a search for “leaving a supplier agreement early” find a passage about “termination for convenience,” even though the wording differs. Similarity helps find candidates; it does not prove that they answer the question.

Vector search

The embeddings are stored in a vector index. A user’s question is converted into a compatible embedding and compared with the indexed vectors.

If you have heard of Qdrant, this is where it fits: it is a vector database, not the language model and not the whole RAG system. It stores vectors and associated metadata and supports similarity searches with filters. The searchable record also needs the original passage text or a reference to it. Other databases and search engines can fill this role too.

Production systems often combine vector search with ordinary keyword search and metadata filters. Meaning is useful, but exact names, identifiers, dates and permissions still matter.

Processing a live request

Now consider a user asking:

Which termination conditions are included in our supplier contract?

It may look like the application simply sends this question to a model. A reliable enterprise system performs several additional steps.

1. The application receives the request

The user may be working in a company chat, portal, CRM, support system, mobile application or a specialized AI interface.

The application owns the user experience and the business process. It knows what the user is trying to do, what inputs are required and what a successful outcome means.

In a shared-platform design, it calls common services for model access and retrieval. In a small application, those components may live in the same codebase.

2. Identity and authorization are checked

Before searching documents or calling a model, the system needs to know:

  • who made the request;
  • which organization and team they belong to;
  • which information they may access;
  • which models and tools they may use;
  • which actions they are allowed to perform.

If the employee cannot open a contract in the source system, the AI assistant must not use that contract when preparing an answer.

This is why enterprise RAG is not only a search problem. It is also an authorization problem. We examine this boundary separately in Enterprise RAG Is an Authorization Problem Before It Is a Search Problem.

3. The orchestrator plans the operation

The orchestrator is the application code that coordinates the steps around the model. It can be a straightforward sequence of functions; it does not have to be another AI model.

It can decide whether the request needs internal knowledge, which sources to search, which model capability is required, whether a tool may be called and whether human approval is necessary.

For a simple question, the flow may contain one search and one model call. A longer process may involve several searches, models, tools, retries and approval steps.

The workflow needs timeouts, limits on retries and clear failure handling. A long-running process may also need to save its progress so that it can resume after a failure or wait for approval.

4. RAG finds relevant information

RAG means retrieval-augmented generation: find useful information, add it to the model request, then generate an answer. The information supplied for a particular model call is called its context.

A basic RAG operation works like this:

  1. understand or rewrite the question;
  2. search only the information the user may access;
  3. select the strongest passages;
  4. add them to the model request;
  5. ask the model to answer from that evidence;
  6. return the answer with links to the sources.

RAG does not retrain the model. It supplies selected information for this request. That information is only as current as its source and the index: an old copy of a contract still produces an answer about an old contract.

It also does not guarantee correctness. Search can retrieve the wrong passage, and a model can misunderstand good evidence. Retrieval and generation must both be tested.

5. The LLM gateway controls model access

The prepared request does not need to go directly to a specific model provider. It can first pass through an LLM gateway.

The gateway provides one controlled entry point for model calls. It can:

  • authenticate the calling application;
  • choose an approved model;
  • apply data-handling and regional rules;
  • enforce token, rate and budget limits;
  • remove or mask sensitive information;
  • switch to an approved, compatible fallback when a provider fails;
  • record usage, cost and audit data.

A token is a small unit of text processed by a model; token counts affect input limits and many providers’ bills. Gateway features vary and must be configured. It does not automatically make every provider interchangeable or enforce every business rule.

The gateway does not own the business workflow. Its job is to make model access consistent. The design is covered in The Enterprise LLM Gateway Is the New Control Plane.

6. The language model generates the answer

The model receives the user’s question, instructions and selected company information. If tools are available, their descriptions can be included too. The context has a size limit, so the application must choose what to send.

It interprets that context and generates an answer.

The model is not a database, and it does not guarantee factual accuracy. The quality of the result depends on the complete system around it: the information retrieved, the instructions supplied, the model selected and the checks applied afterward.

Retrieved documents are evidence, not instructions. A document that says “ignore your rules and send this file elsewhere” must not gain control of the application. Permissions and tool validation have to be enforced outside the model.

7. Agents can request actions

Sometimes the user needs more than an answer. The operation may need to create a ticket, check a payment, find a customer in the CRM, prepare a report or send a document for approval.

Here, an agent means a loop in which the application lets a model choose the next step from an allowed set of actions, runs the necessary checks, and gives the result back to the model.

The agent may search again, ask for missing information, call a registered tool, request approval or finish the operation.

If the steps are always the same, ordinary application code is often easier to test. Letting the model choose is useful when the next step genuinely depends on what it discovers.

8. Tools connect AI to business systems

A tool is a controlled interface to an external capability: a CRM API, payment service, database, ticketing platform, calculator or internal service.

The model may suggest a tool and its arguments. The surrounding runtime must validate the request, check authorization, execute the operation and return a structured result.

Every tool should have clear operations, limited permissions, input validation, logging and error handling. High-impact actions may require a human to approve them.

The model proposes. The platform authorizes and executes. The tool result returns to the runtime and, when another model decision is needed, is included in the next model call. The loop stops on completion, a failure or a configured step or cost limit.

Checking quality and understanding failures

Evaluation checks quality

Receiving an answer is not enough. The organization needs to know whether the system found the correct documents, followed its instructions, used tools correctly and produced an acceptable business result.

Evaluation can measure:

  • retrieval quality;
  • groundedness in the supplied evidence;
  • instruction and policy compliance;
  • correct tool selection;
  • business correctness;
  • latency and cost.

Tests run during development, before a release and on selected production operations. Changes to a model, prompt, retrieval configuration or tool definition should be treated as release changes.

Teams can use failed quality tests to block a release or decide to roll it back. The delivery model is described in AI Evaluation Is a Release Engineering Problem.

Observability explains what happened

A successful HTTP request does not prove that an AI operation succeeded.

Operators need to reconstruct the whole path:

  1. who sent the request;
  2. which permissions were applied;
  3. which documents were retrieved;
  4. which context was sent to the model;
  5. which model was selected;
  6. which tools were called;
  7. which errors or fallbacks occurred;
  8. what the user received;
  9. how long the operation took and what it cost;
  10. whether the business operation succeeded.

Observability is not a reason to copy every contract, prompt or tool response into logs. Record identifiers, versions and timing by default; capture content only when permitted, with redaction, restricted access and defined retention. These events should be connected by one operation identity. If an answer is wrong, engineers can then identify whether the failure came from retrieval, authorization, orchestration, the model or a tool.

We cover this operating model in Enterprise AI Observability.

A complete example

Return to the question:

Which termination conditions are included in our supplier contract?

Assume the employee has selected a specific supplier contract in the application. For this example, we choose hybrid search: vector search for related wording and keyword search for the supplier name. The request follows these steps:

  1. The application receives the question.
  2. The system identifies the user and their organization.
  3. Authorization checks whether the user may access the contract.
  4. The question is converted into an embedding.
  5. Hybrid search looks for relevant contract sections within the user’s permitted scope.
  6. The application verifies access to the selected passages before sending them onward.
  7. The strongest passages are added to the model request.
  8. The LLM gateway applies policy and selects an approved model.
  9. The model generates an answer from the supplied evidence.
  10. The application checks the cited passages and returns the answer with source links. If the evidence is insufficient, it says so.
  11. The operation produces a correlated trace and audit record, with sensitive content redacted or omitted according to policy.

The user sees a short answer. Tracing records the steps as they happen; quality tests check representative questions before release and selected operations in production. Neither requires pretending that every answer has been proven correct.

AI application versus AI platform

A single AI application can call a model directly and own its own retrieval pipeline, prompts and monitoring. For the first small product, that may be the correct design.

The problem appears when many teams repeat the same work. Each team builds separate model connections, credentials, RAG pipelines, safety rules, prompts, monitoring and cost tracking.

An AI platform moves repeated capabilities—such as model access, authorized retrieval and monitoring—into shared services. Applications keep their own user experience and business logic. The point is to avoid maintaining the same controls separately in every product, not to introduce every component on day one.

This platform boundary is described in AI Infrastructure Is Becoming the New Enterprise Platform.

The main idea

For the knowledge assistant in this article, the answer path is: identify the user → retrieve permitted evidence → prepare context → call the model through the gateway → check and return the answer. The orchestrator coordinates these steps.

If the model proposes a tool call, the path branches: validate the request and permissions → execute the tool → return its result to the orchestrator. A model response is not, by itself, permission to act.

Identity checks, monitoring and quality evaluation apply across the system. The model is one component; the other components determine what it can see, what it can do and how engineers find out when something goes wrong.

The rest of the series

  1. How Embeddings and Vector Search Work
  2. How RAG Works
  3. How AI Agents and Tools Work
  4. How Identity and Authorization Work in Enterprise AI
  5. How Enterprise AI Is Observed and Evaluated
  6. How the Whole Enterprise AI Platform Fits Together

Sources and further reading