An employee asks an AI assistant to explain a contract clause and create a follow-up task. The answer may require company knowledge, a language model and a business API. It also requires identity, authorization, workflow state, audit records and a way to understand failures.

Those capabilities are often discussed separately. In production, they have to work as one system.

This final article in the series puts the pieces together. The goal is not to prescribe one product stack. It is to show where each responsibility belongs, which parts can be shared and which decisions must stay with the application team.

Open the enterprise AI platform diagram at full size.

The diagram is a map, not a checklist

A small classification service may call one model and never need RAG, an agent or a separate gateway. A knowledge assistant may need retrieval but no tools. A customer-service workflow may use retrieval, models, tools and human approval.

The platform should provide the capabilities a workload needs without forcing every application through unnecessary components. The layers in the diagram describe responsibilities. They do not require a separate product, service or team for every box.

The architecture has four horizontal layers:

1. business applications;
2. the AI runtime and enforcement layer;
3. AI capability services;
4. the platform foundation.

Operations and governance apply across all four.

The application plane owns the business outcome

The application plane contains the products people actually use: copilots, portals, support workflows, internal services and industry-specific applications.

This layer knows why the request exists. It owns the user experience, the business rules and the definition of success.

For a contract assistant, the application decides which contract the employee is working with, whether the user wants an explanation or an action, and what should happen when evidence is insufficient. For a claims workflow, it knows the claim state, required documents and escalation rules.

A shared AI platform should not take ownership of those domain decisions. It cannot decide whether a payment should be released, a customer should receive credit or a legal interpretation is acceptable. It supplies reusable technical controls; the application remains accountable for the result.

The AI runtime coordinates and constrains the request

The runtime and enforcement layer sits between applications and the underlying knowledge, models and tools. It handles live requests and answers three different questions:

• What steps should this operation perform?
• What is this caller allowed to see and do?
• Which approved model endpoint should process the request?

Orchestration owns the workflow

The orchestrator coordinates the operation. It can retrieve information, assemble model context, request a model response, validate a proposed tool call, wait for approval and handle failures.

It may be ordinary application code, a workflow engine or an agent runtime. If the sequence is predictable, explicit code is usually easier to test. A model-directed loop is useful when the next step genuinely depends on what the system discovers.

The orchestrator also owns timeouts, retry limits and workflow state. A model should not decide how many times an expensive operation may retry or whether a failed payment action is safe to repeat.

Identity and policy define the permitted scope

Identity connects the request to a user, service, organization and tenant. Policy turns that identity and the current business context into permissions.

These checks have to reach the resources being used. A user who cannot read a document in the source system should not receive it through RAG. A model that proposes a tool call does not grant permission to execute it.

Authorization can be enforced at several points: before retrieval, while filtering search results, before including evidence in model context and before executing a tool. The exact design depends on the data system and risk, but the final enforcement cannot live only inside a prompt.

We covered this boundary in How Identity and Authorization Work in Enterprise AI.

The LLM gateway controls model access

The gateway gives applications a governed route to approved model endpoints. It can authenticate callers, route requests, enforce rate and token limits, attribute usage, apply provider-specific controls and record model access.

The gateway does not replace the orchestrator. It knows how to control a model call; it does not know whether the business workflow should issue that call or what should happen afterward.

“Control plane” is often used informally for this shared governance boundary. In stricter infrastructure terminology, the proxy instances handling prompts and responses are part of the runtime or data plane; controllers and configuration APIs form the control plane. Keeping that distinction clear matters because configuration can be temporarily unavailable while an appropriately designed runtime continues serving existing routes.

It is also a failure domain. If every model call passes through one gateway, the gateway needs appropriate capacity, deployment safety and regional design. Central control is useful only when it is operated as production infrastructure.

The pattern is examined in The Enterprise LLM Gateway Is the New Control Plane.

AI capability services supply knowledge, models and actions

The next layer contains three different kinds of capability. Keeping them separate makes the architecture easier to reason about.

Knowledge services retrieve authorized evidence

Knowledge services connect approved sources to the live request. They may contain ingestion pipelines, document parsing, chunking, embeddings, keyword and vector indexes, ranking and source references.

Retrieval should return evidence with enough metadata to verify the source, version and access scope. It should also be able to say that it found nothing useful. A weak result should not be disguised as a confident answer.

How Embeddings and Vector Search Work explains the search layer. How RAG Works follows that evidence into a model request.

Model services provide inference

Models interpret the supplied context and generate a response, classification, extraction or tool request. They may be hosted by a provider, deployed in a private environment or selected from several approved backends.

A model is not the owner of company data, permissions or business transactions. It receives a bounded request and returns an output. Model and prompt versions still matter because changing either can change system behaviour even when the surrounding code stays the same.

Tools expose controlled business operations

Tools connect AI workflows to ticketing systems, CRM platforms, payment services, databases and internal APIs.

The model can propose a tool and arguments. The runtime validates the structure, checks permissions, applies limits and executes the operation. The tool returns a structured result that the workflow can verify.

High-impact operations may require approval or a deterministic rule. Idempotency matters too: retrying “read customer” is different from retrying “send payment.”

The full execution loop is described in How AI Agents and Tools Work.

The platform foundation keeps everything running

Under the AI-specific components is ordinary production infrastructure: compute, networking, storage, secrets, queues, deployment pipelines and service monitoring.

This layer is easy to overlook because it is not unique to AI. It is also where many production failures occur.

The platform needs to handle unavailable providers, slow retrieval, exhausted quotas, expired credentials, partial tool failures and unsafe configuration changes. Long-running workflows may need durable state. Sensitive services may need private networking and regional data boundaries.

AI does not remove the need for reliability engineering. It adds new dependencies and more variable behaviour to systems that still have to meet operational expectations.

Operations and governance cross every layer

Telemetry, evaluation, audit, cost controls and release management are shown as a vertical block because they are not one final step after the answer. They apply across the request.

A useful operation record can connect:

• the application and release that received the request;
• the user, service and policy decision;
• the document and index versions used for retrieval;
• the prompt, model and gateway route;
• the tools requested and actually executed;
• the response, business outcome, latency and attributed cost.

That does not mean storing every prompt and document in general-purpose logs. Sensitive content should be minimized, redacted where appropriate, access-controlled and retained for a defined period.

Evaluation checks whether the system meets defined expectations. Observability helps locate the responsible step when it does not. Prompt, model, retrieval and tool changes should be versioned and evaluated like release changes, with a way to stop or roll back a bad version.

See How Enterprise AI Is Observed and Evaluated for the operating loop.

Two flows make the platform easier to understand

The complete platform contains two connected flows.

The knowledge flow

Sources are ingested, parsed, divided into useful sections and indexed with metadata and access information. Changes, deletions and permission updates have to propagate. The output is a governed knowledge service that can retrieve current evidence for an authorized request.

This flow runs before the user asks a question and continues as source data changes.

The request flow

A live request starts with identity and business context. The orchestrator decides which capabilities are needed. It may retrieve permitted evidence, call an approved model through the gateway and execute validated tools. The application returns the answer or business outcome while telemetry records the operation.

The request flow consumes the output of the knowledge flow. Treating them as one undifferentiated “AI pipeline” makes freshness, permission and failure problems harder to diagnose.

Follow one operation end to end

Consider the request: “Explain the renewal clause in this supplier agreement and create a task 90 days before the deadline.”

1. The application identifies the selected agreement and the requested outcome.
2. Identity and policy confirm that the employee may read the agreement and create a task in the selected project.
3. The orchestrator asks the knowledge service for the relevant agreement sections and amendments.
4. Retrieval returns authorized passages with source and version references.
5. The orchestrator builds the model request from the question, instructions and evidence.
6. The gateway applies model policy and routes the call to an approved endpoint.
7. The model explains the clause and proposes task details.
8. Application code validates the date, project, owner and permission. If policy requires approval, the workflow waits.
9. The tool creates the task and returns the task identifier.
10. The application reports the explanation and confirmed action. Telemetry links retrieval, model and tool records to the same operation.

If the task is not created, the system should not claim that it was. If the amendment is missing, the system should not invent the current deadline. Each boundary turns an ambiguous AI response into an inspectable software operation.

What should be shared?

Sharing becomes useful when several applications need the same capability and control.

Common candidates include model access, credential management, usage attribution, telemetry conventions, evaluation infrastructure, tool registration and reusable retrieval services. Applications can still keep workload-specific prompts, workflow logic, acceptance criteria and user experience.

Do not centralize a component merely because it appears in the diagram. A shared service creates its own API contract, capacity requirements, failure domain and operating cost. Centralize when the consistency and reduced duplication are worth those responsibilities.

A practical build order

Start with one real workload and its acceptance criteria.

1. Define the user, business outcome and prohibited outcomes.
2. Implement the smallest explicit request flow that can meet them.
3. Enforce identity and authorization at data and tool boundaries.
4. Add the knowledge, model and tool capabilities the workload actually needs.
5. Record enough evidence to reconstruct failures without collecting unnecessary sensitive content.
6. Create evaluation cases from expected behaviour and real failures.
7. Extract shared platform services only when repeated demand and ownership are clear.

This avoids two common mistakes: building a large platform before any workload proves what is needed, and letting every application create incompatible controls that later become difficult to govern.

The whole system in one sentence

The application defines the outcome; the runtime coordinates the request and enforces configured controls; knowledge, models and tools provide capabilities; the foundation runs them; and operations and governance make the result observable, testable and accountable.

That is the enterprise AI platform. The language model is important, but it is only one service inside the system.

Enterprise AI Systems, Explained

1. How Enterprise AI Systems Work
2. How Embeddings and Vector Search Work
3. How RAG Works
4. The Enterprise LLM Gateway Is the New Control Plane
5. How AI Agents and Tools Work
6. How Identity and Authorization Work in Enterprise AI
7. How Enterprise AI Is Observed and Evaluated
8. How the Whole Enterprise AI Platform Fits Together

Sources and further reading

• Microsoft Azure Well-Architected Framework: Architecture pattern for AI workloads
• Microsoft Azure Architecture Center: Design and develop a RAG solution
• Microsoft Azure Architecture Center: Access language models through a gateway
• AWS Prescriptive Guidance: Agentic AI architecture in the enterprise
• NIST AI Risk Management Framework Core

The post How the Whole Enterprise AI Platform Fits Together appeared first on Arcentra Systems.

Open the RAG diagram at full size.

Suppose an engineer asks an internal AI assistant:

What changed in the payment service before yesterday’s incident?

Assume the language model was not trained on yesterday’s deployment record, the latest runbook or the incident timeline. It cannot reliably supply those facts from what it learned during training.

A retrieval-augmented generation system searches approved company sources for relevant evidence, adds it to the model’s context—the information supplied for this request—and asks the model to answer from what was found.

That is RAG: retrieve first, augment the request with evidence, then generate an answer.

The basic idea is simple. The engineering work is in deciding what may be retrieved, which passages are relevant, how they are presented to the model, and how the resulting answer is checked.

In the diagram, the upper row prepares searchable knowledge; the lower row handles a user request. Query preparation is grouped into retrieval. Optional reranking, omitted from the overview, belongs after the access check and before context assembly. Permissions already constrain retrieval; the separate access check adds another safeguard.

What RAG is—and what it is not

Retrieval-augmented generation connects a language model to external information at request time. That information can come from documents, databases, search indexes, ticketing systems, product catalogs or other enterprise sources.

The 2020 paper that introduced the term described two sources of information: knowledge stored in the model’s learned weights, called parametric memory, and an external collection searched when needed, called non-parametric memory. In operational terms:

• the model provides language understanding and generation;
• the retrieval system provides evidence for the current request.

RAG does not retrain the model whenever a document changes. It does not permanently insert company facts into model weights. It supplies selected information in the request context, much like giving an engineer the relevant pages before asking for an explanation.

RAG can use different storage and search technologies. Vector search is common, but keyword search, SQL queries, knowledge graphs and external APIs can also supply information for the model request.

Two flows: preparing knowledge and answering questions

The architecture shown here contains two distinct flows:

1. Knowledge preparation runs as enterprise data changes.
2. Request processing runs every time a user asks a question.

The quality of the answer depends on both flows. If the knowledge pipeline is stale, incomplete or insecure, the model receives bad evidence even when the request is carefully written.

Flow one: preparing company knowledge

1. Connect approved sources

The knowledge pipeline reads from explicitly approved sources: document repositories, wikis, support systems, databases, object storage or internal APIs.

Each connector needs an owner, credentials, synchronization rules and a failure path. The system must detect new, changed and deleted records, as well as changes to access permissions. Those changes must reach the search index and any response caches, so an old copy cannot continue granting access.

2. Parse the source correctly

A PDF is not automatically clean text. Documents contain headings, tables, footnotes, diagrams, repeated headers and scanned pages. Parsing determines which of those structures survive into retrieval.

If a table loses its row labels or a procedure loses its section heading, the index may contain words without the context required to interpret them.

3. Divide content into useful passages

Large documents are normally split into smaller passages called chunks. Small chunks can match a precise question; larger chunks preserve more context. The correct boundary depends on the content.

A fixed character count is easy to implement, but structure-aware chunking is often more useful. A contract clause, troubleshooting step and source-code function should not be divided in the same way.

4. Attach metadata and access attributes

Each passage should retain metadata: fields describing its source, version and access rules. Include the source identifier, modification date and authorization attributes, plus fields such as owner, language or jurisdiction when the application needs them. In a shared service, a tenant identifies the organization or account whose data must remain isolated.

Metadata lets the request flow select current information, enforce exact filters and return citations to the original record.

5. Embed and index

When using vector search, the pipeline converts each passage into an embedding: a list of numbers that represents features of its content for comparison. It stores these vectors in a search index alongside the passage text or a reference to it. A text index can also support searches for specific words, identifiers and error messages.

At search time, the query is converted into an embedding compatible with the stored passage vectors. This often uses the same model, with query and document settings recommended by its provider. Comparing these vectors helps locate related content. RAG then uses the retrieved content to help generate an answer.

Flow two: answering a request

1. Receive the question with identity and context

The application sends the question together with the authenticated identity, tenant, conversation state and relevant application context.

Identity must arrive before retrieval. If the system searches globally and filters sensitive passages only after they reach the model, the security boundary is already in the wrong place.

2. Prepare the query

A user’s wording is not always a good search query. The orchestrator, the application component that coordinates these steps, may correct spelling, expand an abbreviation, identify an exact product code or split a complex question into smaller searches.

Query rewriting must remain traceable. Operators should be able to see the original question, the rewritten queries and the sources each query returned.

3. Retrieve a broad candidate set

The retriever searches one or more eligible sources and returns candidate passages: results that might help answer the question. Tenant and access constraints restrict this search; the later access check adds a second safeguard. A common pattern is hybrid retrieval, which combines vector and keyword search:

• vector search finds passages with similar meaning;
• keyword search finds exact names, codes and phrases;
• metadata filters restrict the search space.

The goal of this stage is recall: avoid missing evidence that may answer the question. The first result list can therefore contain passages that are related but not precise enough.

4. Enforce permissions

The platform verifies that the requesting identity may use every candidate under the current permissions. Unauthorized content must be removed before it is returned to the user or sent to another service, including an external ranking service or generation model. Implementations can restrict the search in advance, use separate indexes for different access groups, or filter results using access attributes. If the platform cannot verify access, it must stop rather than pass the content onward.

A relevant document is not necessarily an authorized document. This boundary is examined in Enterprise RAG Is an Authorization Problem Before It Is a Search Problem.

5. Rerank for precision

Vector similarity measures general relatedness. Keyword scores measure term matches. Neither necessarily tells us whether a passage answers this exact question.

A reranker evaluates the query and candidate passages together, then places the most useful evidence first. The aim is precision: a larger share of the selected passages should help answer the question. Microsoft describes this balance between recall and precision. AWS also supports reranking before generation.

Reranking adds latency—the time a user waits for a response—and cost. Measure whether the improvement justifies that overhead for the workload.

6. Assemble the context

The orchestrator selects the final passages and builds the model request, or prompt. A typical RAG prompt contains:

• system instructions;
• the user’s question;
• the retrieved passages with source identifiers;
• rules for conflicting or missing evidence;
• the required answer and citation format.

Irrelevant passages dilute the useful evidence and use part of the model’s limited input space. Context assembly should select enough relevant material to answer the question while preserving any conditions needed to interpret it.

7. Generate the answer

The language model receives the assembled prompt and produces an answer. The instructions should tell it to use the supplied evidence, identify uncertainty and refuse to invent missing details.

The model is still generating text probabilistically. RAG improves the information available to it; RAG does not turn generation into a deterministic database lookup.

8. Return citations and trace the operation

The application maps statements or answer sections back to the supplied source passages. A citation should identify evidence supplied to the model, not merely attach a plausible-looking document after generation.

Check both that a citation points to a supplied passage and that the passage supports the associated claim. A valid source identifier alone does not make the claim correct.

Correlate the request identity, source identifiers, applied filters, ranking decisions, prompt and model versions, timing and outcome. Do not log full documents, prompts or answers by default. Capture content only when permitted, with redaction, restricted access and defined retention; source versions or protected references can support investigation without duplicating sensitive data in ordinary logs.

A small worked example

Return to the payment-service question. This is a fictional teaching example; the records and values below are invented.

Suppose the engineer is allowed to read two records:

• Deployment record D1: “At 14:05, the payment service’s request timeout changed from 2 seconds to 5 seconds.”
• Incident timeline I1: “At 14:12, the payment service’s error rate began to rise.”

Retrieval finds these passages. The application checks access and supplies their text and source identifiers to the model. A supported answer would be:

The request timeout increased from 2 seconds to 5 seconds at 14:05 [D1], seven minutes before the recorded rise in errors [I1]. These records establish the sequence of events, but do not establish that the timeout change caused the incident.

The labels refer to the invented records above. If the deployment record is unavailable, the assistant should say that it cannot identify the change from the available evidence.

Retrieved documents are data, not instructions

A document can be authorized for a user and still contain malicious instructions. A retrieved page might tell the assistant to ignore its rules, reveal other context or call a tool. This is indirect prompt injection.

Treat retrieved text as untrusted evidence, not as authority to change the workflow. Separate instructions from source content, test adversarial documents, and enforce tool permissions and input validation outside the model. A prompt alone is not a security boundary. High-impact actions may also need explicit approval. See the OWASP RAG Security guidance.

Why RAG can still produce a wrong answer

A hallucination is an incorrect or unsupported claim presented as fact. RAG gives the model better evidence and makes answers more traceable, but it cannot eliminate these errors.

The final result can still fail at several stages:

• the required document was never indexed;
• parsing removed important structure;
• chunking separated a statement from its conditions;
• retrieval missed the correct passage;
• permission filters removed too many candidates—or too few;
• reranking selected a related but incorrect source;
• the prompt did not explain how to handle conflicting evidence;
• the model ignored or misread the supplied context;
• citations were attached to claims they do not support.

“The model hallucinated” is therefore not a sufficient incident diagnosis. The failure may have occurred before the model was called.

Standard RAG and agentic RAG

Standard RAG follows a designed sequence: accept the question, run a known search, assemble context and call the model. This works well when a request maps to a predictable source and a single retrieval step.

Agentic RAG treats retrieval as a tool. An agent uses a model to choose its next step: which source to search, whether to break a question into smaller searches, or whether to retrieve again after inspecting intermediate results.

That flexibility is useful for complex tasks, but it also adds variable cost, latency and behavior. A fixed pipeline is often the better starting point. Agentic retrieval should be introduced when the workload requires dynamic decisions, not because it sounds more advanced.

Evaluate retrieval and generation separately

End-to-end answer quality alone does not show where the system is improving or failing.

Retrieval evaluation asks:

• Did the system find the required passage?
• How much of the returned context was relevant?
• Did filters preserve all authorized evidence?

Generation evaluation asks:

• Is the answer correct and complete?
• Is it faithful to the retrieved context?
• Do the citations support the claims?
• Does the system refuse when evidence is insufficient?

AWS evaluates retrieval and generation separately. Context relevance asks whether the returned passages relate to the question; context coverage asks whether they include the information in a prepared reference answer. Faithfulness checks whether the generated answer stays within the retrieved evidence. Citation precision checks whether citations are used correctly, while citation coverage checks whether the answer has the citations it needs. Context coverage requires reference information in the test dataset. These measures help diagnose different failures; none alone proves an answer is correct.

A practical production checklist

Before calling a RAG system production-ready, verify that the team can answer these questions:

• Which sources are approved, and who owns them?
• How are content updates, deletions and permission changes propagated?
• How are tables, images and document structure parsed?
• How are chunk boundaries chosen and versioned?
• Which identity and policy restrict each search?
• When are keyword, vector or structured queries used?
• How are candidates merged and reranked?
• What happens when sources conflict or provide no answer?
• Can each claim be traced to evidence the model received?
• Can operators reconstruct the entire request?
• Which test set measures retrieval and answer quality?
• What latency, cost and failure budgets apply?

Where RAG fits in the enterprise AI system

RAG is not the whole AI platform. It is the evidence path between enterprise knowledge and model generation.

The surrounding platform still owns identity, model access, policy, secrets, monitoring, evaluation, cost attribution and release control.

The essential runtime sequence is:

User and identity → query preparation → retrieval within access boundaries → current permission check → optional reranking → context assembly → model generation → answer and validated citations.

RAG supplies selected evidence for a request without changing the model’s learned weights. The reliability of the answer depends on the engineering of that entire evidence path.

Continue with How AI Agents and Tools Work, Part 4 of the series.

Sources and further reading

• Lewis et al.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Microsoft Azure Architecture Center: Design and develop a RAG solution
• Microsoft Azure Architecture Center: RAG information-retrieval phase
• Microsoft Azure Architecture Center: RAG prompt engineering
• AWS: How Amazon Bedrock Knowledge Bases work
• AWS: Evaluate the performance of RAG sources
• AWS: RAG evaluation metrics
• OWASP: RAG Security Cheat Sheet

The post How RAG Works appeared first on Arcentra Systems.

Open the enterprise AI diagram at full size.

When people talk about enterprise AI, the discussion often jumps straight to models: GPT, Claude, Gemini or a model running inside the company.

But the model is only one part of the system.

A large language model, or LLM, can interpret a request and generate text. By itself, it has no reliable access to the company’s current documents and cannot enforce user permissions or safely operate business systems.

To use AI in a real organization, engineers have to build a complete system around the model.

Take a simple question: “Which termination conditions are included in our supplier contract?” Something must find the right contract, check that the employee may read it, and give the relevant passages to the model. We will follow that path, then look at what changes when the assistant also needs to perform an action.

This is a reference design for a knowledge assistant that can also use business tools, not a mandatory checklist for every AI application. Classification, extraction or a small internal assistant may need only some of these components. A separate gateway, vector index and agent loop are architectural choices, not prerequisites.

What is an enterprise AI system?

In this series, an enterprise AI system is an application built around a language model, with the data access, permissions and operational controls needed for use inside an organization.

For this knowledge-and-tools reference design, the components include:

• user-facing applications;
• identity and access control;
• search over company data;
• RAG;
• an LLM gateway;
• one or more language models;
• agents and tools;
• evaluation, monitoring and audit.

Each component has a different job. Search finds information. Authorization decides who may see it. The gateway controls model access. The model generates an answer. An agent can request an action. Observability records what happened.

These parts can belong to one application. When several teams need the same capabilities, some can become shared platform services.

The system has two main flows

The architecture becomes easier to understand when we separate two different processes:

1. preparing company knowledge;
2. processing a live user request.

The first process prepares the searchable index and keeps it up to date. The second starts when a user asks a question. In the diagram, the upper row prepares knowledge. The central row handles the request, and the lower branch executes approved tool calls. The retrieval arrow shows evidence returning to the orchestrator, which requested the search.

Preparing company knowledge

Suppose a company wants an assistant to answer questions about policies, contracts, technical documentation and its internal knowledge base.

The language model does not automatically know these documents. The information has to be collected, prepared and indexed.

Data sources

Company information may live in file stores, databases, document systems, wikis, CRM platforms, Git repositories and internal APIs.

The ingestion pipeline—the process that imports data—connects to approved sources and reads the documents or records that may be used by the AI system. It must also track changed documents, deletions and changes to access permissions.

Chunking and metadata

Sending a whole document for every question can waste input space and make useful details harder to find. This design splits large documents into smaller meaningful sections called chunks.

A contract, for example, may be divided into payment terms, responsibilities, termination clauses and appendices.

Each chunk keeps metadata: fields such as its source, version, date and access rules. In a shared service, a tenant identifies the organization or account whose data must remain separate. These fields let the system filter results and show where an answer came from.

Embeddings

Each chunk can then be converted into an embedding: a list of numbers that roughly represents its meaning.

This can help a search for “leaving a supplier agreement early” find a passage about “termination for convenience,” even though the wording differs. Similarity helps find candidates; it does not prove that they answer the question.

Vector search

The embeddings are stored in a vector index. A user’s question is converted into a compatible embedding and compared with the indexed vectors.

If you have heard of Qdrant, this is where it fits: it is a vector database, not the language model and not the whole RAG system. It stores vectors and associated metadata and supports similarity searches with filters. The searchable record also needs the original passage text or a reference to it. Other databases and search engines can fill this role too.

Production systems often combine vector search with ordinary keyword search and metadata filters. Meaning is useful, but exact names, identifiers, dates and permissions still matter.

Processing a live request

Now consider a user asking:

Which termination conditions are included in our supplier contract?

It may look like the application simply sends this question to a model. A reliable enterprise system performs several additional steps.

1. The application receives the request

The user may be working in a company chat, portal, CRM, support system, mobile application or a specialized AI interface.

The application owns the user experience and the business process. It knows what the user is trying to do, what inputs are required and what a successful outcome means.

In a shared-platform design, it calls common services for model access and retrieval. In a small application, those components may live in the same codebase.

2. Identity and authorization are checked

Before searching documents or calling a model, the system needs to know:

• who made the request;
• which organization and team they belong to;
• which information they may access;
• which models and tools they may use;
• which actions they are allowed to perform.

If the employee cannot open a contract in the source system, the AI assistant must not use that contract when preparing an answer.

This is why enterprise RAG is not only a search problem. It is also an authorization problem. We examine this boundary separately in Enterprise RAG Is an Authorization Problem Before It Is a Search Problem.

3. The orchestrator plans the operation

The orchestrator is the application code that coordinates the steps around the model. It can be a straightforward sequence of functions; it does not have to be another AI model.

It can decide whether the request needs internal knowledge, which sources to search, which model capability is required, whether a tool may be called and whether human approval is necessary.

For a simple question, the flow may contain one search and one model call. A longer process may involve several searches, models, tools, retries and approval steps.

The workflow needs timeouts, limits on retries and clear failure handling. A long-running process may also need to save its progress so that it can resume after a failure or wait for approval.

4. RAG finds relevant information

RAG means retrieval-augmented generation: find useful information, add it to the model request, then generate an answer. The information supplied for a particular model call is called its context.

A basic RAG operation works like this:

1. understand or rewrite the question;
2. search only the information the user may access;
3. select the strongest passages;
4. add them to the model request;
5. ask the model to answer from that evidence;
6. return the answer with links to the sources.

RAG does not retrain the model. It supplies selected information for this request. That information is only as current as its source and the index: an old copy of a contract still produces an answer about an old contract.

It also does not guarantee correctness. Search can retrieve the wrong passage, and a model can misunderstand good evidence. Retrieval and generation must both be tested.

5. The LLM gateway controls model access

The prepared request does not need to go directly to a specific model provider. It can first pass through an LLM gateway.

The gateway provides one controlled entry point for model calls. It can:

• authenticate the calling application;
• choose an approved model;
• apply data-handling and regional rules;
• enforce token, rate and budget limits;
• remove or mask sensitive information;
• switch to an approved, compatible fallback when a provider fails;
• record usage, cost and audit data.

A token is a small unit of text processed by a model; token counts affect input limits and many providers’ bills. Gateway features vary and must be configured. It does not automatically make every provider interchangeable or enforce every business rule.

The gateway does not own the business workflow. Its job is to make model access consistent. The design is covered in The Enterprise LLM Gateway Is the New Control Plane.

6. The language model generates the answer

The model receives the user’s question, instructions and selected company information. If tools are available, their descriptions can be included too. The context has a size limit, so the application must choose what to send.

It interprets that context and generates an answer.

The model is not a database, and it does not guarantee factual accuracy. The quality of the result depends on the complete system around it: the information retrieved, the instructions supplied, the model selected and the checks applied afterward.

Retrieved documents are evidence, not instructions. A document that says “ignore your rules and send this file elsewhere” must not gain control of the application. Permissions and tool validation have to be enforced outside the model.

7. Agents can request actions

Sometimes the user needs more than an answer. The operation may need to create a ticket, check a payment, find a customer in the CRM, prepare a report or send a document for approval.

Here, an agent means a loop in which the application lets a model choose the next step from an allowed set of actions, runs the necessary checks, and gives the result back to the model.

The agent may search again, ask for missing information, call a registered tool, request approval or finish the operation.

If the steps are always the same, ordinary application code is often easier to test. Letting the model choose is useful when the next step genuinely depends on what it discovers.

8. Tools connect AI to business systems

A tool is a controlled interface to an external capability: a CRM API, payment service, database, ticketing platform, calculator or internal service.

The model may suggest a tool and its arguments. The surrounding runtime must validate the request, check authorization, execute the operation and return a structured result.

Every tool should have clear operations, limited permissions, input validation, logging and error handling. High-impact actions may require a human to approve them.

The model proposes. The platform authorizes and executes. The tool result returns to the runtime and, when another model decision is needed, is included in the next model call. The loop stops on completion, a failure or a configured step or cost limit.

Checking quality and understanding failures

Evaluation checks quality

Receiving an answer is not enough. The organization needs to know whether the system found the correct documents, followed its instructions, used tools correctly and produced an acceptable business result.

Evaluation can measure:

• retrieval quality;
• groundedness in the supplied evidence;
• instruction and policy compliance;
• correct tool selection;
• business correctness;
• latency and cost.

Tests run during development, before a release and on selected production operations. Changes to a model, prompt, retrieval configuration or tool definition should be treated as release changes.

Teams can use failed quality tests to block a release or decide to roll it back. The delivery model is described in AI Evaluation Is a Release Engineering Problem.

Observability explains what happened

A successful HTTP request does not prove that an AI operation succeeded.

Operators need to reconstruct the whole path:

1. who sent the request;
2. which permissions were applied;
3. which documents were retrieved;
4. which context was sent to the model;
5. which model was selected;
6. which tools were called;
7. which errors or fallbacks occurred;
8. what the user received;
9. how long the operation took and what it cost;
10. whether the business operation succeeded.

Observability is not a reason to copy every contract, prompt or tool response into logs. Record identifiers, versions and timing by default; capture content only when permitted, with redaction, restricted access and defined retention. These events should be connected by one operation identity. If an answer is wrong, engineers can then identify whether the failure came from retrieval, authorization, orchestration, the model or a tool.

We cover this operating model in Enterprise AI Observability.

A complete example

Return to the question:

Which termination conditions are included in our supplier contract?

Assume the employee has selected a specific supplier contract in the application. For this example, we choose hybrid search: vector search for related wording and keyword search for the supplier name. The request follows these steps:

1. The application receives the question.
2. The system identifies the user and their organization.
3. Authorization checks whether the user may access the contract.
4. The question is converted into an embedding.
5. Hybrid search looks for relevant contract sections within the user’s permitted scope.
6. The application verifies access to the selected passages before sending them onward.
7. The strongest passages are added to the model request.
8. The LLM gateway applies policy and selects an approved model.
9. The model generates an answer from the supplied evidence.
10. The application checks the cited passages and returns the answer with source links. If the evidence is insufficient, it says so.
11. The operation produces a correlated trace and audit record, with sensitive content redacted or omitted according to policy.

The user sees a short answer. Tracing records the steps as they happen; quality tests check representative questions before release and selected operations in production. Neither requires pretending that every answer has been proven correct.

AI application versus AI platform

A single AI application can call a model directly and own its own retrieval pipeline, prompts and monitoring. For the first small product, that may be the correct design.

The problem appears when many teams repeat the same work. Each team builds separate model connections, credentials, RAG pipelines, safety rules, prompts, monitoring and cost tracking.

An AI platform moves repeated capabilities—such as model access, authorized retrieval and monitoring—into shared services. Applications keep their own user experience and business logic. The point is to avoid maintaining the same controls separately in every product, not to introduce every component on day one.

This platform boundary is described in AI Infrastructure Is Becoming the New Enterprise Platform.

The main idea

For the knowledge assistant in this article, the answer path is: identify the user → retrieve permitted evidence → prepare context → call the model through the gateway → check and return the answer. The orchestrator coordinates these steps.

If the model proposes a tool call, the path branches: validate the request and permissions → execute the tool → return its result to the orchestrator. A model response is not, by itself, permission to act.

Identity checks, monitoring and quality evaluation apply across the system. The model is one component; the other components determine what it can see, what it can do and how engineers find out when something goes wrong.

The rest of the series

1. How Embeddings and Vector Search Work
2. How RAG Works
3. How AI Agents and Tools Work
4. How Identity and Authorization Work in Enterprise AI
5. How Enterprise AI Is Observed and Evaluated
6. How the Whole Enterprise AI Platform Fits Together

Sources and further reading

• Google Cloud: RAG infrastructure using Vector Search
• Microsoft: Access language models through a gateway
• Microsoft: Design a secure multitenant RAG solution
• AWS: Agentic AI architecture in the enterprise
• OpenTelemetry semantic conventions
• NIST AI Risk Management Framework

The post How Enterprise AI Systems Work appeared first on Arcentra Systems.

The first application rarely needs an LLM gateway.

A team can call a model provider directly, keep the API key in a secret store, add a timeout and ship. For one application, that is often the correct architecture. The trouble starts when the same pattern is repeated across a portfolio.

One application retries every 429 response. Another fails immediately. One sends prompts to a provider in a different region. Another records complete request bodies in its logs. A third has no way to attribute token spend to a tenant or workflow. Each integration works in isolation, but the enterprise has no common answer to a basic set of operational questions:

• Which application is making the request?
• Which models is it allowed to use?
• Where may the data be processed?
• What happens when the preferred model is unavailable?
• Who owns the budget?
• What evidence remains after the response is returned?

At that point, the problem is no longer model connectivity. It is control.

What is an enterprise LLM gateway?

An enterprise LLM gateway is a governed runtime boundary between AI applications and the models, inference endpoints and tools they use. Applications authenticate to the gateway and invoke a stable platform contract. The gateway evaluates policy, selects an approved backend, manages provider credentials, applies traffic controls and emits a consistent operational record.

That definition is deliberately narrower than “AI platform.” A gateway does not own the full lifecycle of enterprise knowledge, agent memory, evaluation datasets or business workflows. It sits in the request path and enforces the decisions that must be consistent across them.

The distinction matters. A proxy forwards traffic. A control plane determines the conditions under which traffic is permitted, where it goes and how the result is accounted for.

Why the gateway becomes a control plane

Traditional API gateways already provide authentication, quotas, routing and telemetry. LLM traffic adds a different set of constraints to the same architectural boundary.

A request has a monetary cost that is not known precisely until the response completes. Capacity is frequently expressed in tokens per minute rather than requests per second. Backends that expose similar APIs are not behaviorally interchangeable. Streaming changes timeout and retry behavior. Prompts and responses can contain sensitive business data. A fallback model may return valid JSON and still change the meaning of a business decision.

Current cloud implementations reflect this shift. Microsoft documents token limits, backend load balancing, circuit breakers, content controls and telemetry as AI gateway capabilities. AWS publishes a multi-provider gateway architecture that centralizes credentials, routing, guardrails, usage tracking and model access behind a unified interface. These are not merely SDK conveniences. They are shared operating controls.

The enterprise LLM gateway becomes the AI control plane when applications stop owning those controls individually.

The six decisions made on every request

1. Establish the caller

The gateway should know more than an API key. It needs an application identity, environment, tenant or business unit, and—where the use case requires it—the delegated identity of the user or agent initiating the request.

This identity becomes the basis for policy, quota and audit. Without it, rate limits collapse into one shared bucket and cost allocation becomes an exercise in inference.

2. Resolve the permitted capability

Applications should request a capability or a platform model alias, not an arbitrary provider model identifier. A contract such as reasoning-standard, document-extraction or low-latency-chat gives the platform room to change providers without forcing every application to understand the provider estate.

The alias must be versioned. Silently replacing the model behind a stable name is operationally convenient and behaviorally dangerous. Applications need to know whether a change is a capacity adjustment, an approved compatible revision or a new release requiring evaluation.

3. Evaluate policy

Policy answers whether this caller may use this capability with this class of data in this region. It may also decide whether input or output inspection is required, whether a request can leave a private network, and whether a human approval state must be present.

Policy should operate on explicit metadata. Asking the gateway to guess data sensitivity from the prompt is not a substitute for classification at the source. Content inspection can be a defense layer, but it should not become the primary authorization mechanism.

4. Select a backend

Routing is where many gateway designs become too clever. The safest production policy is usually deterministic: filter to approved backends, remove unhealthy or exhausted targets, then select according to a declared strategy.

Useful inputs include region, model capability, context length, reserved capacity, latency, recent error rate and unit cost. Semantic request classification can be added later, but it creates another model-dependent decision that must itself be observed and evaluated.

5. Enforce consumption limits

Request limits are insufficient for LLM workloads. Ten small classification calls and ten long-context generations are not equivalent. The gateway needs token-rate limits, budget scopes and usage attribution tied to the caller.

There is an unavoidable timing problem: final output-token usage is available only after generation. Pre-request estimation can reject obviously oversized prompts, while post-response accounting updates the actual budget. Under concurrency, a tenant may temporarily exceed a nominal limit. The system should document that behavior rather than pretend the counter is exact.

6. Produce evidence

Every request should leave a trace that can be joined to the application workflow. At minimum, operators need the caller, route, requested and resolved model, policy decision, latency, token usage, retry or fallback history, outcome and cost attribution key.

Prompt and response bodies require a separate retention decision. Recording everything creates a useful debugging archive and a serious data-governance problem. In many environments, metadata is retained by default while content capture is sampled, redacted or disabled for sensitive routes.

OpenTelemetry defines generative-AI attributes for operations, providers, requested and returned models, token usage, agents, tools and retrieval. Using a common telemetry vocabulary makes the gateway observable through the same systems that already operate the rest of the platform.

The gateway contract should be smaller than the provider APIs

A common mistake is to promise complete provider portability. Teams expose every parameter supported by every backend through one “unified” API. The result is a contract that is both lowest-common-denominator and permanently coupled to provider-specific extensions.

A stable enterprise contract should cover the semantics the organization is prepared to support: messages or input, a declared capability, structured output requirements, streaming behavior, tool definitions, request metadata and trace context. Provider-specific controls can exist behind an explicit escape hatch, but they should not leak into every application by default.

Portability is not achieved because two providers accept similar JSON. It is achieved when the organization defines which behaviors are contractual, tests them and controls changes.

Failover is a product decision, not only a routing decision

HTTP infrastructure encourages a simple mental model: if one backend fails, retry another. That is safe only when the backends are genuinely interchangeable for the task.

Models differ in instruction following, tool selection, refusal behavior, context handling and structured-output reliability. A fallback can preserve availability while degrading the business process in a way that does not produce an obvious infrastructure error.

Each route therefore needs an explicit failure contract:

• Which errors are retryable?
• Can a partially streamed response be retried?
• Is the operation idempotent?
• Which fallback models have passed the same evaluation suite?
• Should the application receive a degraded-mode signal?
• When must the system fail closed?

For a low-risk summarization workflow, a fallback model may be acceptable. For a regulated decision or an agent preparing a financial transaction, returning an explicit unavailable state may be the safer behavior.

What belongs in the gateway—and what does not

The gateway should own controls that are cross-cutting, enforceable in the request path and consistent across applications:

• caller authentication and application identity;
• provider credential brokering;
• model aliases and approved backend inventories;
• policy enforcement and regional constraints;
• rate limits, token quotas and budget attribution;
• timeouts, circuit breakers and controlled failover;
• standard telemetry, audit metadata and redaction hooks.

It should not become the place where every AI concern is centralized. Domain prompts, retrieval logic, business approvals, agent memory and workflow state belong closer to the applications and services that understand their semantics. Evaluation policy may be managed at platform level, but evaluation execution belongs in the delivery lifecycle rather than the synchronous request path.

A gateway that absorbs domain behavior becomes a distributed monolith with a particularly expensive hot path.

Control plane and data plane must be separated

The runtime gateway is the data plane. It processes requests under an already approved configuration. The management services around it form the control plane: model registry, route definitions, policy bundles, credential references, quota assignments and release history.

Separating them provides two operational advantages.

First, a control-plane outage does not need to stop inference. Gateway instances can continue operating from the last valid signed configuration. Second, configuration changes become deployable artifacts. A routing rule can be reviewed, tested, canaried and rolled back instead of being edited directly in a production console.

This is also the boundary at which enterprise governance becomes executable. A policy document may state that sensitive workloads must remain in an approved region. The control plane translates that requirement into an allowed backend set; the data plane enforces it on every request.

Operating the gateway as production infrastructure

The gateway sits on the critical path of every AI-enabled application. Its availability target should be higher than the target of any single model backend, and its failure modes should be intentionally boring.

That requires:

• stateless or carefully partitioned runtime instances;
• bounded retries with jitter and retry budgets;
• circuit breakers based on provider response semantics;
• load tests that include streaming and long-context requests;
• configuration validation before rollout;
• cardinality controls for telemetry labels;
• content logging disabled or redacted by default;
• capacity and quota dashboards by application, tenant and backend.

It also needs end-to-end traces. Gateway latency alone cannot explain a slow agent. Operators must correlate the model call with retrieval, tool execution and the business workflow around it. This is why an LLM gateway complements, rather than replaces, the broader production AI observability described in our production AI operations model.

A practical adoption sequence

Building the full control plane before the first use case is usually unnecessary. A safer sequence follows the controls that become shared first.

Stage 1: Establish the boundary

Move provider credentials out of applications. Introduce application identities, a stable endpoint, basic quotas and uniform request telemetry. Preserve simple deterministic routing.

Stage 2: Make routing and policy explicit

Add versioned model aliases, approved backend inventories, regional constraints, circuit breakers and tested fallback classes. Attribute usage to applications and business owners.

Stage 3: Operate it as a platform contract

Manage configuration through code and release gates. Connect route changes to evaluation evidence. Add self-service onboarding, budget delegation and policy packs for different workload classes.

The inflection point is the same one that turns isolated AI integrations into enterprise AI infrastructure: multiple applications require the same controls, and duplicating those controls is now riskier than operating them centrally.

Build, buy or extend an API gateway?

The answer depends less on feature count than on the operating environment.

Extending an existing API management layer can work when the organization already has strong identity, networking, policy and telemetry practices there. A purpose-built LLM gateway can accelerate multi-provider routing and token accounting. A managed cloud gateway can reduce operational burden when the model estate is concentrated in one platform.

None of these choices removes the architectural work. The organization still has to define its model contract, identity model, failure semantics, evidence requirements and ownership boundaries. Buying a gateway product before defining those decisions merely moves ambiguity into configuration.

The most durable design keeps applications dependent on an enterprise contract, not on the internal implementation of the gateway. That preserves the option to replace the runtime without rebuilding every AI application.

The control plane is the product

An enterprise LLM gateway is useful because it centralizes traffic. It becomes strategic when it centralizes decisions.

The durable asset is not the proxy process or the provider adapter. It is the governed contract around model access: who may call, what they may request, where execution may occur, how failures are handled, how consumption is attributed and what evidence is retained.

That contract is what allows an enterprise to change models without surrendering operational control. It is also what separates a portfolio of AI integrations from an AI platform that can be operated.

Sources and further reading

• AWS: Guidance for Multi-Provider Generative AI Gateway
• Microsoft: AI gateway capabilities in Azure API Management
• Microsoft: LLM token limit policy
• OpenTelemetry: Generative AI semantic attributes
• NIST AI Risk Management Framework

The post The Enterprise LLM Gateway Is the New Control Plane appeared first on Arcentra Systems.

In practice, a capable model is usually the easiest component to replace.

The difficult work begins when an AI system must operate inside the enterprise: use live data, preserve existing permissions, resolve inconsistent identifiers, call business APIs, survive partial failures and leave evidence that operators and auditors can reconstruct.

A model can be connected through an API in minutes. Connecting that model to a bank, fintech platform, retailer or industrial operator can take months because the integration must carry more than data. It must carry identity, authority, workflow state and evidence across systems that were not designed to participate in probabilistic execution.

That is the real enterprise AI integration problem.

What Is Enterprise AI Integration?

Enterprise AI integration is the engineering discipline that connects models and agents to corporate identities, data, applications, events and transaction systems through controlled, observable workflows.

It includes connectors and APIs, but it is not defined by their number. A production integration must preserve four properties across every boundary:

• Identity: who initiated the request, which service or agent is acting and on whose behalf.
• Authority: which records, tools and operations are permitted in the current context.
• State: what has already happened, which version of a record is current and whether a workflow is complete.
• Evidence: which sources, decisions, calls and outcomes must be retained for operations, audit and improvement.

If any one of these properties disappears between the AI application and a system of record, the workflow becomes unreliable even when the model response appears correct.

This is why enterprise AI integration requires architecture rather than a growing collection of point-to-point connectors. The model speaks in tokens. The enterprise operates through identities, records, transactions, events and approval rules. The integration layer is where those two environments are reconciled.

The Model Call Is One Step in a Distributed Transaction

An enterprise AI request should be treated as an execution path, not as a prompt followed by an answer.

A user request or business event first establishes an identity. The system then determines delegated authority, retrieves current context, applies policy, chooses a model, invokes tools or APIs and records evidence about the outcome. The model call sits near the middle of this path. It does not own the path.

The surrounding integration may involve an identity provider, SSO, Active Directory or LDAP; CRM, ERP or core banking systems; SharePoint, Confluence or document repositories; relational databases; message brokers and event buses; secrets management; an API gateway; a workflow engine; and security monitoring.

The complexity does not come from connecting twenty endpoints. It comes from preserving meaning across them.

A customer number in a CRM may not match the identifier in a core banking platform or fraud system. A document may remain searchable after the policy it describes has expired. An employee may be permitted to view a customer record but not the investigation associated with it. A downstream system may accept an update but time out before acknowledging it.

These are not language-model problems. They are distributed-systems and enterprise-integration problems.

A Banking AI Request Is Not a Question-Answering Flow

Consider a request such as: “Show me the latest suspicious transactions for customer John Smith.”

The visible task looks like information retrieval. The actual banking AI integration may need to:

1. Authenticate the employee and establish the active session.
2. Determine whether the employee is allowed to access suspicious-activity information.
3. Resolve which John Smith is intended without leaking records for other customers.
4. Map the customer across CRM, core banking and fraud-monitoring identifiers.
5. Retrieve current transactions and their investigation status.
6. Search relevant policies and case documents with source-level permissions.
7. Assemble evidence for the model without crossing disclosure boundaries.
8. Generate the response and apply output policy.
9. Record the sources, decisions and downstream calls in an audit trace.

The model may complete its part in seconds. The surrounding system must establish which customer the user means, whether the request is authorized, which records are current, what each fraud flag means and which evidence may be disclosed.

In February 2026, the Central Bank of the Republic of Uzbekistan announced that solutions selected through its AI Challenge were moving into banking-system implementation. The program includes an Uzbek-language speech-to-text model and a retrieval-based assistant for regulatory and institutional information. The Central Bank states that the solutions will run on internal servers to protect confidential information and banking secrecy.

The significant engineering transition is not only the development of models for Uzbek language and banking terminology. It is their integration into controlled institutional workflows, source systems and operational boundaries.

Authentication Is Not End-to-End Authorization

Logging a user into an AI application does not automatically authorize every retrieval or action performed on that user’s behalf.

Many demonstrations use one privileged service account for all downstream calls. This reduces implementation time, but it removes the relationship between the employee’s authority and the data returned by the model. Every request effectively receives the permissions of the integration account.

Production systems need separate identities for the user, agent and downstream service. Authority must be delegated explicitly and narrowed for the task. The integration chain should be able to answer:

• Which human or service initiated the task?
• Which agent is executing it?
• Which permissions were delegated?
• Which tool and parameters were authorized?
• When does that authority expire?
• Which system enforced the decision?

The NIST National Cybersecurity Center of Excellence addresses these questions in its 2026 concept paper on software and AI agent identity and authorization. It treats delegation, least privilege, auditing and the binding of agent actions to human authority as separate requirements—not as consequences of successful login.

An agent allowed to read a transaction should not automatically inherit the authority to freeze an account, export a complete customer history or open a fraud investigation. Tool descriptions tell a model what an operation does. They do not grant legitimate permission to execute it.

Agents Turn Integration Flows Into Transactions

Integration becomes more demanding when the AI system moves from reading information to changing enterprise records.

An agent may select tools dynamically, revise its plan or repeat a call after an ambiguous response. In a transactional environment, an uncontrolled retry can create a duplicate payment, reopen a resolved case, reserve inventory twice or send the same customer notification repeatedly.

An agentic integration therefore needs mechanisms familiar from distributed transaction processing:

• idempotency keys for operations that may be retried;
• explicit state transitions rather than inferred progress;
• timeouts and circuit breakers around unreliable dependencies;
• compensating actions when a multi-system workflow fails partway through;
• approval gates for high-impact or irreversible operations;
• durable records of tool inputs, outputs and business outcomes.

Model reasoning cannot replace these controls. A convincing explanation of an action does not make the action authorized, atomic or reversible.

The AWS enterprise architecture for agentic AI separates access to models, tools and knowledge sources for this reason. Tool discovery and secure execution are distinct platform capabilities, with authorization applied to the actor and execution context.

Google Cloud’s reference architecture for orchestrating access to disparate enterprise systems follows the same principle. MCP servers form an anti-corruption layer between the agent and backend systems, while IAM, structured logging and distributed tracing remain separate operational requirements.

MCP can standardize how a tool is described and invoked. It does not remove the need for authorization, transaction semantics or failure handling behind that tool.

A Fluent Answer Can Hide an Integration Failure

Traditional enterprise applications usually expose failure. A missing record produces an error. A rejected transaction returns a status. A timed-out dependency prevents the screen from completing.

AI systems create a more dangerous possibility: part of the workflow can fail while the final answer remains fluent.

A response may combine current transaction data with an obsolete policy document. A search connector may return only part of an authorized corpus. One action may succeed while a second system times out. The model can still produce a coherent narrative because fluency is not evidence that every dependency completed correctly.

This changes the definition of AI integration testing. Evaluating answer quality is necessary, but insufficient. Tests must also cover:

• revoked permissions during an active session;
• ambiguous users, customers or account identifiers;
• stale, contradictory or partially indexed sources;
• unavailable tools and degraded downstream APIs;
• duplicated requests and delayed acknowledgements;
• partially completed multi-system actions;
• policy changes between retrieval and execution;
• audit records that cannot be correlated end to end.

The central test is not whether the model answers under ideal conditions. It is whether the integrated system behaves safely when its dependencies provide incomplete, delayed or contradictory evidence.

AI Integration Monitoring Must Follow the Business Transaction

Monitoring only the model call cannot explain why an integrated AI workflow failed.

Teams need a trace connecting the initiating identity, authorization decisions, retrieved sources, model requests, tool selection, downstream API calls, retries and final business outcome. Without that chain, operators cannot distinguish a model problem from a stale index, slow permission service, retry loop or unavailable system of record.

The OpenTelemetry guidance for generative AI observability standardizes telemetry for model operations, token usage, tool calls and tool results. That provides an important part of the execution narrative. Enterprise AI integration must extend the same trace into identity services, retrieval infrastructure, workflow engines and business APIs.

A successful model response is not a successful transaction if the required downstream action did not occur.

End-to-end tracing also changes cost management. The cost of an AI workflow includes retrieval, data movement, tool execution, retries, external services and operational capacity—not only model tokens. Usage must be attributed to the application and business workflow that created it.

This is one reason observability is a core capability in the enterprise AI infrastructure platform, rather than an optional dashboard attached to a model provider.

Enterprises Need a Shared AI Integration Layer

Large organizations cannot sustainably build a separate integration mechanism for every assistant and agent.

A shared AI integration layer should provide stable contracts for:

• identity propagation and delegated authority;
• access to models, knowledge and approved tools;
• API orchestration and event handling;
• retries, idempotency and common failure policies;
• workflow state and approval gates;
• end-to-end telemetry and audit evidence;
• usage attribution and operational controls.

This layer does not eliminate integration work. It moves repeated controls into shared engineering components while leaving business meaning with the teams that understand each domain.

The platform team may own the gateway, tool registry, orchestration runtime and telemetry contract. A banking team must still define what constitutes a suspicious transaction, which investigation states are valid and when human approval is mandatory. A retailer must still own inventory, pricing and fulfilment semantics.

Centralize the mechanics that must be consistent. Federate the decisions that require domain accountability.

Arcentra Systems applies this boundary across Design, Build and Operate: architecture, implementation and long-term operation are distinct responsibilities, but they must share one integration contract.

When Should the Shared Layer Be Built?

Not every prototype requires a platform. A direct model call can be appropriate when the use case is isolated, read-only and low-risk.

The shared layer becomes justified when several of the following conditions appear:

1. Multiple AI applications connect to the same systems of record.
2. Teams independently rebuild identity propagation, retrieval or tool authorization.
3. Agents can create, update or delete enterprise records.
4. Workflows span systems with different transaction and failure semantics.
5. Operators cannot reconstruct a request across the model and downstream services.
6. Audit, security or finance teams cannot attribute actions and cost consistently.
7. Provider or system changes require coordinated edits across many applications.

At that point, the organization already has an AI integration layer—it is simply distributed across application code, duplicated connectors, privileged service accounts and operational workarounds.

The architectural task is to make it explicit.

Better Models Do Not Remove the Integration Problem

As models improve, the model’s share of the total engineering problem often becomes smaller.

Models can be replaced without changing the business objective. Enterprise identifiers, permission structures, transaction rules, source semantics and operational dependencies remain specific to the organization. A stronger model may reduce reasoning errors, but it cannot determine which customer record is authoritative, repair a broken delegation chain or make a non-idempotent banking API safe to retry.

Companies do not struggle with enterprise AI only because models are insufficient. They struggle because useful AI must operate inside systems that already encode decades of business meaning and control.

The future of enterprise AI will not be decided by better models alone.

It will be decided by integration architecture designed to preserve identity, authority, state and evidence across the complete transaction path.

The model generates the answer. Engineering determines whether the answer can be trusted—and whether the enterprise survives acting on it.

The post AI Integration Is Harder Than AI Itself appeared first on Arcentra Systems.

Those systems remain essential. But another shared layer is forming beside them: enterprise AI infrastructure.

This is not simply a larger collection of models. It is the technical and operational boundary through which business applications reach models, enterprise knowledge, tools and agents. As adoption spreads, that boundary starts carrying responsibilities that no individual AI feature can manage well on its own: identity, policy enforcement, routing, evaluation, observability, audit, data controls and cost allocation.

The architectural transition is easy to miss because it begins as a series of local product decisions. A support team adds an assistant. Engineering adopts a coding agent. Operations automates document handling. Risk introduces AI-assisted analysis. Each initiative looks independent until the same dependencies and controls appear in every design review.

At that point, AI has crossed the line from application capability to enterprise platform.

What Is Enterprise AI Infrastructure?

Enterprise AI infrastructure is the shared technical foundation that enables business applications to use AI models, governed enterprise knowledge and agents securely and reliably. It combines model access, retrieval, orchestration and agent execution with reusable controls for identity, authorization, safety, evaluation, observability, audit and cost.

An enterprise AI platform is the operating form of that infrastructure. It gives application teams stable contracts for using AI without requiring each team to rebuild provider integrations, knowledge connectors, policy checks and telemetry pipelines. The platform can span public cloud, private infrastructure and on-premises systems; its defining property is not where it runs, but which responsibilities it standardizes.

That distinction matters. A model can generate an answer. A platform must determine whether the request is allowed, which model and data boundary are appropriate, what tools may be invoked, how quality will be measured and how the entire execution can be reconstructed later.

AI Starts as a Feature and Becomes a Shared Dependency

Most organizations do not decide to build an enterprise AI platform on day one. They begin with a queue of narrow requests:

• add a customer-service assistant;
• improve enterprise search;
• summarize documents;
• support software development;
• automate a back-office workflow;
• introduce an agent that can act across internal systems.

The first implementation usually optimizes for speed. The application calls one model provider directly. Prompts live in the application repository. Retrieval is built for one corpus. Model names, token limits and fallback behavior become application configuration. Logging captures HTTP success or failure, but not the reasoning path, retrieved evidence, tool activity or output quality.

That design is rational for a prototype. It becomes expensive when repeated.

The second and third teams need the same mechanisms: provider credentials, model selection, prompt versions, content filtering, permission-aware retrieval, tool registration, human approval, evaluation datasets, tracing and usage reporting. Independently implemented, these mechanisms drift. Identical risks receive different controls. Provider changes require edits across many applications. Audit evidence is collected in incompatible formats.

A technology becomes a platform when multiple products depend on common services and operating rules. Enterprise AI infrastructure meets that test as soon as repeated controls become more important than any single model integration.

This transition is also visible in industry standards. OpenTelemetry semantic conventions establish common meanings for telemetry across traces, metrics, logs and events; the GenAI conventions cover concepts such as model operations, token usage, evaluation and tool calls. Standards of this kind become valuable when operations must work consistently across libraries, providers and applications—not just inside one demo.

The Model Is a Dependency, Not the Platform

The model is the most visible part of an AI system because it produces the visible output. Architecturally, however, it is one dependency inside a larger execution path.

For any enterprise request, the surrounding system must answer a sequence of questions:

1. Who is making the request, and on whose behalf?
2. Which data is that identity permitted to retrieve?
3. Which model is acceptable for the task, risk level, latency target and data location?
4. Which tools may an agent discover and execute?
5. What output checks or human approvals are required?
6. What evidence must be retained for operations, audit and improvement?

These are platform questions because the answers should remain coherent across a portfolio of applications. If every team embeds them in prompts and application code, policy becomes distributed, implicit and difficult to verify.

AWS makes a similar separation in its enterprise agentic AI reference architecture: applications consume shared capabilities for model access, tools and knowledge bases, while security, observability and discoverability span the architecture. The specific implementation is vendor-dependent; the architectural principle is not. Models, knowledge and tools are separate resources that require a shared control boundary.

A Reference Architecture for an Enterprise AI Platform

Arcentra Systems models the platform as four planes inside one trust and operations envelope. This is deliberately different from a simple vertical “app → model → database” diagram. In production, models, knowledge and agents are peer capabilities coordinated by a control plane; enterprise systems remain the sources of authority and action.

1. Experience plane

The experience plane contains the business applications and channels where AI creates value: CRM and ERP extensions, portals, mobile applications, internal APIs and industry workflows.

Product teams should continue to own the user experience, domain rules and accountable outcomes. A central platform team cannot decide whether a credit-review workflow is correct, what a service agent should escalate or which evidence a regulated decision requires. Those are domain decisions.

The platform supplies the common contract below them.

2. AI control plane

The AI control plane is the governed entry point for AI execution. It normally includes three groups of capabilities:

• Gateway services authenticate callers, protect provider credentials, enforce quotas and record usage.
• Policy and routing select models and deployment boundaries according to task, risk, data location, latency and cost. They also define fallbacks and approval gates.
• Orchestration manages prompt versions, workflow state, multi-step execution and supervised agent behavior.

This layer is more than a reverse proxy. An HTTP gateway can distribute traffic; an AI control plane must understand enough execution context to apply AI-specific policy. A summarization request over public material and an agent request to update a financial system may use the same model family, yet require entirely different authorization, evidence and oversight.

3. Intelligence services

The platform exposes three peer capability groups:

• Models: external providers, private models and specialized inference endpoints.
• Knowledge: retrieval, enterprise search, vector or graph stores, and connectors that preserve source permissions.
• Agents and tools: agent runtimes, tool registries, MCP servers, workflow engines and isolated execution environments.

Keeping these capabilities separate prevents a common design error: treating retrieval as a hidden part of the model call or treating tool permission as a prompt instruction. Knowledge access must inherit enterprise authorization. Tool execution must be enforced by identity and policy. Model choice should remain replaceable without rewriting either mechanism.

4. Runtime foundation

The runtime foundation provides compute, Kubernetes or another scheduler, storage, networking, secrets, delivery pipelines and platform monitoring. It may be delivered in public cloud, private cloud, on-premises infrastructure or a hybrid topology.

Deployment location should follow workload boundaries. Sensitive knowledge may remain inside a controlled environment while selected requests use an external model through a governed route. High-volume inference may run on dedicated infrastructure while experimental workloads use managed APIs. The platform contract should allow those decisions to change without forcing business applications to absorb the underlying topology.

The trust and operations envelope

Identity, safety, quality, telemetry and governance cross every plane. They cannot be added as a final box at the bottom of a diagram.

The NIST AI Risk Management Framework is intentionally lifecycle-oriented, and its Generative AI Profile extends the framework with actions for risks specific to generative systems. The architectural implication is straightforward: governance needs evidence from design, deployment and operation. Runtime controls without evaluation are incomplete; evaluation without traceable production behavior is equally incomplete.

Standalone AI Application vs. Enterprise AI Platform

A packaged AI application can be useful and production-ready. The distinction is not “bad product versus good platform.” It is the scope of responsibility.

The standalone approach remains appropriate for a contained use case with limited integration and risk. The platform approach becomes necessary when several applications need the same controls, when AI can act on enterprise systems, or when audit and operating consistency matter across business units.

What a Mature AI Platform Changes

Maturity is not measured by the number of models deployed. It is measured by the quality of the contracts between applications and shared AI capabilities.

Model selection becomes policy

Early applications hard-code a provider because that is the fastest route to a working feature. A mature platform treats model selection as a policy decision. The route can consider task type, sensitivity, residency, latency, quality history, capacity and unit cost.

This does not require routing every request among many models. It requires keeping the decision outside business code so that the organization can change it deliberately.

Knowledge access preserves source authorization

Retrieval quality is not enough. If a user cannot read a source document in its system of record, an AI answer must not expose that document indirectly. The knowledge layer therefore needs permission-aware ingestion, query-time filtering, provenance and lifecycle rules—not only embeddings and a vector database.

This is one reason enterprise RAG belongs in shared infrastructure. The reusable asset is not merely a retrieval library; it is the security and evidence model around retrieval.

Agents receive identities and bounded authority

Agents introduce a different risk boundary because they can take action. A tool description tells a model what a function does; it does not grant legitimate authority to execute it.

The World Economic Forum’s 2026 playbook for trusted AI agents proposes an Agent Capability and Authorization Profile that joins delegation policy, system design and operational oversight. The useful architectural idea is that an agent’s authority should be explicit, enforceable and auditable across its lifecycle.

In practice, the platform should know which agent is acting, which user or service delegated the task, what resources and operations are allowed, when approval is mandatory and how authority expires. Arcentra Systems applies the same distributed-systems discipline in our work on moving agent ecosystems from prototype to production.

Evaluation becomes part of delivery

Traditional tests verify deterministic behavior around the model. AI evaluation measures the probabilistic behavior that those tests cannot capture: groundedness, task success, safety, retrieval relevance, tool selection and policy compliance.

A mature platform provides common evaluation interfaces and datasets while allowing domain teams to define what “good” means for their workflow. Results become release evidence. A new model, prompt, retrieval configuration or tool definition should pass an appropriate evaluation gate before receiving production traffic.

Observability connects quality to execution

Infrastructure metrics remain necessary, but CPU, latency and error rate cannot tell an operator whether an answer used the wrong source or an agent chose an unsafe tool sequence. AI telemetry needs to connect model calls, retrieval events, tool invocations, policy decisions, evaluation scores and cost to the originating business transaction.

That is why observability belongs in the platform contract rather than in a vendor dashboard. The organization needs one execution narrative across application, gateway, model, knowledge and tools. Our broader approach to production AI infrastructure starts with that operational requirement.

The Platform Team Should Centralize Controls, Not Business Logic

Platform initiatives often fail by centralizing too much. A team that owns every prompt, agent and knowledge source becomes a delivery bottleneck and quietly assumes responsibility for domains it does not understand.

A better operating model is federated:

• The AI platform team owns gateways, provider integration, runtime services, common policy enforcement, evaluation tooling, telemetry and paved-road delivery patterns.
• Security, risk and data governance define control requirements and review evidence.
• Domain product teams own workflows, user experience, business rules, domain evaluations and outcomes.
• Operations teams own service objectives, incident response and capacity with shared telemetry from the platform.

The boundary can be expressed as a simple principle: centralize the controls that must be consistent; keep decisions that require domain accountability with the application team.

This is platform engineering, not the construction of one universal AI application. Arcentra Systems’ Design, Build and Operate model follows the same lifecycle: architecture, implementation and long-term operation are different responsibilities, but they must share one technical contract.

When Should an Enterprise Build the Shared Layer?

Not every organization needs a full platform immediately. Premature abstraction can be as wasteful as duplicated infrastructure. The strongest signals are operational:

1. Multiple applications integrate with the same models or knowledge sources.
2. Teams are rebuilding identity, retrieval, guardrails or evaluation independently.
3. Provider changes require coordinated edits across product codebases.
4. Agents need access to business systems with different permission boundaries.
5. Risk, audit or finance teams cannot reconstruct usage and decisions consistently.
6. Production incidents cross the boundary between application, model, data and tool execution.

Once several of these conditions are present, the question is no longer whether a shared layer exists. An accidental one already exists in duplicated libraries, informal standards and operational workarounds. The architectural choice is whether to make it explicit.

The transition should be incremental:

• Establish one governed model-access path with identity, secrets and usage metering.
• Standardize telemetry and a minimal evaluation contract.
• Add reusable knowledge access with permission and provenance controls.
• Introduce agent and tool authorization before scaling autonomous execution.
• Separate platform interfaces from specific providers and deployment locations.

Each step should remove duplicated risk or operational work. A platform justified only by future elegance will struggle for adoption; a platform that shortens delivery while improving control becomes a product teams choose to use.

The Future Question Is Architectural

Within a few years, asking whether an enterprise “uses AI” will carry little architectural meaning. Almost every software portfolio will contain models, assistants and automated decisions.

The differentiating questions will be more demanding:

• How does the platform decide which models and data boundaries are acceptable?
• Can applications move between providers without rewriting business logic?
• Does knowledge access preserve enterprise permissions and provenance?
• How are agents identified, authorized and stopped?
• Can operators connect quality, risk, latency and cost to a business workflow?
• What evidence supports a production release or an audit?

Enterprises are unlikely to operate dozens of independent AI stacks indefinitely. They will converge on shared contracts for model access, knowledge, agents and control—just as they standardized identity, integration, delivery and observability for earlier generations of enterprise systems.

AI is not becoming another feature of the enterprise.

AI infrastructure is becoming part of the enterprise architecture itself.

For Arcentra Systems, that defines the work: we engineer the platform on which enterprise AI can operate, scale and survive change.

The post AI Infrastructure Is Becoming the New Enterprise Platform appeared first on Arcentra Systems.