Enterprise AI Platform Architecture Explained

Enterprise AI Systems, Explained — Part 8

An employee asks an AI assistant to explain a contract clause and create a follow-up task. The answer may require company knowledge, a language model and a business API. It also requires identity, authorization, workflow state, audit records and a way to understand failures.

Those capabilities are often discussed separately. In production, they have to work as one system.

This final article in the series puts the pieces together. The goal is not to prescribe one product stack. It is to show where each responsibility belongs, which parts can be shared and which decisions must stay with the application team.

Open the enterprise AI platform diagram at full size.

The diagram is a map, not a checklist

A small classification service may call one model and never need RAG, an agent or a separate gateway. A knowledge assistant may need retrieval but no tools. A customer-service workflow may use retrieval, models, tools and human approval.

The platform should provide the capabilities a workload needs without forcing every application through unnecessary components. The layers in the diagram describe responsibilities. They do not require a separate product, service or team for every box.

The architecture has four horizontal layers:

business applications;
the AI runtime and enforcement layer;
AI capability services;
the platform foundation.

Operations and governance apply across all four.

The application plane owns the business outcome

The application plane contains the products people actually use: copilots, portals, support workflows, internal services and industry-specific applications.

This layer knows why the request exists. It owns the user experience, the business rules and the definition of success.

For a contract assistant, the application decides which contract the employee is working with, whether the user wants an explanation or an action, and what should happen when evidence is insufficient. For a claims workflow, it knows the claim state, required documents and escalation rules.

A shared AI platform should not take ownership of those domain decisions. It cannot decide whether a payment should be released, a customer should receive credit or a legal interpretation is acceptable. It supplies reusable technical controls; the application remains accountable for the result.

The AI runtime coordinates and constrains the request

The runtime and enforcement layer sits between applications and the underlying knowledge, models and tools. It handles live requests and answers three different questions:

What steps should this operation perform?
What is this caller allowed to see and do?
Which approved model endpoint should process the request?

Orchestration owns the workflow

The orchestrator coordinates the operation. It can retrieve information, assemble model context, request a model response, validate a proposed tool call, wait for approval and handle failures.

It may be ordinary application code, a workflow engine or an agent runtime. If the sequence is predictable, explicit code is usually easier to test. A model-directed loop is useful when the next step genuinely depends on what the system discovers.

The orchestrator also owns timeouts, retry limits and workflow state. A model should not decide how many times an expensive operation may retry or whether a failed payment action is safe to repeat.

Identity and policy define the permitted scope

Identity connects the request to a user, service, organization and tenant. Policy turns that identity and the current business context into permissions.

These checks have to reach the resources being used. A user who cannot read a document in the source system should not receive it through RAG. A model that proposes a tool call does not grant permission to execute it.

Authorization can be enforced at several points: before retrieval, while filtering search results, before including evidence in model context and before executing a tool. The exact design depends on the data system and risk, but the final enforcement cannot live only inside a prompt.

We covered this boundary in How Identity and Authorization Work in Enterprise AI.

The LLM gateway controls model access

The gateway gives applications a governed route to approved model endpoints. It can authenticate callers, route requests, enforce rate and token limits, attribute usage, apply provider-specific controls and record model access.

The gateway does not replace the orchestrator. It knows how to control a model call; it does not know whether the business workflow should issue that call or what should happen afterward.

“Control plane” is often used informally for this shared governance boundary. In stricter infrastructure terminology, the proxy instances handling prompts and responses are part of the runtime or data plane; controllers and configuration APIs form the control plane. Keeping that distinction clear matters because configuration can be temporarily unavailable while an appropriately designed runtime continues serving existing routes.

It is also a failure domain. If every model call passes through one gateway, the gateway needs appropriate capacity, deployment safety and regional design. Central control is useful only when it is operated as production infrastructure.

The pattern is examined in The Enterprise LLM Gateway Is the New Control Plane.

AI capability services supply knowledge, models and actions

The next layer contains three different kinds of capability. Keeping them separate makes the architecture easier to reason about.

Knowledge services retrieve authorized evidence

Knowledge services connect approved sources to the live request. They may contain ingestion pipelines, document parsing, chunking, embeddings, keyword and vector indexes, ranking and source references.

Retrieval should return evidence with enough metadata to verify the source, version and access scope. It should also be able to say that it found nothing useful. A weak result should not be disguised as a confident answer.

How Embeddings and Vector Search Work explains the search layer. How RAG Works follows that evidence into a model request.

Model services provide inference

Models interpret the supplied context and generate a response, classification, extraction or tool request. They may be hosted by a provider, deployed in a private environment or selected from several approved backends.

A model is not the owner of company data, permissions or business transactions. It receives a bounded request and returns an output. Model and prompt versions still matter because changing either can change system behaviour even when the surrounding code stays the same.

Tools expose controlled business operations

Tools connect AI workflows to ticketing systems, CRM platforms, payment services, databases and internal APIs.

The model can propose a tool and arguments. The runtime validates the structure, checks permissions, applies limits and executes the operation. The tool returns a structured result that the workflow can verify.

High-impact operations may require approval or a deterministic rule. Idempotency matters too: retrying “read customer” is different from retrying “send payment.”

The full execution loop is described in How AI Agents and Tools Work.

The platform foundation keeps everything running

Under the AI-specific components is ordinary production infrastructure: compute, networking, storage, secrets, queues, deployment pipelines and service monitoring.

This layer is easy to overlook because it is not unique to AI. It is also where many production failures occur.

The platform needs to handle unavailable providers, slow retrieval, exhausted quotas, expired credentials, partial tool failures and unsafe configuration changes. Long-running workflows may need durable state. Sensitive services may need private networking and regional data boundaries.

AI does not remove the need for reliability engineering. It adds new dependencies and more variable behaviour to systems that still have to meet operational expectations.

Operations and governance cross every layer

Telemetry, evaluation, audit, cost controls and release management are shown as a vertical block because they are not one final step after the answer. They apply across the request.

A useful operation record can connect:

the application and release that received the request;
the user, service and policy decision;
the document and index versions used for retrieval;
the prompt, model and gateway route;
the tools requested and actually executed;
the response, business outcome, latency and attributed cost.

That does not mean storing every prompt and document in general-purpose logs. Sensitive content should be minimized, redacted where appropriate, access-controlled and retained for a defined period.

Evaluation checks whether the system meets defined expectations. Observability helps locate the responsible step when it does not. Prompt, model, retrieval and tool changes should be versioned and evaluated like release changes, with a way to stop or roll back a bad version.

See How Enterprise AI Is Observed and Evaluated for the operating loop.

Two flows make the platform easier to understand

The complete platform contains two connected flows.

The knowledge flow

Sources are ingested, parsed, divided into useful sections and indexed with metadata and access information. Changes, deletions and permission updates have to propagate. The output is a governed knowledge service that can retrieve current evidence for an authorized request.

This flow runs before the user asks a question and continues as source data changes.

The request flow

A live request starts with identity and business context. The orchestrator decides which capabilities are needed. It may retrieve permitted evidence, call an approved model through the gateway and execute validated tools. The application returns the answer or business outcome while telemetry records the operation.

The request flow consumes the output of the knowledge flow. Treating them as one undifferentiated “AI pipeline” makes freshness, permission and failure problems harder to diagnose.

Follow one operation end to end

Consider the request: “Explain the renewal clause in this supplier agreement and create a task 90 days before the deadline.”

The application identifies the selected agreement and the requested outcome.
Identity and policy confirm that the employee may read the agreement and create a task in the selected project.
The orchestrator asks the knowledge service for the relevant agreement sections and amendments.
Retrieval returns authorized passages with source and version references.
The orchestrator builds the model request from the question, instructions and evidence.
The gateway applies model policy and routes the call to an approved endpoint.
The model explains the clause and proposes task details.
Application code validates the date, project, owner and permission. If policy requires approval, the workflow waits.
The tool creates the task and returns the task identifier.
The application reports the explanation and confirmed action. Telemetry links retrieval, model and tool records to the same operation.

If the task is not created, the system should not claim that it was. If the amendment is missing, the system should not invent the current deadline. Each boundary turns an ambiguous AI response into an inspectable software operation.

What should be shared?

Sharing becomes useful when several applications need the same capability and control.

Common candidates include model access, credential management, usage attribution, telemetry conventions, evaluation infrastructure, tool registration and reusable retrieval services. Applications can still keep workload-specific prompts, workflow logic, acceptance criteria and user experience.

Do not centralize a component merely because it appears in the diagram. A shared service creates its own API contract, capacity requirements, failure domain and operating cost. Centralize when the consistency and reduced duplication are worth those responsibilities.

A practical build order

Start with one real workload and its acceptance criteria.

Define the user, business outcome and prohibited outcomes.
Implement the smallest explicit request flow that can meet them.
Enforce identity and authorization at data and tool boundaries.
Add the knowledge, model and tool capabilities the workload actually needs.
Record enough evidence to reconstruct failures without collecting unnecessary sensitive content.
Create evaluation cases from expected behaviour and real failures.
Extract shared platform services only when repeated demand and ownership are clear.

This avoids two common mistakes: building a large platform before any workload proves what is needed, and letting every application create incompatible controls that later become difficult to govern.

The whole system in one sentence

The application defines the outcome; the runtime coordinates the request and enforces configured controls; knowledge, models and tools provide capabilities; the foundation runs them; and operations and governance make the result observable, testable and accountable.

That is the enterprise AI platform. The language model is important, but it is only one service inside the system.