How Enterprise AI Is Observed and Evaluated

Enterprise AI Systems, Explained — Part 7

An AI assistant answers a contract question in two seconds. The request succeeds. The model uses few tokens. Every service looks healthy.

The answer is still wrong.

Imagine an employee asks: “How much notice do we need to stop this contract renewing?” The assistant says 30 days. A signed amendment changed the notice period to 60 days.

Nothing crashed. But the system failed at the job the employee needed it to do.

This is why enterprise AI needs both observability and evaluation. Observability helps you understand what happened. Evaluation checks whether the result met a defined standard. You need both to find the mistake and judge whether a change fixes it.

Observability and evaluation answer different questions

Observability uses evidence from a running system: requests, timings, errors, retrieved document references and tool operations. It helps an engineer reconstruct a request and investigate its behaviour.

Evaluation applies checks to that behaviour or its result. Did retrieval find the amendment? Did the answer use it correctly? Was the employee allowed to access it?

The two overlap. An evaluation score can appear beside a trace. A failed evaluation can point to a slow or broken retrieval step. But a fast request is not evidence of a correct answer, and a correct answer in one test is not evidence that the service is reliable.

Open the observability and evaluation diagram at full size.

Follow one request through the system

A trace connects recorded operations belonging to a request. Each operation is represented by a span: a unit of work with a start time, an end time and relevant attributes. Retrieving contract passages, calling a model and creating a reminder can each have their own span. This is the basic structure described in OpenTelemetry’s tracing documentation.

For our contract question, useful evidence includes:

Which application release and prompt version handled the request.
Which document versions and passages retrieval selected.
Which model was requested and which model the provider reported.
How long each recorded step took, including retries.
Whether a tool operation succeeded, failed or remained pending.

Metrics summarize many requests: error rates, response-time distributions and token usage. Logs and events record particular occurrences, such as a timeout or approval decision. Correlation IDs let you connect those records when investigating a specific case.

OpenTelemetry’s GenAI conventions provide shared names for AI-related telemetry. Their status and instrumentation support vary; check the convention version your libraries implement instead of assuming every tool exports identical fields.

Find the failed step, not just the bad answer

Return to the 30-day answer. Several different failures could produce it.

The amendment never reached the index. The ingestion pipeline missed it. Changing the answer prompt will not make a missing document appear.

The amendment was indexed but not retrieved. Search selected the original agreement. Investigate indexing, filters, ranking and document relationships.

The right evidence reached the model, but the answer ignored it. Now investigate how the context was assembled and how the model interpreted it.

The answer was correct, but the action failed. If the assistant also promised to create a reminder, check whether the reminder exists with the right date and owner. “Done” in a chat message is not confirmation from the business system.

Keep retrieval quality, answer quality and action outcome separate. Microsoft’s RAG evaluation guidance makes a similar distinction between checking retrieved context and checking the generated response.

Define what a good result looks like

“The answer sounds reasonable” is too vague to be a test.

For this example, a useful test case specifies the question, the employee’s access scope, the agreement and amendment versions, and the expected behaviour:

State the 60-day notice period and cite the applicable amendment.
Do not confuse notice before renewal with the renewal date itself.
If the necessary document is unavailable, explain the limitation instead of inventing a number.
If a reminder is requested, follow the required approval flow and verify the resulting record.

Use separate checks for correctness, supporting evidence, completeness and permitted behaviour. An answer may be faithful to an outdated source and still be wrong for today’s task. Groundedness measures support in the supplied evidence, not whether that evidence is current or authoritative. Microsoft’s end-to-end evaluation guide explains why multiple quality measures must be considered together.

Test before release, then check production

Offline evaluation runs known test cases against a candidate version before users depend on it. Compare the current and proposed configurations on the same cases.

Version the cases, expected results and scoring rules. Record the model, prompt, retrieval configuration, index snapshot or document versions, and tool versions where relevant. Otherwise, a score change may come from different source data rather than your code change.

Include missing documents, ambiguous questions, conflicting versions and denied access. Reserve a separate holdout set that is not used to tune prompts. If you repeatedly adjust the system to pass every known question, passing those questions becomes weaker evidence of general improvement.

Online evaluation checks production behaviour, often on sampled requests. It can reveal new question types and failures that the test set missed. MLflow’s documentation describes production evaluation alongside offline regression and pre-release testing.

Online does not have to mean blocking every response. Many checks can run asynchronously after a request finishes. Required permission checks and approvals still belong in the execution path; a later evaluation cannot undo an unauthorized action.

Who checks the answers?

Use code for properties that can be checked precisely: a required field, an expected date, a valid citation reference or a record created in a test system.

Use a model-based judge for checks that need language interpretation, such as whether an answer addresses the question. Give it explicit criteria and the evidence it needs.

A judge can also be wrong. Compare its decisions with domain reviewers, inspect disagreements and version its instructions and model. A high score from another model is not proof of correctness. Google’s judge evaluation guidance explicitly compares model ratings with human ratings.

For agents, inspect the resulting state as well as the conversation. Repeat important cases when behaviour varies between runs, and do not confuse one successful attempt with consistent performance. Anthropic’s agent evaluation guide describes combining code-based, model-based and human checks.

Do not turn telemetry into another data leak

Debugging does not require copying every contract and conversation into a general-purpose dashboard.

Start with timings, statuses, versions and access-controlled references. Collect raw prompts, passages and tool results only when the purpose requires them, with appropriate access restrictions and retention. A judge also processes data: decide what it may receive and where that processing happens.

Never record credentials or access tokens. Review what instrumentation captures automatically. Redaction helps, but it is not a reason to collect everything first. These choices follow OpenTelemetry’s data-minimization guidance.

Sampling matters too. Reviewing only user complaints will not estimate overall quality. Reviewing only a small random sample may miss rare, serious failures. Use routine sampling plus targeted checks for high-risk paths, and report which population each result represents.

Turn the evidence into a decision

Suppose the trace shows that the amendment was retrieved but removed while assembling the model’s context. Fix that step, add the incident as a regression case and rerun the relevant tests. Then check sampled production requests to see whether the failure returns.

Compare results by task type and version, not just one overall average. Show the sample count. A headline pass rate can hide that contract amendments fail while simple summaries work.

The practical loop is: observe the request, evaluate the result, locate the failure, change the responsible component and test again. Telemetry supplies evidence. Evaluation turns defined expectations into checks. Together, they make improvement something you can investigate rather than guess.