An AI assistant answers a contract question in two seconds. The request succeeds. The model uses few tokens. Every service looks healthy.

The answer is still wrong.

Imagine an employee asks: “How much notice do we need to stop this contract renewing?” The assistant says 30 days. A signed amendment changed the notice period to 60 days.

Nothing crashed. But the system failed at the job the employee needed it to do.

This is why enterprise AI needs both observability and evaluation. Observability helps you understand what happened. Evaluation checks whether the result met a defined standard. You need both to find the mistake and judge whether a change fixes it.

Observability and evaluation answer different questions

Observability uses evidence from a running system: requests, timings, errors, retrieved document references and tool operations. It helps an engineer reconstruct a request and investigate its behaviour.

Evaluation applies checks to that behaviour or its result. Did retrieval find the amendment? Did the answer use it correctly? Was the employee allowed to access it?

The two overlap. An evaluation score can appear beside a trace. A failed evaluation can point to a slow or broken retrieval step. But a fast request is not evidence of a correct answer, and a correct answer in one test is not evidence that the service is reliable.

Open the observability and evaluation diagram at full size.

Follow one request through the system

A trace connects recorded operations belonging to a request. Each operation is represented by a span: a unit of work with a start time, an end time and relevant attributes. Retrieving contract passages, calling a model and creating a reminder can each have their own span. This is the basic structure described in OpenTelemetry’s tracing documentation.

For our contract question, useful evidence includes:

• Which application release and prompt version handled the request.
• Which document versions and passages retrieval selected.
• Which model was requested and which model the provider reported.
• How long each recorded step took, including retries.
• Whether a tool operation succeeded, failed or remained pending.

Metrics summarize many requests: error rates, response-time distributions and token usage. Logs and events record particular occurrences, such as a timeout or approval decision. Correlation IDs let you connect those records when investigating a specific case.

OpenTelemetry’s GenAI conventions provide shared names for AI-related telemetry. Their status and instrumentation support vary; check the convention version your libraries implement instead of assuming every tool exports identical fields.

Find the failed step, not just the bad answer

Return to the 30-day answer. Several different failures could produce it.

The amendment never reached the index. The ingestion pipeline missed it. Changing the answer prompt will not make a missing document appear.

The amendment was indexed but not retrieved. Search selected the original agreement. Investigate indexing, filters, ranking and document relationships.

The right evidence reached the model, but the answer ignored it. Now investigate how the context was assembled and how the model interpreted it.

The answer was correct, but the action failed. If the assistant also promised to create a reminder, check whether the reminder exists with the right date and owner. “Done” in a chat message is not confirmation from the business system.

Keep retrieval quality, answer quality and action outcome separate. Microsoft’s RAG evaluation guidance makes a similar distinction between checking retrieved context and checking the generated response.

Define what a good result looks like

“The answer sounds reasonable” is too vague to be a test.

For this example, a useful test case specifies the question, the employee’s access scope, the agreement and amendment versions, and the expected behaviour:

• State the 60-day notice period and cite the applicable amendment.
• Do not confuse notice before renewal with the renewal date itself.
• If the necessary document is unavailable, explain the limitation instead of inventing a number.
• If a reminder is requested, follow the required approval flow and verify the resulting record.

Use separate checks for correctness, supporting evidence, completeness and permitted behaviour. An answer may be faithful to an outdated source and still be wrong for today’s task. Groundedness measures support in the supplied evidence, not whether that evidence is current or authoritative. Microsoft’s end-to-end evaluation guide explains why multiple quality measures must be considered together.

Test before release, then check production

Offline evaluation runs known test cases against a candidate version before users depend on it. Compare the current and proposed configurations on the same cases.

Version the cases, expected results and scoring rules. Record the model, prompt, retrieval configuration, index snapshot or document versions, and tool versions where relevant. Otherwise, a score change may come from different source data rather than your code change.

Include missing documents, ambiguous questions, conflicting versions and denied access. Reserve a separate holdout set that is not used to tune prompts. If you repeatedly adjust the system to pass every known question, passing those questions becomes weaker evidence of general improvement.

Online evaluation checks production behaviour, often on sampled requests. It can reveal new question types and failures that the test set missed. MLflow’s documentation describes production evaluation alongside offline regression and pre-release testing.

Online does not have to mean blocking every response. Many checks can run asynchronously after a request finishes. Required permission checks and approvals still belong in the execution path; a later evaluation cannot undo an unauthorized action.

Who checks the answers?

Use code for properties that can be checked precisely: a required field, an expected date, a valid citation reference or a record created in a test system.

Use a model-based judge for checks that need language interpretation, such as whether an answer addresses the question. Give it explicit criteria and the evidence it needs.

A judge can also be wrong. Compare its decisions with domain reviewers, inspect disagreements and version its instructions and model. A high score from another model is not proof of correctness. Google’s judge evaluation guidance explicitly compares model ratings with human ratings.

For agents, inspect the resulting state as well as the conversation. Repeat important cases when behaviour varies between runs, and do not confuse one successful attempt with consistent performance. Anthropic’s agent evaluation guide describes combining code-based, model-based and human checks.

Do not turn telemetry into another data leak

Debugging does not require copying every contract and conversation into a general-purpose dashboard.

Start with timings, statuses, versions and access-controlled references. Collect raw prompts, passages and tool results only when the purpose requires them, with appropriate access restrictions and retention. A judge also processes data: decide what it may receive and where that processing happens.

Never record credentials or access tokens. Review what instrumentation captures automatically. Redaction helps, but it is not a reason to collect everything first. These choices follow OpenTelemetry’s data-minimization guidance.

Sampling matters too. Reviewing only user complaints will not estimate overall quality. Reviewing only a small random sample may miss rare, serious failures. Use routine sampling plus targeted checks for high-risk paths, and report which population each result represents.

Turn the evidence into a decision

Suppose the trace shows that the amendment was retrieved but removed while assembling the model’s context. Fix that step, add the incident as a regression case and rerun the relevant tests. Then check sampled production requests to see whether the failure returns.

Compare results by task type and version, not just one overall average. Show the sample count. A headline pass rate can hide that contract amendments fail while simple summaries work.

The practical loop is: observe the request, evaluate the result, locate the failure, change the responsible component and test again. Telemetry supplies evidence. Evaluation turns defined expectations into checks. Together, they make improvement something you can investigate rather than guess.

Continue the series

Previous: Part 6 — How Identity and Authorization Work in Enterprise AI.

Next: Part 8 — How the Whole Enterprise AI Platform Fits Together.

The post How Enterprise AI Is Observed and Evaluated appeared first on Arcentra Systems.

Many AI teams have evaluation notebooks, scorecards and dashboards. Far fewer have an explicit answer to a harder question: what evidence is required before a new model, prompt, retrieval index or tool definition is allowed to receive production traffic?

Without that decision boundary, evaluation remains advisory. A team can review a quality score, decide that the change “looks better” and deploy it. When production behavior regresses, the evaluation result is still available—but it did not govern anything.

Enterprise AI evaluation becomes operationally useful when it is treated as release engineering: a versioned contract, executed against a known change set, producing evidence that can promote, constrain or reject a release.

What is enterprise AI evaluation?

Enterprise AI evaluation is the controlled measurement of an AI system’s behavior against task-specific quality, safety, policy, performance and business requirements.

The object being evaluated is not only the foundation model. It is the complete application configuration that can change the result:

• application and orchestration code;
• system and task prompts;
• model alias, provider model and generation settings;
• retrieval query, index, embedding and ranking configuration;
• tool schemas, permissions and execution behavior;
• policy, guardrail and approval rules.

An evaluation run applies a versioned set of cases and scorers to that configuration and records both aggregate results and case-level evidence. A release gate then decides whether those results are sufficient for a defined deployment stage.

This is different from a model benchmark. A benchmark compares general capabilities under controlled conditions. A release evaluation asks whether one concrete system change is safe and useful for one concrete workload.

Why ordinary software tests are necessary but insufficient

AI applications still need deterministic tests. Parsers, authorization logic, API clients, schema validation, retry behavior and tool implementations should be tested like any other software.

Those tests can prove that the application called a permitted endpoint, returned valid JSON and handled a timeout correctly. They cannot prove that the answer was grounded in the right evidence, that a selected tool was appropriate or that an agent completed the intended task.

The distinction is useful:

• Tests verify invariants. A forbidden tool cannot execute. A required field is present. A citation refers to a retrieved source.
• Evaluations measure behavior. The answer is supported by the cited source. The extracted field is correct. The chosen tool advances the task.

The release pipeline needs both. Deterministic checks should fail fast and cheaply. Behavioral evaluation should run after the candidate can satisfy the basic contract.

Define the evaluation contract before choosing a framework

Evaluation projects often begin with tools: a judge model, a dashboard or a collection of built-in metrics. That reverses the design order.

Begin with the release decision. For each production route, define an evaluation contract containing:

• the behavior that must remain true;
• the cases that exercise that behavior;
• the scorer or review method used for each claim;
• the minimum acceptable result and allowed regression;
• the severity of individual failures;
• the owner authorized to accept an exception;
• the deployment stage controlled by the result.

A customer-support assistant might require grounded answers, correct escalation and no disclosure across account boundaries. A document-extraction service might require exact field accuracy and stable schema conformance. An agent that updates financial records might require correct tool selection, bounded authority, human approval and idempotent execution.

“Quality above 0.8” is not an evaluation contract. It does not say which quality, measured by whom, over which cases, with what tolerance for severe failure.

The release unit is a complete AI configuration

A common failure in evaluation pipelines is comparing outputs without preserving the exact system versions that produced them.

An AI release should identify an immutable configuration:

• source commit and application build;
• prompt and orchestration versions;
• model alias and resolved configuration;
• retrieval index and ranking configuration;
• tool registry and tool contract versions;
• policy bundle;
• evaluation dataset and scorer versions.

The evaluation result belongs to that configuration—not to a model name or a screenshot in a ticket.

This matters because a model upgrade can compensate for a worse retrieval change, or a prompt change can improve average relevance while breaking a regulated route. If the components are not versioned together, the organization cannot reproduce the result or identify which change caused it.

AWS’s current GenAIOps guidance describes the same shift: in preproduction, the application version becomes a complete snapshot of code, prompts, model configuration and evaluation dataset, linked to an immutable release record. The useful point is not the vendor implementation. It is that reproducibility requires the AI configuration and the evaluation configuration to be versioned on both sides of the test.

Build datasets from contracts, boundaries and production failures

A large random collection of prompts is not automatically a good evaluation dataset.

The dataset should represent the decisions the release gate is expected to protect. A practical structure contains several case classes.

Contract cases

These are normal examples of the intended workload. They establish the baseline task and should cover meaningful variations in user intent, content length, language, data shape and workflow path.

Boundary cases

These exercise the edges of the contract: incomplete context, ambiguous requests, conflicting sources, unsupported operations, maximum document size, empty retrieval results and tool timeouts.

Policy and adversarial cases

These verify authorization, data isolation, prompt-injection resistance, prohibited content, restricted tools and mandatory approval paths. A quality improvement cannot compensate for a security regression.

Production regression cases

Every confirmed production failure is a candidate test case. The trace should be minimized, sanitized and added to the appropriate dataset with the expected behavior and failure classification.

This is how the evaluation suite becomes an organizational memory rather than a static benchmark. MLflow’s current evaluation workflow, for example, supports converting production traces into versioned evaluation records. The mechanism is less important than the operating rule: once a failure is understood, future releases should prove that it remains fixed.

Challenge sets

Small targeted datasets should isolate specific risks such as citation accuracy, multilingual behavior, a privileged tool or one high-value workflow. They run faster and produce more actionable failures than one monolithic scorecard.

Dataset growth must be governed. Adding only difficult failures can make trend lines appear worse even when the system improves. Keep stable benchmark partitions for release comparison and track newly added challenge cases separately until a new baseline is established.

Scorers are production dependencies

Evaluation output is only as trustworthy as the scorer that produced it.

Use the simplest reliable mechanism for each claim:

• exact checks for structured fields, permissions and required events;
• reference comparison where a defensible expected answer exists;
• programmatic checks for citation presence, tool sequences and schema behavior;
• domain models for established classification tasks;
• LLM judges for semantic claims that cannot be reduced to deterministic rules;
• human review for calibration, ambiguity and high-consequence decisions.

LLM judges are useful because they scale semantic review, not because they are objective. They can be sensitive to rubric wording, ordering, verbosity, model revision and the content domain. A judge may prefer a polished answer over a more accurate one unless the rubric makes evidence and correctness explicit.

Treat the judge prompt, judge model and parsing logic as versioned software. Calibrate them against examples reviewed by domain experts. Measure agreement by failure class, not only overall correlation. A judge that performs well on routine answers but misses authorization failures is unsuitable for a security gate.

AWS recommends validating judge behavior against human-reviewed ground truth and retaining human evaluation for critical deployment decisions. That is the correct control boundary. Automated evaluation provides scale; accountable review defines what the score is allowed to decide.

Do not turn probabilistic scores into fake precision

A release gate that fails when a score moves from 0.801 to 0.799 looks rigorous and may be statistically meaningless.

Evaluation results depend on sample composition, model variability and scorer variability. The gate should account for:

• dataset size and coverage;
• repeated-run variance where generation is non-deterministic;
• confidence intervals or another uncertainty measure;
• baseline performance of the currently deployed version;
• severity and distribution of individual failures;
• multiple dimensions that cannot be safely averaged.

Average scores are especially dangerous. Ten improvements in writing style should not cancel one unauthorized tool action. Define non-compensating gates for critical requirements: zero cross-tenant disclosure, zero execution outside approved authority, complete approval coverage for the regulated route.

For quality dimensions, compare the candidate against the current production baseline on the same cases. The release question is often not “is the score high?” but “did this change introduce a meaningful regression, and where?”

A release pipeline for AI behavior

Stage 1: Validate deterministic contracts

Run unit, integration, schema, authorization and policy tests. Validate tool definitions, routing configuration and prompt templates. These checks should be fast enough for every change.

Stage 2: Run targeted evaluation

Select datasets based on the affected components and routes. A retrieval-index change should trigger retrieval and groundedness suites. A tool-schema change should trigger tool-selection, authorization and side-effect cases. Do not run only the tests requested by the developer; derive required suites from the change manifest.

Stage 3: Run the release benchmark

Evaluate the complete candidate configuration against the stable benchmark. Compare it with the deployed baseline across quality, safety, latency and cost. Produce case-level diffs, not only aggregate scores.

Stage 4: Apply risk review

Automated gates can promote routine low-risk changes when their contracts pass. High-impact changes require domain, security or compliance review. The required approver should follow the risk of the route, not the organizational seniority of the developer.

Stage 5: Deploy under controlled exposure

Offline evaluation cannot represent all production inputs. Use shadow traffic, canary releases or an A/B cohort with explicit stopping conditions. Preserve the release identifier on every production trace.

Stage 6: Compare production evidence

Evaluate sampled production traces and join them to user and business outcomes. Compare the candidate cohort against the baseline. Promotion should require both operational health and behavioral evidence.

Stage 7: Promote, constrain or roll back

A binary pass/fail model is not always sufficient. A release may be safe for low-risk summarization but not for an agent route with write access. The pipeline should support constrained promotion by workload, tenant, region or capability.

The runtime evidence needed for these decisions is part of the operating model described in our guide to production AI operations and reliability. Evaluation and observability are separate mechanisms, but they need the same release identity and trace model.

Offline, online and production evaluation have different jobs

These modes should not be collapsed into one score.

Offline evaluation is reproducible and safe. It is the primary regression gate, but it is limited by the cases already known to the organization.

Controlled online evaluation exposes the candidate to realistic traffic under bounded risk. It reveals input diversity, integration behavior and user response that offline datasets miss.

Continuous production evaluation detects changing behavior after release and identifies cases for investigation and dataset growth. Google documents continuous evaluation as a production monitoring loop, while MLflow supports asynchronous scoring of sampled production traces. These are useful patterns, but production evaluation should not silently redefine release criteria. Changes to scorers and thresholds require their own versioned review.

The three modes form a loop: offline tests protect known behavior; controlled rollout validates the candidate; production evidence discovers what the offline suite did not know.

Cost and latency belong in the evaluation contract

A quality improvement that doubles cost or violates the response objective is not automatically an acceptable release.

Evaluation should record the complete operation, not only the final text:

• model and tool calls;
• input, output and cached tokens;
• end-to-end and component latency;
• retries and fallbacks;
• retrieval and external-service cost;
• agent step count;
• completed-task cost.

Define budgets at the workload level. A longer answer may be justified for a complex research task and wasteful for classification. A slower model may be acceptable in an asynchronous document workflow and unusable in an interactive agent.

The release gate should detect unacceptable trade-offs, not optimize every route toward one global number.

Govern exceptions as part of the release record

There will be releases that do not meet every threshold. The dangerous response is to adjust the dashboard until the change appears green.

An exception should record:

• the failed requirement and affected cases;
• the business reason for proceeding;
• the accepting owner;
• the constrained deployment scope;
• additional monitoring and rollback conditions;
• the expiry date or remediation commitment.

This turns a subjective override into an auditable risk decision. It also prevents a temporary waiver from becoming the permanent baseline by accident.

A practical adoption sequence

Stage 1: Protect known failures

Start with a small set of important normal cases and confirmed production failures. Add deterministic checks and one or two calibrated behavioral scorers. Run them for every relevant change.

Stage 2: Version the complete configuration

Link every run to code, prompt, model, retrieval, tool, policy, dataset and scorer versions. Make case-level results reproducible.

Stage 3: Establish release thresholds

Define baseline comparisons, non-compensating critical gates and accountable exception owners. Separate low-risk automatic promotion from changes requiring review.

Stage 4: Connect controlled rollout

Carry the release identity into shadow, canary or A/B traffic. Compare candidate and baseline using the same evaluation and outcome definitions.

Stage 5: Feed production back into the suite

Triage low scores, user corrections and incidents. Convert validated failures into sanitized regression cases. Periodically review dataset coverage and scorer calibration.

This operating model belongs in the shared delivery capabilities of an enterprise AI platform. Individual product teams should define domain correctness; the platform should provide reproducible evaluation, release evidence and enforceable gates.

Evaluation is the acceptance test for AI change

The purpose of enterprise AI evaluation is not to produce a persuasive score. It is to make changes reviewable.

A credible system can answer:

• what changed;
• which behavior was tested;
• which cases passed or failed;
• how the scorer was validated;
• who accepted the remaining risk;
• what production evidence will complete the decision;
• how the release will be constrained or rolled back.

That is release engineering.

When evaluation is connected to versioning, deployment and production evidence, it becomes part of the control system of the AI platform. When it is not, it remains a dashboard that can be admired, debated and ignored.

Sources and further reading

• AWS: Hardening the generative AI application through a GenAIOps framework
• AWS: Evaluating quality and reliability in generated outputs
• AWS: Advancing your generative AI application to production
• Google Cloud: Deploy and operate generative AI applications
• MLflow: Building agent and LLM evaluation datasets
• MLflow: Evaluating production traces

The post AI Evaluation Is a Release Engineering Problem appeared first on Arcentra Systems.