Evaluate AI Before and After Production

An AI demonstration answers one question: can the system produce an impressive result under selected conditions?

A production evaluation program answers a harder one: can the organization define, measure, release, observe, and improve useful behavior across the conditions that matter?

That requires more than an accuracy number. Enterprise AI may retrieve protected data, generate unsupported claims, choose a tool, spend money, or prepare a business action. Evaluation has to follow the whole task.

Write the task contract first

Define who uses the capability, what they are trying to accomplish, which sources are authoritative, what a useful result contains, which failures are unacceptable, when a person reviews the output, and what authority the system has.

Turn those statements into observable criteria. For a document assistant, measure whether it finds the right source, cites it, answers from it, refuses when evidence is missing, respects permissions, and completes within a useful latency and cost envelope.

Build a representative dataset

Start with real task shapes, safely anonymized or synthesized when necessary. Include normal cases, edge cases, conflicting sources, missing evidence, stale content, ambiguous requests, restricted documents, prompt injection, long inputs, malformed tool responses, and requests the system should refuse.

Tag cases by risk and business category. A single average can hide a complete failure on the small set of cases that matter most.

Version the dataset. Record why each case exists and which incident, requirement, or risk it protects.

Evaluate components and outcomes

For RAG, test retrieval before generation: relevant-document recall, permission filtering, freshness, ranking, and citation mapping. Then evaluate groundedness, completeness, and unsupported claims.

For tools and agents, evaluate tool selection, argument validity, authorization, side-effect handling, stopping behavior, recovery, and final task outcome. A correct final sentence does not excuse an unauthorized intermediate call.

Add operational measures: latency, token and tool cost, rate-limit behavior, availability, retries, and degraded mode.

Use humans where judgment matters

Automated evaluators help scale regression checks. They can also share the same blind spots as the system being evaluated.

Use domain experts for high-consequence quality, nuanced correctness, and usefulness. Give reviewers a rubric and evidence, not a vague “looks good” question. Track disagreement; it can reveal an unclear business requirement.

Gate changes, not only releases

Models, prompts, system instructions, retrieval indexes, chunking, tools, policies, and orchestration can all change behavior. Treat each material change as a versioned release input.

Run a fast critical suite on pull requests and a broader evaluation before release. Compare candidate and current versions, enforce thresholds by category, and require explicit review for regressions on high-risk cases.

Microsoft Foundry supports dataset and cloud evaluations, including predeployment, recurring, response-based, and trace-based patterns. Some trace and conversation capabilities are preview features; production architecture should verify current support and service-level expectations.

Learn from production safely

Sample real interactions according to privacy and retention policy. Monitor user feedback, refusals, retrieval misses, tool failures, latency, cost, and escalation. Route severe safety or security signals to an incident process.

Turn confirmed failures into regression cases. Production monitoring without a path back into the evaluation set becomes a dashboard, not a learning system.

Avoid common evaluation traps

• Testing only handpicked happy paths.
• Using one aggregate score.
• Evaluating prose while ignoring retrieval and tools.
• Letting the model grade itself without independent evidence.
• Storing sensitive prompts indefinitely because they may be useful later.
• Changing model and prompt together without isolating the cause.
• Treating a vendor safety filter as the complete control system.
• Shipping a preview dependency as if it carried a production SLA.

Release checklist

1. Is the task and authority boundary explicit?
2. Does the dataset represent users, risks, and failure modes?
3. Are critical categories gated separately?
4. Are retrieval and tool behavior measured?
5. Does human review use a consistent rubric?
6. Are latency, cost, and reliability within limits?
7. Are data retention and evaluator access governed?
8. Can results be reproduced by version?
9. Does production evidence become regression coverage?
10. Who can stop or roll back the capability?

Evaluation is a product and architecture discipline. It creates a durable connection between what the business expects and what the AI system actually does.

Sources and further reading

• Microsoft Foundry cloud evaluation
• Evaluate deployed interactions with Microsoft Foundry
• NIST AI Risk Management Framework
• NIST Generative AI Profile