Before you begin
Make a copy or preserve a known working version before introducing production data or major changes.
Identify the person who will review uncertain, exceptional, or incorrect results.
Prepare the application
An AI feature is not ready because the output looks plausible.
It is ready when you can explain what a correct result must contain.
Write
“A good result must…”
Examples
“A good result must extract every termination date exactly as written, cite the clause, and flag any notice period longer than 30 days.”
“A good result must recommend only vendors that meet all required criteria and explain each score using the submitted data.”
“A good result must answer only from approved policies and show the source.”
Then define failure
“A result is wrong if…”
Use this definition to review examples and create an eval. The eval does not need to be complicated. It needs to reflect the judgment that matters to the user.
The clearer the success criterion, the easier it becomes to compare models, improve prompts, catch regressions, and decide whether a workflow can be trusted.
Common mistakes
Using “accurate” as a criterion without defining the expected output.
Changing the test set while comparing versions.
Measuring only average performance and ignoring critical failures.
Frequently asked questions
What is the difference between a preview and production?
A preview proves that the workflow can exist. Production adds realistic data, tests, permissions, monitoring, deployment, support, and a defined response when the system is wrong.
Do I need to complete every production step at once?
No. Identify the next material risk and address it. Typical next risks are data, evaluation, edge cases, deployment, or human review.