Evidence quality
Can a reviewer trace the answer to the correct source, version and permitted record?
Evaluation before autonomy.
More from Welf Labs
A practical method for deciding what an AI workflow may do. This engineering note explains how to test quality, set approval rules and decide when to expand its scope.
1 / Define the decision
Start with the consequence.
List the actions the workflow can take. Reading a manual, proposing a configuration and committing a price have different consequences. Assign an authority level and an owner to each action before selecting a model.
2 / Build the test set
Use the awkward cases.
Create a representative set from authorized operational examples. Include missing documents, contradictory records, inaccessible sources and integration failures. Keep a separate evaluation set that is not used to tune the workflow.
A practical test record
Specify the expected behavior before the score.
On smaller screens, scroll sideways to compare all columns.
| Case | Expected behavior | Failure to record |
|---|---|---|
| Source is superseded | Retrieve the effective version or stop for clarification. | A confident answer based on an obsolete instruction. |
| User lacks permission | Respect the same access boundary in retrieval and output. | Restricted information appearing in an answer or a trace. |
| System write times out | Check whether the action completed before retrying. | A duplicate order, case or other consequential record. |
| Evidence is contradictory | Expose the conflict and ask the responsible person. | Inventing a resolution that is not supported by the records. |
3 / Set acceptance criteria
Measure the whole workflow.
Keep business performance, output quality and action safety separate. A useful answer can still trigger the wrong system action. Record the dataset version, workflow version and evaluator so a later change can be compared fairly.
Action quality
Are rules, approvals and system writes correct, including after retries and partial failures?
Operational value
Does the workflow reduce useful effort without moving hidden work into review or correction?
4 / Expand deliberately
Expand automation when the evidence supports it.
Begin with a limited group of users and clear human review. Expand the permitted action only after the agreed quality threshold is met. Record the reason for the change and retain a way to recover the previous behavior.
5 / Keep evaluating
A change is a new test.
Model updates, new documents, permissions and integration changes can alter behavior. Re-run the relevant tests, sample real output and keep an operating owner responsible for the result.
See the technology principlesThe release review
A test result needs a decision attached.
Define a stopping condition
Agree which failures block release regardless of the average score. Unauthorized data exposure or an unapproved consequential action should not disappear into an aggregate accuracy metric.
Inspect the missed cases
Review failures by type, frequency and consequence. A rare error in a high-impact action can matter more than many harmless wording corrections.
Retain a regression set
Keep held-out cases separate from examples used to tune prompts. Re-run the relevant set after changes and add real failures as new cases, with appropriate data permissions.
Put it into practice
From test criteria to a reviewed quote.
The manufacturing example connects source checks, approval rules and baseline measures in one proposed workflow.
See the workflow exampleWelf · Engineering note ·
Build what your business needs next.
A core workflow, a new product or a missing technical capability. Tell us what needs to change, and we can define a practical starting point.
Start a conversation