"The agent seems to work" is not an evaluation strategy. It is what teams say right before the first production mistake that matters.

The failure rarely looks dramatic at first. The agent sends the wrong follow-up. It moves a workflow forward without enough reason. It gives an answer that sounds good but is not grounded in the context it had. By the time the team notices, trust is already damaged.

That is why we treat evaluation as part of the system design, instead of a final step after the workflow is built. It is also how CoEdify works day to day. We do not have a separate QA team. Engineers own quality, backed by automated tests, CI gates, independent review, and a person who accepts the work before it counts as done. Things still go wrong; the aim is to catch them early, before a customer does.

Whoever does the checking, evaluating an agent well takes a method. Here is ours.

Why agents are harder to test than normal software

Traditional software testing assumes that the same input produces the same output. Agents do not work that way. Three things make them harder to evaluate.

1. The output is often open-ended

A good outreach draft, a summary note, or an assessment question has no single correct answer. It has a range of quality.

So you are testing judgment as well as correctness.

2. The output depends on context

The same instruction can be right in one workflow state and wrong in another.

A follow-up that suits an active prospect is wrong for someone who already opted out. An assessment question that is useful for one role may be weak for another.

3. The agent takes actions, not just words

This is the biggest shift.

Once an agent updates records, sends messages, or moves work through a workflow, evaluation is about system behavior as much as language quality.

So your evaluation has to cover both:

  • what the agent said
  • what the agent did

Evaluation as an engineering pipeline

Teams that evaluate agents well do not rely on one metric or one judge model. They use layers.

Layer 1: Deterministic behavior checks

Some things should be checked like normal software. For example:

  • did the workflow move to the right stage
  • was the action logged
  • did the agent skip a blocked contact
  • did the system write the expected structured fields

These are system checks with yes-or-no answers.

If an agent sends outreach to a contact marked "do not contact," that is a failure, not a matter of style.

Make this layer as deterministic as possible.

Layer 2: Rubric-based quality review

For open-ended output, exact matching is the wrong test. Score it against a rubric instead:

  • relevance
  • personalization
  • clarity
  • groundedness (does it only use facts it was given?)
  • fit for the current workflow stage

This is where many teams misuse LLM-as-judge, where a second model grades the first. The judge model is rarely the problem. Vague criteria are.

"Is this good?" is a weak evaluation prompt.

"Does this draft use only information present in the provided context, and is the call to action appropriate for the current workflow stage?" is much better.

The judge needs a rubric.

Layer 3: Business rule and policy checks

Many failures are neither language problems nor system bugs. They break business rules. For example:

  • contacting someone too soon
  • moving a workflow forward without a valid reason
  • producing output outside the approved scope
  • citing facts that were never in the context

These checks should live close to the workflow itself. They are part of how the product behaves, as much as they are part of the testing setup.

Layer 4: Human review of samples and edge cases

People do not need to review every run. They do need to review:

  • flagged runs
  • ambiguous cases
  • new types of workflow
  • regular samples, to catch drift

That keeps quality grounded without turning evaluation into a fully manual process.

What this looks like in devsko

devsko, our AI hiring and assessment product, is a useful example because it sits exactly where weak evaluation becomes dangerous.

It runs an AI-led assessment, and its output affects how candidates are understood and how hiring teams judge them. So the system cannot rely on "looks reasonable."

The evaluation has to ask questions like:

  • does the assessment stay aligned with the role-specific skills
  • is the output complete enough to be useful
  • are the scores internally consistent
  • does the workflow capture enough evidence for a person to verify

The important point: a system does not become trustworthy because AI generated the output. It becomes trustworthy because the output sits inside a structured evaluation flow.

The same principle applies to agents in support, sales, operations, and internal tools.

Evaluation and guardrails are different jobs

Evaluation and guardrails are related, but they do different jobs.

Evaluation tells you whether the system is performing well. Guardrails stop the system from doing something harmful in real time.

A useful way to split them:

  • evaluation measures quality and catches regressions
  • guardrails block unsafe or invalid behavior as it happens

For example:

  • evaluation can show that groundedness is getting worse over time
  • a guardrail can stop an output with unsupported claims from being sent

With evaluation and no guardrails, you find problems too late. With guardrails and no evaluation, the system may stay safe while quietly getting worse. You need both.

Where to start

If you are building your first serious agent workflow, start here.

1. Build a small benchmark set

Pick representative cases:

  • normal cases
  • obvious edge cases
  • known failure cases
  • high-risk workflow states

This becomes your baseline for every change. Without it, every prompt change and model change is guesswork.

2. Write deterministic checks first

Before you ask a judge model to score language quality, make sure you can verify:

  • state transitions
  • action logs
  • blocked actions
  • required fields
  • rules that must always hold in the workflow

This layer is the cheapest to maintain and the easiest to trust.

3. Use rubric scoring only for the subjective parts

Use rubrics for:

  • summaries
  • outreach drafts
  • explanations
  • question quality
  • tone and personalization

Keep each rubric narrow and explicit. A loose rubric gives you a loose evaluation.

4. Check groundedness separately

One of the most common failures in agent output is unsupported detail. The answer sounds right and the phrasing is polished, but a specific claim has no basis in the context the system actually had.

That needs its own check. Measure groundedness separately from fluency.

5. Review real runs continuously

Do not wait for failures to pile up. Review a sample of real runs regularly, especially when:

  • prompts change
  • models change
  • new tools are added
  • the workflow's scope grows

This is how you catch drift before users do.

The regression habit that matters most

Agent quality has to be good, and it has to stay good after changes.

So every meaningful change should rerun the regression set:

  • prompt changes
  • tool contract changes
  • retrieval changes
  • model upgrades
  • workflow logic changes

If the team cannot say whether the new version is better or worse on known cases, it is gambling, not iterating.

This habit is one of the clearest differences between agents that only work in demos and agents that hold up in production.

Tools help, but they are not the strategy

Frameworks like DeepEval, RAGAS, and TruLens can help with parts of this work.

None of them answers the core design questions for you:

  • which behavior must be deterministic
  • which quality dimensions matter
  • which business rules must never be broken
  • which outputs must be grounded
  • which cases need human review

Teams often start by choosing an evaluation framework. A better start is deciding the evaluation logic, then picking tools that support it.

The takeaway

Checking an AI agent's work well takes an engineering discipline that combines:

  • deterministic behavior checks
  • rubric-based review of output
  • policy and groundedness checks
  • sampled human review
  • regression tests on every meaningful change
  • runtime guardrails for unsafe actions

That is how agent systems earn trust. An impressive model and a smooth demo do not earn it. A workflow that measures quality before users find the failures for you does.

We apply this standard across CoEdify's own systems.

At CoEdify, if a workflow matters enough to automate, it matters enough to measure: quality checks, regression runs, groundedness review, and runtime guardrails from the start.