Point of View
Agents·2 July 2026·6 min readEvaluating agents like you evaluate people
Why AgentOps borrows more from performance management than from software testing.
Traditional software testing assumes deterministic behaviour: same input, same output, pass or fail. Agents don't work that way. This piece argues that evaluating an agent has more in common with managing a new hire's performance than with running a test suite, and what that means for how you build AgentOps.
The executive problem
Teams shipping their first agents keep reaching for the same toolkit: unit tests, regression suites, a green build. It doesn't fit. An agent's behaviour is probabilistic and context-dependent, so "does it pass" is the wrong question. The right one is closer to "is it performing well, and how would we know if it stopped."
Why it matters now
As agents move from single-shot prompts to multi-step, tool-using workflows, the number of ways they can be subtly wrong grows much faster than the number of ways they can be obviously wrong. Traditional testing catches the obvious failures. It's structurally bad at catching the subtle ones, the ones that erode trust slowly rather than crashing loudly.
Why the conventional response stalls
The instinct is to write more test cases. That helps at the margin, but test cases only cover what you thought to test for, and agent failure modes are dominated by what nobody thought to test for. Coverage numbers go up; confidence doesn't.
The LoweLabs point of view
Evaluate agents the way a good manager evaluates a new hire: define what good performance looks like, observe behaviour in realistic conditions, sample and review rather than exhaustively check, and build in regular check-ins rather than a single pass/fail gate. That's not a metaphor for effect, it's a genuinely more accurate model of what you're managing.
What this looks like in practice
In practice this means a weekly rhythm, not a launch gate. A small panel reads a sample of real transcripts. They score each one against the job description. They flag the ones that were confidently wrong, not just obviously wrong. Over a few cycles a pattern emerges: improving, plateauing, or drifting. It's the same signal a manager gets from months of check-ins, not from a single review.
“You don't performance-review a new hire with a unit test suite. Don't evaluate an agent with one either.”
Building AgentOps that actually catches what matters:
- Define the job description before the eval, what is this agent actually accountable for?
- Sample real transcripts regularly instead of relying solely on synthetic test cases.
- Track drift over time, not just a one-off pass rate at launch.
- Build an escalation path for when the agent is confidently wrong, the same way you would for a person.
- Give the agent a probation period with a named reviewer before it runs unsupervised. Treat it like a new hire, not a finished product.
Founder, LoweLabs
With more than 25 years’ experience across consulting, technology and industry, he has built and scaled multiple Data & AI practices, led complex transformation programs, and worked with some of Australia and Asia’s largest organisations. An INSEAD Global Executive MBA graduate, he combines commercial leadership, deep technical expertise and a practical understanding of how emerging technology can be turned into measurable business value.
What separates a pilot from a product
The gap between a working demo and something people rely on.
Field NoteSynthetic data, real judgement
Where synthetic profiles help, and where they quietly mislead.
Point of ViewWritten at the Wrong Altitude
Why your AI strategy keeps expiring, and why the next rewrite will not fix it.
Point of ViewThe Answer Is in the Portfolio
Eight figures on AI pilots, nothing on the P&L. Why "double down, redirect, or stop" is the wrong question.
Point of ViewNobody Should Own AI
The turf war between your CDO/CIO, CTO and CFO isn't a governance failure. It's a category error.
Point of ViewDesigning the AI Op Model
Before an organisation can delegate work to machines, it has to understand what it is delegating.
Next steps
Building agents that need real evaluation, not just tests?
We help teams design AgentOps that actually catches what matters. Get in touch to talk through your evaluation approach.