Benchmarks don't predict production behavior
Public model benchmarks measure reasoning and knowledge. They don't measure whether an agent recovers gracefully when a tool call fails, whether it loses track of state across a long conversation, or whether it handles a user's ambiguous instruction sensibly. Those are the failures that actually show up in production, and none of them are visible in a leaderboard score. We go deeper on the underlying research in evaluating AI agent reliability before production.
What we actually test
- Golden path first, but it's the smallest part. The obvious, well-formed request has to work — but it rarely reveals anything interesting on its own.
- Injected failures. We deliberately fail a tool call, return malformed data, or make a dependency time out, and check whether the agent recovers sensibly instead of repeating the same failed action or fabricating a result.
- Ambiguous and contradictory input. Real users don't phrase requests the way a spec does. We test what the agent does with an underspecified or self-contradictory instruction — ideally, it asks a clarifying question rather than guessing.
- Replay determinism. Where possible, we test the same scenario multiple times and look at whether the agent's tool-call sequence is stable enough to debug when something eventually goes wrong in production.
Why we review transcripts by hand
Automated scoring catches a lot, but a human reading the full transcript catches the failure modes that don't have a clean automatic check — a technically correct answer delivered in a way that would confuse or alarm a real user, for instance. For anything with compliance stakes, like the clinical workflows inside Physiolaxy, we treat this manual review as non-negotiable before launch; see the fuller picture in AI agents in healthcare.
The discipline transfers across industries
This same rigor — deterministic, repeatable validation before anything ships — is the exact principle behind Upfreq Robotics' simulation-based testing for robotics software. Whether the thing under test is a robot's control software or a customer-facing support agent, the underlying question is the same: does this hold up outside the happy path, and can we prove it before a human is affected?
Frequently Asked Questions
How long does agent evaluation take before launch?
It scales with the agent's blast radius — a low-stakes internal tool might need a day of scenario testing; a client-facing clinical or financial agent gets a much longer, more adversarial pass before we sign off.
Can this evaluation process be fully automated?
Parts of it can — injected-failure testing and replay determinism checks are largely automatable. Ambiguous-input handling and transcript review still benefit from a human in the loop, and we don't cut that step to save time.
Conclusion
An agent that only has to survive a demo needs to get the happy path right. An agent that has to survive production needs to survive everything else, too — and that's the bar our evaluation process is built to test against, on every project, before a client agent goes live.