Why we're looking at this
A model that scores well on a public reasoning benchmark can still fail in production as an agent — not because it can't reason, but because the surrounding system doesn't recover well when a tool call fails, a document is missing, or a user gives an ambiguous instruction. We care about this directly: every client engagement ends in a production handoff, so “it worked in the demo” isn't a bar we take seriously.
What we're seeing
- Most agent failures we've diagnosed are state and recovery failures, not reasoning failures — the agent loses track of what it already tried, or retries a failed action without changing its approach.
- Deterministic, replayable test scenarios matter more for agents than for traditional software, because the same prompt can produce different tool-call sequences run to run. This is the same principle behind why Upfreq Robotics insists on deterministic simulation validation for robotics agents — the discipline transfers directly to software agents.
- “Golden path” testing catches almost nothing interesting. The evaluations that actually predict production behavior deliberately inject missing data, contradictory instructions, and slow or failing tools.
Open questions we're still chasing
We're still refining how much of this evaluation work can be automated versus how much needs a human reviewing transcripts directly. For now, our default is a human-reviewed test suite per client, structured around the failure categories above, before any agent goes live — see how this plays out in practice in how we approached this for healthcare AI.