Enterprise AI teams are giving agents more freedom, while their reliance on automated testing is being undermined.
According to a June 2026 VB Pulse survey of 157 qualified enterprise respondents at companies with 100 or more employees, half of enterprises have implemented an AI agent or LLM feature that has passed internal assessments and still caused a customer-facing failure.
The sample is self-selected rather than a probability sample, so the findings should be read as indicative rather than definitive.
But businesses aren’t responding by slowing down automation: 66% of respondents now allow some production deployment without human review or plans to put systems in place to do so within the next 12 months. Only 5% said they fully trust automated assessments to make these release decisions.
This discrepancy is a valuation gap: the autonomy ceiling rises faster than the provision below it.
It also fits into a broader thesis to be explored VB Transform 2026: enterprises deploy agents first, control layers around identity, valuation, pricing, context and orchestration come later. Next year will be a period of improvement, with buyers shifting budget to systems that make agent placement manageable and reliable.
Why is a passing evaluation not a working agent
Traditional software testing usually asks whether specified input produces the expected result. Agent testing is more difficult because the system can choose its own sequence of steps, call tools, retrieve data, change state, and respond differently from one run to another.
An agent can make several individually plausible decisions and still reach the wrong conclusion. It might get the correct account but update the wrong field. It can compose a valid return request, but send it without confirmation. The sixth step can successfully invoke five tools before leaking sensitive data or leaving the workflow incomplete.
The survey shows that businesses already recognize this limitation. The most common reason for not trusting automated assessment is poor alignment with real-world results, cited by 29% of respondents. 21% bias or inconsistency, 18% inexplicability and 17% data leakage or privacy issues.
This hierarchy is important. Businesses say the score often doesn’t predict what will happen when a customer, employee or business process meets an agent in production—it’s not that automated scoring is too slow or expensive.
NIST makes a similar point in its Generative AI Profile: measurements collected in controlled environments cannot be cleanly transferred to deployment because behavior varies with instructions, users, context, and operating conditions. Its guidance requires field testing, post-deployment monitoring, and clear processes for failure escalation.
Skill is not consistency
One successful run proves that the agent can complete the task. It does not prove that it will perform the task reliably.
Anthropic guidance on agent evaluation It measures whether the system succeeds at least once during repeated attempts and differentiates whether it succeeds each time. This distinction is important for customer-facing or operational workflows. Sometimes a model with excellent response may still be unacceptable if the same task unexpectedly fails on the next attempt.
Therefore, enterprise teams should treat repeatability as a primary metric. This means running the same scenario multiple times, testing different expressions and contexts, tool failures, and measuring whether the end business result is correct even if the route changes.
The evaluation set also needs to evolve. Every production event should be turned into a constant regression test. Customer escalations, failed tool calls, incorrect assertions, and data handling errors should be included in the package before deployment instead of remaining isolated support cases.
Autonomy should expand with risk, not ambition
A request does not mean that every agent action must require a human. Human vision cannot encompass millions of low-consequence decisions.
But zero-human operations must be earned with demonstrated reliability and limited by the consequences of failure.
Low-risk activities, such as compiling internal summaries or classifying documents, may tolerate greater autonomy. Financial transactions, customer communications, code deployment, access control changes, and data deletions all need stricter thresholds, repeatable consistency tests, policy checks, rollback mechanisms, and clear human escalation paths.
Risk is also not evenly distributed by company size. Larger enterprises—those with 2,500 or more employees—are moving the fastest toward zero-human implementations for smaller companies, 70% vs. 64%, and they also send more agents who fail a customer, 54% vs. 48%.
This is a warning to business leaders. Taking a person out of the loop does not remove the uncertainty. Without stronger assurance, it turns uncertainty into an automated manufacturing decision.
The market will move toward greater autonomy because the economic incentive is real. The best-positioned organizations won’t be the ones that churn people out the fastest – they’ll be the ones that treat iteration and regression testing as seriously as the speed of deployment.
