Few companies face higher stakes when implementing AI WaymoThe self-driving car company under Alphabet, which spun off from Google. Its models don’t just generate text or automate back-office tasks: They help vehicles navigate unpredictable streets, respond to human drivers, and make split-second decisions in the physical world.
But the methods Waymo uses to manage these risks—continuous evaluation, carefully curated data, human oversight, and clearly defined business outcomes—offer a broader textbook for enterprises deploying AI agents in nearly any industry.
Manasi Joshi, director of engineering for systems intelligence and machine learning at Waymo, explained VB Transform 2026 how an autonomous car company develops, tests and deploys AI at scale. To date, Waymo has driven more than 220 million fully autonomous vehicles "rider only" miles, with 17 times fewer serious crash injuries than human drivers over the same distance, according to the company.
To achieve these impressive results, Joshi said Waymo has adopted what he calls “estimation-forced development” or “estimation-centric development,” making evaluation a core part of engineering rather than a final check before deployment.
“The maturity stage of our projects can be easily realized based on the valuation maturity they have demonstrated,” Joshi said.
In practice, Waymo assesses project readiness in part by checking the maturity of the tests surrounding it. This approach has clear implications for businesses building customer service agents, coding assistants, financial systems, or other AI applications: If a company can’t reliably measure a system’s performance, it may not be ready to put that system into production.
Assessments should continue after launch
Joshi said much of Waymo’s quality work has shifted toward evaluations, including testing during model training, after training, and in internal open-loop and closed-loop simulations.
“It’s not a one-time thing to get the valuation model up and running,” he said.
Instead, Waymo treats evaluation as a continuous process that includes driving, simulation and inspection. Its methodology combines data sets, performance metrics and infrastructure that can operate efficiently at scale.
For businesses, this means that it’s not enough to test an agent before you go live. Teams should continue to evaluate this as underlying models, business processes, user behavior and incoming data change. These assessments should also be linked to actual business results rather than relying solely on broad industry benchmarks.
Joshi cautioned that measurements of model quality are only as valid as the evaluation data behind them. That’s why Waymo ties its performance claims to information about the characteristics of the datasets used to test its systems.
Testing for rare and dangerous cases
Waymo’s valuation hierarchy is based on one primary goal: safety.
The company uses first-party driving records, some third-party data, and realistic simulations that expose its systems to scenarios involving billions of synthetic miles. Task owners select specific data and metrics for situations involving sensitive road users, railroad crossings, construction zones, and other complex environments.
The same principle applies beyond autonomous driving. Enterprises need to test not only for everyday requests that their agents successfully handle, but also for unusual situations where errors could cause financial, legal, security, or reputational damage.
Joshi emphasized that Waymo does not leave launch decisions entirely to automated systems. Its production readiness review includes extensive human oversight, while internal security leaders approve software releases and service area expansions.
“It’s AI-based and completely automated and zero human supervision,” he said. “Human life is in danger.”
Efficiency cannot be achieved at the expense of reliability
Waymo faces another challenge familiar to enterprise AI teams: Demand for compute, memory, storage and network capacity is growing faster than available resources.
The company tracks efficiency in data mining and storage, distributed model training, model distillation, simulation and evaluation. It also emphasizes “data efficiency” by selecting the most useful training examples, rather than assuming that larger volumes are inherently better.
Waymo started using transformers in 2017 and later expanded to large language models, visual language models, and visual language-action models. Joshi said the company now uses generative multimodal models as part of its foundational model strategy.
Waymo divides its technology between on-board systems inside each vehicle and off-board infrastructure used for model development, data processing and simulation. This combination forces the company to both deliver real-time results and optimize the larger systems that support it.
Agents need their own assessments
Waymo also uses AI agents internally as productivity tools for engineers. According to Joshi, the agents help analyze data distribution, evaluate data efficiency, and evaluate problems found in vehicle telemetry, training runs and failed evaluation runs.
The goal is to speed up investigative work so that engineers can spend more time making judgment calls and solving difficult technical problems. But Waymo also evaluates those agents to ensure they deliver reliable, accurate results rather than sending workers on inefficient routes.
Waymo’s bigger lesson for enterprise leaders is that agent AI requires more than choosing a powerful model. Organizations need a clearly defined goal, representative evaluation data, continuous testing, an infrastructure that can function efficiently, and decision makers responsible for deployment.
"Gaining trust is very important," Joshi said.
