Cruise beats out Waymo in California AV testing miles in 2020
Cruise LLC clocked the most miles of any company testing self-driving vehicles in California in 2020, according to new reports from the state's Department of Motor Vehicles.
Cruise, which is largely owned by General Motors Co., drove more than 770,000 miles last year, while rival Waymo LLC drove just under 629,000. Both drove hundreds of thousands of miles more than other top brands such as Pony AI Inc., Zoox Inc. and Nuro Inc.
Both companies also had the fewest disengagements per 1,000 miles — when a failure of the technology is detected or when a safety driver takes control of the vehicle — with 0.033 for Waymo and 0.035 for Cruise. By comparison, some companies with fewer miles had higher rates: For example, Lyft had 3.75, Apple had 6.91, Mercedes Benz had 38.92, and Toyota Research Institute had 422.6.
However, the companies testing in California and experts warn that the numbers say little about the safety and reliability of vehicles under development. AV companies are all racing to deploy their technologies on a large scale first, but there still are no federal standards to help guide them there.
For example, there are no standards for when safety drivers should take control, and the number of disengagements is highly dependent on where a vehicle is tested, said Sam Abuelsamid, principal analyst at Guidehouse Insights. For example, driving in a city environment during a pandemic when there is less traffic or in a suburban area may lead to longer periods without human intervention.
"Every AV company has its own procedures for how safety operators are expected to take control. Some are more conservative than others and a lower number of miles per disengagement may be due to the operator feeling less confident or simply being less willing to take chances to see if the vehicle will recover itself," Abuelsamid said.
"Unfortunately, this is the only publicly available metric we have, but it is not necessarily an indicator (of) progress without much more context."
Companies have spoken out against the disengagement reports. They include GM-affiliated Cruise and Waymo, the self-driving vehicle unit of Google parent Alphabet Inc.
"The idea that disengagements give a meaningful signal about whether an AV is ready for commercial deployment is a myth," wrote Kyle Vogt, co-founder, president and chief technology officer at Cruise in a 2020 blog post.
But Cruise's Robert Grant, senior vice president of government affairs and social impact, wrote in a Tuesday blog post that the company is proud of its improved disengagement report, since its rate dropped to half of what it was in 2019.
"For all of 2020, we averaged only one reportable disengagement per 28,520 miles," Grant wrote. "For the second half of 2020, we improved to more than 60,000 miles between reportable disengagement. And in the final three months of the year, we had zero reportable disengagements, a 10x improvement over the same period a year before."
Most of Cruise's disengagements were attributed to a control, planning, perception or prediction takeover due to an "other road user behaving poorly."
Waymo, which has AV development and testing partnership with Stellantis NV, wrote on Twitter that it appreciates what the DMC is trying to do with the report "but the metrics provide limited value in assessing the capabilities of the Waymo Driver, or in distinguishing its performance from other A/V companies.
"Further, our testing program spans different vehicles, software platforms, testing environments and objectives, so the disengagement rate in any given year simply can’t be predictive of real world performance."
Waymo reported 21 disengagements over 629,000 miles of autonomous driving in California for a rate of 0.033 per 1,000 miles, The disengagement rate fell for a sixth year in a row, despite additional complexity Waymo said it introduced this year, including adding new testing locations.
Most of Waymo's disengagements were due to either the vehicle failing to detect an object correctly or due to an "unwanted maneuver of the vehicle."
Experts and safety advocates note it's challenging to track progress in autonomous vehicle testing due to minimal data reporting requirements. Companies may voluntarily participate in a data reporting program with the National Highway Traffic Safety Administration, but are not required to do so.
The DMV reported Tuesday there were 63 reports submitted for 2020 that showed AV fleets drove nearly 2 million miles, a decrease of 800,000 from 2019 likely because COVID-19 prevented and limited testing.
In 2019, Cruise drove 831,040 miles, but the company had to halt testing during shelter-in-place orders and the California wildfires.
"At the heart of these reports is one very simple stat — the total autonomous miles driven by self-driving cars. In our case, Cruise drove 770,049 miles in California in 2020. And while the old adage says, 'It’s not the years, it’s the mileage,' last year was definitely an exception to the rule," wrote Grant. "It’s simply impossible to tell the story of those miles without the context of the unprecedented year during which they were driven."
rbeggin@detroitnews.com
Twitter: @rbeggin
khall@detroitnews.com
Twitter: @bykaleahall