Uber’s self-driving car research team is having difficulties driving in heavy rain, navigating when there are large vehicles on the road and detecting crosswalks, according to company research reviewed by The Logic.

Self-driving cars are a key focus for Uber as it gears up for a US$90-billion IPO later this year. The company’s Toronto self-driving research lab is the lynchpin of those ambitions. Uber currently has over 50 people working in that lab, with plans to grow the team to 100 by the end of 2019 as part of the company’s $200-million investment in Canada focused on developing self-driving technology.

The Logic analyzed 38 publications produced by Raquel Urtasun, chief scientist of Uber Advanced Technologies Group in Toronto, and found a number of stubborn problems for the company.

In an emailed statement, Urtasun, who did not agree to an interview for this story, said the company has made improvements on some of the issues flagged in her publications.

Urtasun’s research has repeatedly flagged issues with identifying lane markings. An October 2018 paper titled “Deep Multi-Sensor Lane Detection” cites a number of “failure modes.”

“Heavy rain causes significantly reduced visibility, especially at a distance,” reads the paper. “Large vehicles block the view of the ground for both sensors, showing the negative impact of heavy occlusions.”

Uber relies on clear lines of sight for sensors to see what is happening around the car. When sensors are not blocked, Uber can predict lanes up to 48 metres ahead. However, the company still has difficulty when there’s no paint indicating where lanes should be.

When asked if Uber has found a way to solve these problems, Urtasun said the company has developed a “better” system but offered no specifics.

Crosswalks are also an issue for Uber. In a September 2018 paper, Urtasun proposes a way to combine LiDAR, a laser-based remote sensing method, and camera imagery with online maps to detect crosswalks.

“Drawing crosswalks is not an easy task. As shown in our experiments, crosswalks come in a variety of shapes and styles even within the same city,” reads the paper. “Furthermore, paint quality of the crosswalk markings can often be washed out, making the task hard even for humans.”

In the “failure modes” section of the publication, Urtasun notes an instance where a crosswalk marking with poor paint quality is mistaken for a stop line.

“Our model mistakes the crosswalk for a stop line at an intersection, and does not predict its presence,” reads the paper. Uber has now developed a research prototype which “solves some of these issues,” Urtasun told The Logic.

Uber currently lags behind rivals in self-driving. In 2018 tests, for example, Google sister company Waymo had to have a human take control of its self-driving cars once every 11,154 miles, according to data from the state of California. Uber drivers had to take over once every 0.4 miles.

It’s not just Waymo. Self-driving cars from every major company—including General Motors, Nissan, Nvidia, Apple, Baidu and Mercedes-Benz—were able to operate for longer periods without humans having to intervene. To catch up, Uber needs Urtasun’s research—much of which is in the academic prototype phase and not yet in Uber’s cars—to advance rapidly.

In a number of publications from 2018 and 2019, Urtasun claims Uber has beaten all competitors in high-definition 3D mapping for self-driving cars by combining LiDAR, stereo imagery and camera imagery to create maps that are both more accurate and less expensive.

“Our approach outperforms the state-of-the art while being an order of magnitude faster,” reads a September 2018 paper. “We can perform segmentation of large outdoor scenes of size 160m x 80m in as little as 30ms. In indoor scenarios, we can segment full rooms in less than 15ms.”

Uber has long-term ambitions to use self-driving cars for its ride-hailing service. This requires advances in precision mapping so cars can sense their proximity to other objects and stop at safe distances when picking up passengers. Urtasun claims Uber is beating all competitors on this front.

“Our full system can operate in real-time at 15Hz while achieving centimeter level accuracy across different LiDAR sensors and environments,” reads an October 2018 publication based off a dataset that included 40,000 kilometres of driving.

Urtasun’s group is also working to combine complex tasks, like real-time 3D object detection and motion-tracking, into a single system that saves on computing time.

“Our experiments on a new very large scale dataset captured in several North American cities, show that we can outperform the state-of-the-art by a large margin. Importantly, by sharing computation we can perform all tasks in as little as 30 ms,” reads a November 2018 publication.

It is unclear how quickly Urtasun’s research prototypes end up in Uber’s self-driving car fleet. What’s apparent, however, is that self-driving is particularly important to the company in the run up to IPO. Of the 65 patents that Uber has filed so far in 2019, 60 relate to self-driving cars.

“We have a dedicated team that is responsible for taking my team’s research and bringing it to the production system,” Urtasun told The Logic, adding that her team’s research also indirectly influences the company by exposing non-research engineers to the latest research in self-driving vehicles.  

“This cross-pollination across R&D and technology development is an advantage we have over all other self-driving car companies, largely because we have a sizable world-class in-house research team.”