WeRide's hybrid-cloud storage gateway solution for AI and machine learning workloads

Alluxio has announced WeRide is using its Data Orchestration software as a hybrid cloud storage gateway for applications on-premises to access public cloud storage such as AWS S3.

Alluxio, the open source data orchestration software developer for large-scale analytics and AI and machine learning workloads, says the new data architecture provides a localised cache per location to eliminate redundant requests to S3. The solution directly serves data to engineers working with the same data in the same office, circumventing transfer costs associated with S3 and improving end-user work efficiency.

So far, WeRide, Chinas leading L4 autonomous driving company, has accumulated more than four million kilometres of autonomous driving. The rate of data collection will only increase as more testing vehicles are in service. As well as data collected from test drives, applications such as simulation, SIL (Software in the loop) tests, and model benchmarking produce terabytes of data daily.

"When designing a new algorithm for our self-driving cars or fixing a bug in an existing one, our engineers need to test the algorithm against existing data," says WeRide executive director of Infra - Simulation, Derek Tan.

"Given our data architecture, this caused bottlenecks such as slow iteration in development, high and unnecessary egress costs and error-prone data synchronisation."

Before developing or debugging, developers need to download the latest cloud data to their local environment. Download speeds and network bandwidth often constrain this. Each time data is downloaded from S3, and there is a charge on the egress data transfer.

"At WeRide, they built a custom data uploading process that copies data to the cloud and retains a local copy stored in NAS or HDFS," says Tan.

"The local copy is necessary for giving engineers faster access to data but causes issues with data synchronisation. Currently, WeRide maintains the local copies by running a cron job to clean up local data regularly."

WeRide says it explored existing technologies for a low or no-cost mature technology solution that's battle-tested for large-scale data access and would allow them to scale by utilising better hardware when their budget allows.

"Alluxio became a top choice to accelerate our data access," says Tan. "In addition to being compatible with S3, it provides an easy access interface via its POSIX and HTTP endpoints. As an open source technology, we can incorporate it into our system without adding additional business costs."

WeRide deployed Alluxio in each office as a small on-premise cluster, using S3 as the source of truth. Road test data is directly uploaded to the local Alluxio cluster, which the engineers can immediately use in the same office.

Alluxio automatically uploads the road test data to S3 in the background. As engineers in other offices want to use road test data, they can request their local Alluxio cluster. The data will either be returned immediately if cached by Alluxio or fetched from S3 if not. To further reduce the fetch time of new data from S3, WeRide worked with the Alluxio team to implement a distributed load command which can open multiple simultaneous connections to download data.