Databricks & NVIDIA expand partnership to boost AI workloads

Databricks has announced an expanded collaboration with NVIDIA aimed at optimising data and artificial intelligence (AI) workloads by integrating NVIDIA CUDA accelerated computing into its Data Intelligence Platform. This announcement was made during the Data + AI Summit.

The collaboration is set to enhance the efficiency, accuracy, and performance of AI development pipelines, which are crucial for modern AI factories. The expanded partnership will see Databricks incorporating native support for NVIDIA GPU acceleration within its Data Intelligence Platform. This builds on the existing collaboration between the two firms to improve enterprise experiences, from training classical machine learning models to deploying generative AI applications and optimising digital twins.

Ali Ghodsi, Co-founder and CEO of Databricks, expressed enthusiasm about the extended collaboration, stating, "We're thrilled to continue growing our partnership with NVIDIA to deliver on the promise of data intelligence for our customers from analytics use cases to AI." He further noted the potential for organisations to build their own AI factories on their private data thanks to this enhanced relationship.

Jensen Huang, Founder and CEO of NVIDIA, underlined the importance of data in the generative AI revolution. "Reducing data processing energy demands with accelerated computing is essential to sustainable AI platforms," he said. Huang added that integrating NVIDIA CUDA acceleration into Databricks’s core computing stack establishes a foundation for enterprises to leverage their data in powering generative AI.

Databricks plans to integrate NVIDIA-accelerated computing within its next-generation vectorized query engine, Photon. Photon currently powers Databricks SQL, the company’s serverless data warehouse, which boasts industry-leading price-performance and total cost of ownership. Both companies believe this integration will drive the next frontier of price-performance in data warehousing and analytics workloads.

At the recent COMPUTEX event, Databricks’s open-source model DBRX was made available as an NVIDIA NIM microservice. These microservices provide models as fully optimised, pre-built containers for deployment, significantly boosting enterprise developer productivity. DBRX, which launched in March 2024, was developed entirely on Databricks and trained using NVIDIA DGX Cloud. It serves as a reference architecture for organisations seeking to customise or build their mixture of expert models.

The Databricks Data Intelligence Platform offers a comprehensive solution for constructing, deploying, and managing end-to-end generative AI applications. The platform aims to provide a scalable and secure environment for organisations to benefit from generative AI's capabilities.

This announcement follows closely on the heels of several strategic moves by Databricks. Recently, the company acquired Tabular, a data management start-up, to leverage the capabilities of Apache Iceberg and Linux Foundation Delta Lake. This acquisition positions Databricks as a leader in data compatibility, ensuring organisations are not constrained by their data formats. Additionally, Databricks has been enhancing its Delta Sharing initiative, focusing on breaking down data silos and fostering AI innovation through product innovations and partnerships.