Future of AI is distributed infrastructure built on open source

The exponential growth of AI workloads is driving a 10x-20x increase in demand within a short timeframe. Existing cloud infrastructure is proving inadequate with latency issues, bandwidth constraints, and the sheer volume of requests creating bottlenecks. Traditional cloud models are struggling to keep pace.

"No other space in technology has evolved or changed so fast as artificial intelligence," said Thierry Carrez, the General Manager of OpenInfra Foundation. "We've seen open source being used in AI frameworks, in open models, in inference and in agents. But arguably there is no way you can run those frameworks or those harnesses or that inference without infrastructure that helps with running it."

The cost of that infrastructure is growing exponentially. Over the last year, infrastructure spending has more than doubled with Carrez saying it could reach USD$750B this year.

Powering frontier inference

"When it comes to infrastructure, you have a few choices to make. The first choice is where that infrastructure should run. You can run on someone else's infrastructure. That's the most convenient choice. You don't have to worry about infrastructure anymore. You can focus on your core business. But I would argue that you need to care about infrastructure because that's where your real gains are," said Carrez.

He argues that running AI on-prem as part of your business is the right choice in the age of AI because you can control costs, avoid vendor lock-in and better manage compliance.

"The traditional cloud model is reaching its limits. It's simply not designed to handle the scale and latency requirements of modern AI," said Carrez.

Alongside the physical infrastructure, there are decisions to make about software. And while there are off-the-shelf solutions, he suggests that this is another type of vendor lock-in. Open source, he added, gives control and leverages the existing work of thousands of contributors.

"Open infrastructure is mature. It's been used in production for years at massive scale," he added.

Building the AI stack

Delivering AI at scale requires a hardware and software stack that enables every component to be optimised. Carrez, along with many other open source experts at KubeCon in Shanghai, espoused a four-tier model.

The first-tier focusses on the edge and bringing compute power closer to the source of data whether that's IoT devices, industrial sensors, or end user devices. This reduces latency and bandwidth requirements. The goal is to move processing from centralised clouds to decentralised locations.

Specialised hardware is the basis of the second tier. While GPUs remain important, there's a growing recognition that specialised hardware accelerators like TPUs, FPGAs, and potentially entirely new architectures will be essential for specific AI workloads. But for them to be truly effective open standards and interoperability are key.

Traditional networks are not up to the task. The need for low-latency, high-bandwidth networks that potentially use technologies like RDMA over Converged Ethernet are critical. This networking tier ensures that each layer of the infrastructure can send and receive data without becoming a bottleneck.

The fourth tier focusses on data pipelines and orchestration to enable the efficient movement and processing of the massive datasets required for AI training and inference.

The accelerating adoption of AI presents a significant infrastructural challenge, underscored by a massive increase in demand within the coming years. The limitations of traditional cloud models, such as latency, bandwidth, and the escalating volume of inference requests, highlight the importance of embracing a new approach. 

Moving infrastructure in-house offers strategic control over costs, mitigates vendor lock-in, and supports compliance management. A robust and adaptable AI ecosystem demands a layered architecture that encompasses edge compute, specialised hardware, optimised networking, and sophisticated data orchestration. This can be built on a foundation of open source technologies that ensure scalability and foster continuous innovation.