Open source is powering the next generation of frontier AI

One would have to be living under a rock to have missed the rapid rise of AI as a business and social force. AI is a part of almost every business discussion and legislative agenda. But to truly understand its rapid ascent, one only needs to look at the numbers.

The number of commits being recorded by GitHub doubled in two months earlier this year, largely driven by AI. Wikipedia has seen a 35% increase in the number of queries it handles but a 60% increase in server workloads on the back of activity from AI agents. Earlier this year, bot traffic on the internet exceeded human traffic with Cloudflare CEO Matthew Price saying this has happened far quicker than he anticipated. And language model performance has increased 5x every 5.2 months since 2020.

While the numbers are impressive or scary depending on your perspective, there's another reality that must be considered. The underlying hardware and software that AI demands need to evolve to keep up with new developments and increasingly complex demands.

This is not the first time a major technological change has forced organisations to rethink their approach to technology. At KubeCon in Shanghai, Cloud Native Computing Foundation (CNCF) Executive Director Jonathan Bryce, noted the contrast with the shift to cloud technology.

"One of the things that's interesting about the AI era that's different to what we saw in the cloud era is we are bringing a lot of variation back into our environments. A lot of the cloud model was about how do we standardise environments and make them as similar as possible and then access them dynamically through APIs. When we look at AI, to get the most out of an AI system we often have custom hardware. We have custom kernels, and all the way up from the chipsets, all the way up to agents we must have like tighter integration. That's an interesting problem set."

While the early days of the AI revolution that grabbed public attention were focussed on which company had the best AI models, the race has shifted. Today, Bryce says AI has moved from a model race into becoming a systems discipline.

Inflection point

One of the challenges organisations face when taking an AI project from proof of concept and scaling it to production is infrastructure. This is where the open source community believes it has an advantage. While tools like Kubernetes were once seen as a tool for workload management, they have an advantage as the AI world shifts from model development to inference.

Chris Aniszcyk, the CTO at CNCF, said we have reached an inflection point.

"Inference is growing like crazy as the predominant workloads. Agents are using a lot more tokens today than they were earlier this year, and this continues to grow as models have become better. These things are continuing to grow and put pressure on the systems that we have developed over the years."

He said the infrastructure challenges this brings can be met by distributed systems that are able to direct specific AI workloads to the most suitable resources. Unlike cloud environments that are designed to be homogenous and commoditised, AI demands heterogeneous environments.

"Agents need proper resource installation and sandboxing. They have their own protocols to talk to each other. You have to serve up models and so on. So there's kind of this new infrastructure that is being developed taking advantage of a lot of the cloud native work and principles we've developed over the years," Aniszcyk added.

Aniszcyk said he is seeing this play out in many organisations. China Merchants Bank saw average accelerator utilisation increase from 35% to over 60%, with a reduction of over 60% in the cost of processing one million tokens. This was achieved by leveraging technologies including Kubernetes, KEDA, Prometheus, HAMi, and Fluid. In addition, the bank consolidated nearly 10,000 accelerator cards across AI training, fine-tuning and online inference.

The shrinking AI loop

Perhaps the most marked change over the last year has been the shrinking time window between inference, and model development and improvement.

"If you go back a few months, even a year ago, the standard way of thinking was training being distinct from inference or serving," said Mark Collier, the Executive Director of PyTorch Foundation. "You might spend months training a model and then hand it off to your platform engineering team. They scale it up or scale it out and start to train another model somewhere down the line. But if you look at the frontier, the people that are really pushing the envelope, they're getting much closer to a continuous learning loop. And the bottlenecks in these situations block the flow of information."

The key to closing this loop, he said, is software. He said that Shopify has closed the loop between inference and model development. If the AI agent doesn't answer a question correctly, the failure is returned to training using PyTorch and back into serving the model.

"They're updating the model weights every single day. It gives you a glimpse of where the market's moving. And again, all of this is only possible because of open source," Collier said.

The rapid rise of AI is transforming business and society, fuelled by exponential growth in model activity and infrastructure demand. The accelerating evolution of language models demands infrastructure that can adapt quickly. Open source software is vital in managing this complexity and connecting inference and model development.