Exclusive: Founder of Buildkite reinventing CI/CD to significantly speed software delivery

Buildkite CEO Keith Pitt never intended to build a company, but his passion for problem-solving and engineering led him down an unexpected path.

"I was sort of forced into it," he admitted to TechDay during an exclusive interview.

Pitt, a developer by trade, found himself grappling with security concerns while using existing continuous integration/continuous delivery (CI/CD) tools. These concerns eventually led to the creation of Buildkite, a platform that revolutionises the way CI/CD is handled in modern tech environments.

During his interview, Pitt shared that his journey began when his employer barred him from using their current CI/CD tool due to security issues. The experience pushed him to explore alternatives. "In my previous jobs, I was familiar with all the tools on the market," he explained.

However, the available options were not designed for secure remote work, a problem that became particularly pressing during the COVID-19 pandemic.

Pitt saw a gap in the market: "You ask any security professional, the one thing you don't put on the public internet is your CI/CD tooling," he said, referring to the vulnerabilities associated with traditional CI/CD platforms.

These tools are crucial for a company's software development, and if compromised, they can lead to catastrophic supply chain attacks.

The solution he developed is a hybrid model that combines cloud-based orchestration with private infrastructure. "I thought to myself, well, can there be a cloud-based orchestration tool which is totally fine to put on the public internet, and a secret piece of the puzzle that runs on your own infrastructure? So I built it."

This model allows Buildkite to leverage the computing power of hyperscalers such as Amazon and Google, while also ensuring that sensitive processes remain secure on private infrastructure. As Pitt puts it, "You have the best of both worlds." The hybrid approach resonated with companies, leading Buildkite to grow rapidly. "Thirteen years later, we're now 130 odd people," he shared. The platform is now used by some of the biggest tech companies globally.

One of Buildkite's core strengths is concurrency, Pitt explained. "Delivery is a concurrency game. Ultimately, it's a game of how many things you can do in parallel." With hyperscalers, Buildkite enables businesses to scale their computing power rapidly. "We can spin up from one to 100,000 computers in minutes without breaking a sweat," he added. This capability is essential for companies managing large teams of developers and vast workloads, particularly in environments where speed is critical.

Pitt explained how some of the more popular CI/CD tools are disincentivised from providing faster delivery, as many charge for compute time. This can lead to slower processes, which benefit the platform providers rather than the users. "What incentive do they have to make it go faster? They don't," he said, contrasting Buildkite's business model, which focuses on speeding up delivery.

The platform's ability to adapt to a wide range of environments, from cloud infrastructure to on-device testing, has also made it a key player in fields beyond traditional software development. "We've seen Buildkite installed on cars, light bulbs, and even in fridges, doing on-device testing," Pitt explained, emphasising the flexibility and power of the hybrid architecture. This adaptability makes it a vital tool for industries ranging from Internet of Things (IoT) to artificial intelligence (AI).

When asked how Buildkite helps companies reduce cloud costs, Pitt pointed to the "bring your own compute" model. This allows companies to use their existing relationships with hyperscalers to take advantage of discounts and better hardware.

"We just let you use what you've already got," he said. By using the hyperscalers directly rather than relying on third-party compute services, Buildkite can significantly reduce the costs for businesses while improving performance.

One of Buildkite's latest innovations is the Test Engine, which addresses a common problem in software development: flaky tests. These are tests that sometimes pass and sometimes fail, leading to developer frustration and productivity loss.

"Flaky tests are cancer in software development," Pitt said bluntly.

Test Engine analyses test data, quarantines flaky tests, and provides a workflow for fixing them, helping teams restore trust in their testing processes. This, in turn, saves time and reduces compute costs.

Security is another major focus for Buildkite. Pitt highlighted the platform's open-source build agent, which companies can inspect and even recompile if needed, ensuring complete transparency. "There's a strong line between what we can see and what we can't," he said.

Buildkite's hybrid model ensures that proprietary code remains secure and private, even as it interacts with cloud-based orchestration.

"Our customers don't need to trust Buildkite to use Buildkite, and that's kind of the beauty of the model."

As AI research accelerates, Buildkite has also become an important tool for AI companies that need to protect sensitive models while integrating them into broader development processes. "The hybrid model means they can use their fancy NVIDIA GPUs to do training, and we don't see the models," Pitt explained.

Throughout the interview, Pitt emphasised that while Buildkite is a sophisticated technical platform, at its heart, it's built to solve human problems. "We sometimes get lost in the fanciness of tech, but it's still just a bunch of people at their job trying to do something," he said.

The platform's success, he believes, comes from understanding that companies are fundamentally about people working together efficiently.

As for the future, Pitt is confident that Buildkite's focus on speed and concurrency will continue to set it apart from the competition. 

"The game is getting the code that people write into customers' hands as rapidly as possible."