Exclusive: Thoughtworks' Shayan Mohanty discusses bridging gaps between research and use

Shayan Mohanty discusses AI's risks, challenges, and future, highlighting Thoughtworks' global efforts to bridge research and real-world applications.

AI's Existential Challenges: Insights from Shayan Mohanty of Thoughtworks

Shayan Mohanty, Head of AI Research at Thoughtworks, is no stranger to tackling the profound challenges posed by artificial intelligence. With seven months in his role after Thoughtworks acquired his startup, Mohanty is tasked with addressing some of the industry's most complex issues.

"The biggest existential risk with AI," Mohanty explained, "is that people aren't able to reason about what's working, what's not, and where the risks lie."

He detailed how the industry has transitioned from predictable systems to those capable of near-limitless outputs, often operating in non-deterministic ways. "The toolset we have to reason about AI systems has changed overnight," he said. "Academia hasn't caught up, and industry hasn't caught up."

A Global Perspective
Mohanty's role isn't confined to the technicalities of AI. With a globally distributed team, Thoughtworks maintains a presence in major cities worldwide. "We have folks all over the world," he said.

"This year, we've done a lot of work in APAC, but next year we're planning a world tour across Europe and South America."

This global reach strengthens Thoughtworks' client relationships. "Having a physical presence in a location where our clients are makes things easier," Mohanty added.

Thoughtworks' strong brand also plays a key role. "We're seen as experts in many areas," Mohanty said, referencing the company's contributions to agile development and popular technologies like Selenium.

Challenges of AI Deployment
One of Mohanty's key focus areas is helping companies transition AI from proofs of concept (POC) to production. "You don't get a return on investment if your application never sees the light of day," he noted.

He pointed out that today's generative AI applications often lack reliability, leading to hesitation in deployment. "We lack the metrics and understanding to convey what a model is able to do and what it's not," Mohanty said.

He highlighted the dangers of deploying generative AI models without a clear understanding of their outputs.

"These models can go off the rails," he explained, adding that companies risk reputational damage if AI systems generate harmful or inappropriate content.

Building Vocabulary and Metrics
Thoughtworks' research is dedicated to creating new tools for understanding and measuring AI performance.

"There's a new set of primitives that need to be discovered," Mohanty explained.

He described the difficulties in applying traditional metrics like accuracy and precision to generative AI.

"With these models, you almost have ranges of accuracy," he said. "We need a statistical lens to understand their behaviours."

This scientific approach, Mohanty explained, is crucial for establishing shared vocabularies for AI performance.

"It's no longer engineering—it's science," he emphasised.

Collaboration and Industry Impact
Thoughtworks' collaboration with global clients helps shape industry standards. Mohanty explained that the company's deep technical expertise and exposure to diverse industries enable it to identify common challenges.

"We gain perspective by connecting dots across industries," he said. "Coupled with our deep expertise, we help shape the entire industry by providing actionable insights."

The company's contributions to foundational technologies like CICD and Selenium illustrate its broader impact. "Thoughtworks has always been about engineering best practices," Mohanty added.

Practical Applications of Research
Thoughtworks' AI research has already delivered tangible benefits. Mohanty cited their work in synthetic data generation, outperforming GPT-4 in specific scenarios. "This is crucial for industries like fraud detection, where high-quality data is essential," he explained.

The team has also developed methods for understanding when to fine-tune AI models versus relying on prompt engineering. "We found two human-interpretable metrics that indicate the best approach," he said.

Another breakthrough involved predicting which words in prompts influence AI outputs.

"This can save companies significant costs by reducing unnecessary computational overhead," he noted.

The Future of AI Standards
Looking ahead, Mohanty sees the rise of agentic AI—systems capable of autonomous actions—as a major challenge. "If we don't have the vocabulary to describe a single interaction with a model, how can we describe a system that acts on its own?" he asked.

Thoughtworks is also critically examining benchmarks and evaluations. While benchmarks are useful for comparing models, Mohanty argued they fall short for real-world applications.

"Benchmarks are fine for general capabilities, but they don't tell you if a model will do what you want," he explained.

Instead, Thoughtworks is focused on creating evaluation tools to understand AI behaviours.

"We're trying to cut through the hype and build foundational ways of understanding what models do and how they think," Mohanty said.

As the AI industry continues to evolve, Thoughtworks is committed to bridging the gap between research and practical deployment.

"The building blocks we create today will determine the future of AI," Mohanty concluded.