IT leaders split over AI's role in production fixes

Grafana Labs has published survey findings showing that most IT leaders are comfortable with AI investigating incidents, but fewer are willing to let it make changes in production. The research points to a gap between AI adoption in observability and trust in autonomous action.

The survey found that 85% of IT leaders are comfortable with AI carrying out read-only investigations, while 52% are comfortable with fully autonomous remediation. It also found that 68% of organisations have moved AI beyond investigation into pilots or production use within observability.

According to the findings, AI is already affecting day-to-day operations for engineering teams. Sixty-seven per cent of respondents said AI-based tools had improved the on-call experience, including 28% who said the technology had significantly reduced on-call burden and alert fatigue.

Anthony Woods, Co-founder of Grafana Labs, said the divide reflected a broader question about how far companies are prepared to let software act without human approval.

"AI is already changing both sides of observability. Teams are using it to understand and operate increasingly complex systems, while at the same time they need visibility into the AI applications and agents they're putting into production," Woods said.

"The next step is figuring out how much authority we're prepared to give AI. There's a meaningful difference between trusting it to tell you what's wrong and letting it make a change in production. A future of autonomous action isn't possible unless AI has a clear understanding of how your software and systems work.

"That's where we're focused: Grafana Assistant uses the telemetry you already collect to understand your infrastructure and applications, and AI Observability extends that to show how your agents are behaving. The more autonomous these systems become, the more important that visibility is."

Trust and control

The findings suggest trust remains a central issue as companies move from AI-assisted diagnosis to AI-led action. Respondents identified a proven track record for accuracy and greater explainability as the clearest routes to stronger confidence in AI-generated observability insights, with 45% citing each factor.

Caution persists even as adoption widens. The report said AI is becoming more deeply embedded across observability workflows, helping organisations investigate incidents faster, automate operational work and ease pressure on engineers handling on-call duties.

Spending rises

Observability budgets are also increasing. The survey found that 78% of respondents had raised observability spending over the past year, while 73% expected to increase it again in the coming 12 months.

AI is influencing that growth in two ways, according to Grafana Labs. Teams are adding AI-related features to their observability stacks, while AI-assisted software development is increasing the amount of infrastructure and application activity that needs to be monitored.

Most respondents said the additional spending was delivering some return. Ninety-seven per cent reported at least some value from higher observability investment, although more than a third said they could not yet assign a clear figure to that return.

Platform shift

The research also pointed to continued interest in consolidation. More than half of respondents, 53%, said their organisations now use a unified observability platform across the business, while 79% described consolidation as very or extremely important.

Results were stronger among organisations that had already completed that shift. Among those using a unified platform, 96% said their consolidation efforts had been successful, compared with 66% of organisations that remained siloed by team or function.

LLM oversight

Another theme in the survey was the growing need to monitor AI systems themselves. Eighty per cent of organisations said they were already building or operating products that use large language models, suggesting such tools are becoming part of mainstream production environments.

Among those developing LLM-based applications, 63% said they monitor model output quality or accuracy. A further 53% said they trace LLM calls for latency and token usage, showing that technical oversight is extending to model behaviour as well as traditional software performance.

The findings also suggested that cost discipline around observability has become more established. Nine in 10 organisations reported having some approach to managing observability costs.

The survey was conducted with ETR, a Futurum company, and examined how organisations are using observability tools as AI becomes more deeply embedded in software operations and digital products.