AI leaders warn of shadow risks as adoption races ahead

Technology leaders are using AI Appreciation Day to warn that rapid adoption of artificial intelligence is outpacing controls, governance and data foundations in many organisations. Executives from New Relic, Barracuda and GitLab say the next phase of AI will depend on how companies frame problems, manage security risks and track AI-generated code.

AI Appreciation Day has become an informal marker of how quickly generative tools and AI agents have entered mainstream business workflows. Vendors and customers are now grappling with how much autonomy to give these systems and how to maintain visibility over what they produce.

At New Relic, Senior Director of Product Management Peter Marelas said the most important development was not conversational fluency, but machine reasoning. He pointed to a shift from systems that respond on request to agents that run multi-step processes and refine their own outputs.

"The most significant shift in AI isn't that it produces fluent answers. It's that agents can now reason through complex, multi-step problems, testing and refining their own work until they reach a reliable result. We've moved from tools that respond, to systems that think through a problem. However, the value we get from that still depends on us. The difference between a mediocre result and a genuinely useful one rarely comes down to the model. It comes down to how the problem is framed. The people getting real leverage from AI treat it less like a vending machine and more like a conversation, exploring a subject with it first, stating not just what they want but why, and asking it to explain and verify its own reasoning. A great example of how visible this is right now is in how organisations keep their digital services running. When a banking app stalls or a checkout fails, engineers are often buried under thousands of alerts with no clear sense of what actually broke. AI has already changed this, sifting through the noise, correlating signals across systems, and pointing to a likely cause in a fraction of the time it once took. It has made the response time fix these systems meaningfully faster. The next step is letting AI act on what it finds, not just advise. However, an agent is only as good as what it can see. Give it fragmented data and it will move toward the wrong fix, but give it a clear, connected view of the system and it can be trusted to act confidently. That's the real frontier for AI. Not building more capable models, but building the environments that let them act with confidence. The organisations that get this right will be the ones that turn AI's promise into something they can rely on."

Marelas pointed to production systems such as banking apps as early evidence of this shift. He said the focus is now on giving agents a complete view of system data so they can act with confidence rather than simply recommend fixes.

Shadow AI risks

Security specialists are seeing a different kind of growing pain. In Australia, Barracuda Senior Solutions Architect Matt Caffrey said unofficial use of AI tools across the workforce has become a fast-growing blind spot for organisations.

"As we mark AI Appreciation Day on 16 July, it's worth recognising that AI's greatest success may also be creating one of organisations' fastest-growing security blind spots: shadow AI. Employees are embracing AI to work faster, write better and solve problems more efficiently, in some cases using tools that have never been approved by IT or security teams. The challenge isn't AI itself, but the lack of visibility into how it's being used. Attempts to ban AI are unlikely to succeed and may simply push usage further underground, increasing the risk of sensitive data exposure, compliance breaches and unmanaged security risks. Instead, organisations should focus on discovering AI usage across business operations, assessing the risk level of tools and establishing sensible guardrails. By implementing practical governance that enables, rather than restricts, innovation, businesses can empower employees to harness AI's productivity benefits while protecting critical data," Caffrey said.

His comments reflect concern among security teams that formal policies have not kept pace with staff experimenting day to day with public AI tools. Many are now prioritising discovery projects to map which services employees already use.

Speed without control

In software development, AI coding tools have rapidly embedded themselves in engineering workflows. GitLab Chief Product and Marketing Officer Manav Khurana said new survey data showed strong uptake and measurable productivity gains, while raising questions about traceability.

"As we mark AI Appreciation Day, the AI Accountability Report shows AI coding adoption and ROI are strong. 91% of organisations have two or more AI coding tools in active use, and 78% report that developers are writing and committing code faster since adopting AI tools. But speed is running ahead of control, with 43% of respondents reporting that they cannot reliably distinguish AI-generated code from human-written code in their own codebase. This comes with a forward-looking concern. 73% of respondents are concerned about the maintainability of AI-generated code in their organisation's codebase, and 82% say it risks creating a new form of technical debt their organisation is not yet prepared to manage. AI coding tools have delivered on their promise of speed. But the events of the past few months, including supply chain attacks, reliability issues, and regulators tightening expectations around AI traceability and provenance are making clear that speed without control is a liability, not an advantage. The teams thinking ahead are already asking the harder question: can we actually control all the code we're generating? The organisations that will ship trusted software faster are the ones building the foundations of accountability with context, traceability, and governance baked into the platform, not just bolted on after the fact," Khurana said.

He said the next stage of AI in software development will depend on embedded governance features that record provenance and give teams visibility into where and how AI contributes to codebases.