academyEX flags AI tension as systems grow autonomous

academyEX has published a quarterly Business Technology Briefing on artificial intelligence and superintelligence, focused on the tension between rapid development and restraint.

Founder and Chair Frances Valintine said the discussion examines how societies should respond as AI systems become more autonomous and more embedded in daily life. She framed the issue around the effect on people, rather than the technical race alone.

She said the debate now reflects a widening split between those pushing AI systems as far as possible and those calling for limits on their use. In her view, that tension is no longer confined to research labs or large technology companies. It is now showing up in consumer products, education policy and government procurement.

"This quarterly briefing is particularly focused on some of the friction that we have in the world right now between those who really want to race ahead and push AGI, or at least AI, as far as it possibly can go and those who want to put the brakes on," said Frances Valintine, Founder and Chair, academyEX.

One source of concern, she said, is the speed at which AI systems are beginning to build on earlier models and operate with less human involvement.

"AI is now also forming its own AI, so it's building itself. We've got AI making autonomous decisions, improving itself over time without any human intervention or even human instruction," Valintine said.

Everyday use

The briefing points to signs that AI is already changing social behaviour. Valintine cited research discussed in the session showing that 72 per cent of US teenagers have used an AI companion, a trend she linked to wider concerns about isolation and demographic change.

"This is a really big consideration where you've got a population dropping in almost every part of the world, with a few exceptions, where relationships are harder to find, where loneliness is a real issue," she said.

She also said some governments have begun drawing sharper lines around where AI tools should be limited, especially in education. Schools and public systems, she argued, have become a main arena for restrictions, reflecting concern over how these tools affect learning, decision-making and student welfare.

"Education and AI is very hot. I could have put full pages of different jurisdictions who are banning AI in schools. This one is particularly large, AI in New York public schools, 600,000 students, two-thirds of systems for enrolment will be affected by having removed all access to AI, including companion chatbots," Valintine said.

Sovereign systems

Alongside restrictions, the briefing highlighted efforts by some countries to reduce dependence on foreign AI platforms. Valintine cited South Korea and France as examples of governments pursuing domestic alternatives or state-backed systems.

She said South Korea moved after widespread use of OpenAI products among its population raised questions about sovereignty and where user data sits. France, she added, was pursuing locally developed communications software instead of relying on established US products.

"South Korea had 23 million users of OpenAI, a dependency on a tool that was not embedded with any sovereign data and sitting outside of their jurisdiction. As of July 13th, the government there put out a bid for a public-private partnership called Everyone's AI, a free unlimited chatbot for everyday use," Valintine said.

"France's government is currently ditching Microsoft Teams and Zoom and building a sovereign platform. By the end of 2027, they're going to replace Microsoft Teams and Zoom with locally developed video conferencing," she said.

In her view, New Zealand has less room to pursue the same strategy because of its scale. That leaves the country more exposed to offshore decisions and more dependent on international providers.

"We're not like South Korea where we've got an alternative coming through. Our economy doesn't support that, so we are more affected in some ways," Valintine said.

Human response

Valintine said the answer is not to reject AI outright, but to be deliberate about how it is used and to protect forms of human connection that technology cannot replace. Her comments placed social judgement and community at the centre of the response, rather than technical rivalry.

She also described an unusual moment for the technology sector, with companies trying to win market share while also publicly acknowledging risks to society.

"There's not many times in history where companies have come out and said, look, we are trying to grow our market share, but at the same time, we know this could have a detrimental effect for humanity. It's a conflict that we don't really have a place to do something with, because it's so unprecedented," she said.

The briefing is part of academyEX's regular analysis of business and technology trends for New Zealand leaders. It spans issues including work, education and public policy, but in this edition the focus remained firmly on the balance between machine progress and human judgement.

"I am such an advocate for surrounding yourself with people in conversation more than ever. This is the antidote to a lot of this. Off screens, have the conversations," Valintine said.

"Humanity is incredibly powerful. We are very good at navigating and doing things that are good for each other. We are a community-based country. We believe in the collective. Make sure you're part of it."