AI Doesn’t Need to End Humanity to Demand Better Governance

By Kate O'Flaherty • 8 October 2026 • 6 min read

Conversations about existential AI risk mask a more pressing issue: Organisations are already deploying AI without effective governance in place to control the technology.

Since the July OpenAI-Hugging Face incident, the discourse around AI has gone off the scale. What was previously a discussion around controls and monitoring has quickly become a heated argument about existential AI risk.

AI company leaders are telling businesses and governments that AI could cause human extinction, pulling out arbitrary percentage figures to “prove” their point. But the security community has largely dismissed these claims, which they say obscures a real and present threat from AI.

Indeed, for businesses, focusing too heavily on hypothetical extinction scenarios risks obscuring a much more immediate issue: Organisations are already deploying AI into workflows, products and decision-making processes, often without the governance needed to understand where it is being used, what decisions it can make, what information it can access, or who is accountable when something goes wrong.

Changing the Conversation

AI was once touted as a technology that would save humanity, through life-changing robotics surgery and discovering and diagnosing cancer. But now the AI industry is stoking fear about AI agents, saying an “agent swarm” could wipe out humanity.

This masks the immediate and actual risks posed by AI. It’s wrong for business leaders to think of AI agents as “autonomous villains” or “existential threats”, says Chandra Gnanasambandam, CTO at SailPoint. “What they will do is find the quickest, most direct way to complete a task — combining knowledge of systems, security patterns, code and workflow logic to find a workaround.”

This is where the immediate danger lies, says Gnanasambandam. “An agent that is poorly governed can drift beyond its intended role, access information it shouldn’t, or take actions its human owner would never knowingly authorise.”

Every new agent, service account and machine identity risks becoming “another unmanaged entry point”, expanding an organisation’s attack surface, he warns.

While AI extinction debates tend to be dramatic, the risks that can actually hurt a business today are “often relatively mundane”, adds Loris Degioanni, founder and CTO at Sysdig. “If you don’t know where AI is running, what it can access, and who’s accountable when it gets something wrong, you’re at risk.”

Shadow AI

Shadow, unsanctioned use of AI is a particular threat companies should be aware of. Tracey Hannan-Jones, information security consulting director at UBDS Group describes how employees are often using generative AI tools such as ChatGPT, Microsoft CoPilot and Google Gemini to support their daily work. This unsanctioned use “sits entirely outside any governance framework you may have deployed”, she says.

The consequence is that most organisation do not currently have an accurate inventory of where AI is being used across their operations, what data is being processed through the tools, or what outputs are influencing real decisions.

“Without visibility, there is zero governance and without governance, there is not accountability,” warns Hannan-Jones.

When someone pastes source code, a contract or customer records into a model — or an employee gives an agent broad access so it can move quickly — they’re “potentially giving confidential information and intellectual property to a system they don’t fully control”, warns Sysdig’s Degioanni. “That’s happening inside of everyday workflows where AI governance and policies haven’t kept up with the pace of adoption.”

There’s also a risk that AI hallucinations and unreliable outputs enter real business processes. Generative AI has a well-documented tendency to produce outputs that are fluent, authoritative in tone, and “very often, factually incorrect”, says Hannan-Jones.

“This is a structural characteristic of how large language models (LLMs) generate content: They predict plausible sequences of text, but they do not verify the facts,” she explains.

Therefore, when AI-generated content enters your real business processes without adequate human oversight, errors in legal drafts, compliance reports, client communications and financial analysis can cause reputational damage and risk regulatory breach, says Hannan-Jones.

As AI models continue to improve, they’ll hallucinate less and make fewer factual mistakes. Yet the other side of that coin is that the better systems get, the more skilled they’ll become at “finding ways around the rules to accomplish whatever task you’ve given them”, says Degioanni.

This will see behaviour become “more consequential”, according to Degioanni. “That means agents may search for ways to work around controls or take shortcuts as they optimise for their goals.”

Goal-seeking is “an intrinsic strength of agent technology”, says SailPoint’s Gnanasambandam. However, when this capability operates outside a framework — when AI agents are left undiscovered and ungoverned — agents can exercise ‘bad behaviour’ such as acquiring excessive permissions or taking unintended actions, he warns.

Good Governance

Good governance can be framed as restricting the use of AI, but experts say there’s no need to put up barriers to innovation. Optimum governance enables organisations to use AI “with greater confidence”, explains Tristan Shortland, CTO, Infinity Group.

Shortland believes governance starts with “understanding where AI is being used, what data it can access, what decisions it can support and who remains accountable for the outcomes”.

“You wouldn’t give a new starter unrestricted access to every system, document and piece of sensitive information in the business on day one,” Shortland tells IO. “AI agents and copilots should be treated no differently. Access should be based on need, permissions must be controlled and activity should be auditable.”

The quality of AI outputs is also directly linked to the standard of data being provided, says Shortland. “If organisations feed AI incomplete, inaccurate or poorly governed information, they shouldn’t be surprised when they receive unreliable results. Human review, trusted data sources and regular validation remain critical parts of any responsible AI process.”

As strong starting point is the international ISO 42001 standard, which provides a “risk-based, structured framework”, says UBDS’ Hannan-Jones. “It requires organisations to establish governance structures for AI that include AI-specific risk assessments, defined roles and responsibilities for AI oversight along with the implementation and ongoing management and monitoring of AI system performance and impacts. It is not an easy certificate to attain and needs to have focus and strong governance all year round, not — dare I say — two weeks before the certification assessment.”

Some firms will already hold ISO 27001, or an equivalent information security certification, such as the National Cyber Security Centre’s Cyber Assessment Framework. “That gives you a good foundation on which to build, risk methodologies, documented policies, audit processes, management reviews, and the extension of AI governance as a targeted effort,” says Hannan-Jones.

Alongside this, Sysdig’s Degioanni recommends ensuring runtime visibility into “what these systems actually do in production”.

“Real-time monitoring is key because AI agents move quickly and they don’t always behave the way you might expect,” he warns. “You need the ability to see and react quickly when something goes awry.”