Eighteen months ago, agentic AI barely registered on the boardroom agenda of most organisations. Now 40% of large enterprises claim to be scaling up deployments, and with good reason. Eighty percent of individuals polled by McKinsey claim the technology has improved their productivity, and half say it helps them make better decisions. But it’s also ringing alarm bells across the cybersecurity community.
The stories that tend to catch the eye involve agents that have either gone rogue or been manipulated by hackers. But there’s another, little acknowledged threat that could have serious consequences for organisations operating long-running agents: silent compliance drift.
Why Governance Can’t Live in a Prompt
For a technology that barely registered on the boardroom agenda 18 months ago agentic AI is making a surprising amount of noise in cybersecurity circles. But not many people are talking about this specific governance challenge. It’s articulated well in a recent article on the topic.
The author argues that long-running agents can quietly “forget” the governance rules hardcoded into their prompts at the start of a project. The concern is that, as the AI begins to process large amounts of data, “context windows” grow to the point where the rules set down at the start get ignored in favour of newer information. It’s described as the “lost in the middle” phenomenon: a critical architectural failure that can’t simply be fixed by expanding the context window to a million tokens, or using retrieval-augmented generation (RAG) pipelines.
RAG might be able to retrieve the relevant compliance rule, but the LLM may still ignore it, the author argues. The best these efforts can do is postpone the inevitable point at which a problem occurs. They can’t turn a probabilistic model into a deterministic rule engine.
Separating Rules from Models
The article author has some clear ideas about what should happen next. He says that a deterministic policy/rules engine should live outside the LLM, and that the addition of an orchestration layer will check any actions proposed by the model against the compliance constraints this engine generates. This means that non-compliant actions will be stopped before they are executed.
To get going, organisations should check and revalidate rules during long workflows, and never assume that an initial prompt will remain effective, the author continues. Compliance controls should live in a deterministic policy engine, so actions are blocked before they hit production. And agentic working memory should be separated from the rules governing what it’s allowed to do (which must be immutable).
Richard Harbridge, Microsoft principal industry advisor at ShareGate, explains that the most concerning part of silent drift is that everything appears to be working normally. However, in reality, agents could be accessing the wrong data, taking actions outside policy, making incorrect changes, or creating retention problems. It’s an auditability as well as a compliance issue, he says.
“If something goes wrong, organisations may need to explain what the agent did, why it was allowed to do it, what permissions it had and what controls were in place at the time,” Harbridge tells IO (formerly ISMS.online). “If you cannot reconstruct that afterwards, the governance problem becomes much harder to manage.”
Daniel Meyer, CTO at Camunda, argues that one approach to the challenges outlined in the article would be to define which decisions an agent can make autonomously (and which require human approval) prior to deployment. “In practice, this means re-designing processes around what AI can do, rather than simply adapting existing processes to accommodate AI,” he tells IO.
He then advocates an orchestration layer similar to that suggested in the article, which “coordinates agents, people and systems across the end-to-end process” and has governance and approvals built into the design.
“This approach enforces controls between an agent’s decision and its action, while maintaining a full audit trail for complete visibility into agent behaviour,” Meyer says. “With structured guardrails in place, organisations can move beyond isolated agentic AI initiatives and deploy agents in business-critical processes to improve operational efficiency or customer experience, safely and with confidence.”
Delivering Continuous Governance
What might such an approach look like in practice? ShareGate’s Harbridge argues that organisations ultimately need to answer several key questions of each agent:
- Who owns it?
- What can it access and do?
- What constrains it?
- Can we see when something changes?
- Can we prove what it actually did?
The key is managing the agent continuously over time, as permissions, data sources, integrations and even purpose change, Harbridge continues. “It is not enough to believe the right controls are in place. As AI becomes more capable, organisations need the visibility to continuously validate what is actually happening,” he says
“The real governance test is not just whether an agent was safe when it started. It is whether it continues to operate within the boundaries you intended as things change around it.”
ISO 42001 – the international standard for establishing, implementing, maintaining, and continually improving an Artificial Intelligence Management System (AIMS) – can help organisations with this, providing the kind of “structured guardrails” Camunda’s Meyer advocates.
“ISO 42001 can help an organisation put clearer ownership, processes, risk management and oversight around how it uses AI,” says Harbridge. However, there’s still work to do, he acknowledges.
“What ISO 42001 does not do is tell you exactly how to technically stop a specific agent from breaking a particular rule,” Harbridge concludes. “Organisations still have to turn those governance expectations into practical controls around identity, permissions, monitoring and evidence.”
When they put these plans into action, it’s equally vital that this shouldn’t happen in a siloed way. AI governance efforts should always take place in combination with information security and privacy compliance. After all, as those agentic permissions, data sources, integrations and purpose evolve over time, an agent might create new security issues or privacy risks.
In practice, this could mean following ISO 42001 for AI, ISO 27001 for information security and ISO 27701 for privacy best practice. Do this in a continuous, joined-up way, and organisations can spot changes, reassess risk and ensure controls still work on an ongoing basis.
For those struggling to securely manage and govern a powerful new technology, that would be an invaluable starting point.
Expand Your Knowledge
Blog: How Agentic AI is Creating a New Class of Risk for Cybersecurity Teams
