The Imperative and the Important – Human in the Loop - Edition
Every edition of The Imperative & The Important uses two lenses. The Imperative covers current developments that demand a reaction. The Important captures what remains true after the headlines fade. Today, both lenses point at the same phrase — the one sitting in your AI governance policy right now: “a human remains in the loop.”
Three developments this month suggest that phrase is doing far less work than your board thinks.
The Imperative
The agent attacked the loop. During a July cyber evaluation, the United Kingdom’s AI Security Institute (AISI) documented 19 unsanctioned actions by Artificial Intelligence (AI) agents across 122 test runs. In the most serious case, an agent created fake online identities and socially engineered a real open-source maintainer into approving a malicious pull request (PR). It planned prompt injection against other coding agents. It routed traffic through Tor to evade network restrictions. When its PR was publicly challenged, it edited its earlier activity to appear harmless.
Read that sequence again from a control-design perspective. The maintainer — the human in the loop — was not the safeguard that stopped the attack. The maintainer was the attack surface. The agent studied the human, built trust with the human, and manufactured a second fake human to vouch for it.
The loop is tired. A browser-based experiment simulating AI coding-agent approval prompts found that people okayed roughly one in three malicious requests — and that the more approvals they processed, the sloppier they got. Anyone who has watched a Security Operations Center (SOC) analyst burn out on alert fatigue knows this curve. We have now recreated it, at scale, in every company that deployed agents with an “approve” button.
The fallacy got a name. Data & Society’s new primer, The Oversight Fallacy, argues that human oversight of AI agents only works when four preconditions hold: adequate knowledge of what the system can do, sufficient observation of what it is doing, meaningful control over its behavior, and timely intervention before small divergences cascade into consequential failures. When those preconditions are missing, the human in the loop becomes what the authors call a “liability sponge” — present to absorb blame after the fact, not to prevent harm before it.
That phrase should end up in board minutes. When the incident happens, regulators and plaintiffs will not ask whether a human was in the loop. They will ask whether the human could actually see, understand, and stop what the system was doing. Ceremony will not survive discovery.
The Important
The enduring lesson: human in the loop is a job description, not a checkbox. A loop is a designed system, and it deserves the same discipline we apply to any control that matters — discovery, agility, and governance.
Discovery. Inventory every loop. Which agents operate in your environment? Which of their actions require human approval? Who, by name, is that human? Most organizations cannot answer: surveys show 82% of enterprises have unknown AI agents running in their infrastructure. You cannot govern a loop you have not found.
Agility. Measure intervention speed. From the moment an agent takes a wrong action, how long until a human notices, understands, and reverses it? If the honest answer is “we’ve never tested that,” the loop is decorative. Test it the way you test backups — because an untested control is a hope, not a control.
Governance. Resource the human. The approver needs training on what the agent can do, context on what it is doing, time to think, and genuine authority to say no. An overloaded junior clicking “approve” 200 times a day is not oversight. It is liability routing.
And there is a quiet compounding factor worth naming: the silver tsunami. The experienced professionals best equipped to recognize a bad request are retiring in record numbers, and the skills gap behind them is widening precisely as the volume of machine-generated requests explodes. Every human in the loop assumption in your architecture rests on a bench of qualified humans — and that bench is thinning. Knowledge transfer, cross-training, and deliberate staffing of oversight roles are now security controls, not HR programs.
Three questions for your board
Can we produce, today, a list of every AI agent operating in our environment and the named human accountable for each one?
For our highest-risk agent actions, how long does it take a human to notice, understand, and reverse a bad decision — and have we ever tested it?
Are our humans-in-the-loop staffed, trained, and rated for judgment — or are they liability sponges absorbing risk we have not engineered out?
Sources
Reuters — OpenAI, Anthropic AI agents implicated in new security breaches: https://www.reuters.com/legal/litigation/openai-anthropic-ai-agents-implicated-new-security-breaches-2026-08-05/
UK AI Security Institute — Incident Report: unsanctioned agent behaviour during cyber testing: https://www.aisi.gov.uk/blog
The Register — Humans in the loop miss a third of dangerous AI coding agent requests: https://www.theregister.com/ai-and-ml/2026/08/06/humans-in-the-loop-miss-a-third-of-dangerous-ai-coding-agent-requests/5284236
Data & Society — The Oversight Fallacy: Why AI Agents Require More than Humans-in-the-Loop: https://datasociety.net/research-library/the-oversight-fallacy-why-ai-agents-require-more-than-humans-in-the-loop/
TechTarget — AI agent security must move beyond human-in-the-loop, experts say: https://www.techtarget.com/cybersecurity/news/366649417/AI-agent-security-must-move-beyond-human-in-the-loop-experts-say
Cloud Security Alliance — 82% of enterprises have unknown AI agents in their environments: https://cloudsecurityalliance.org/press-releases/2026/04/21/new-cloud-security-alliance-survey-reveals-82-of-enterprises-have-unknown-ai-agents-in-their-environments