Agentic AI and governance: why autonomy sharpens the control question

The vocabulary of agentic AIagentic AISystems where a model takes actions — calling tools, executing multi-step plans — amplifying both capability and every failure mode; governed with action allowlists, approvals and full logging.Open full entry → is everywhere right now: harness, loop, context, memory, tool use, orchestration, guardrails, human in the loop. These terms describe how an agent works, the engineering that lets a model do tasks on its own rather than just respond to questions. They are worth knowing. But knowing what an agent is made of does not tell you how to keep it under controlcontrolThe concrete, testable measure that reduces a specific risk, and through that risk protects the principle behind it. Also called a risk management measure, risk response, or risk treatment. Always traceable to the risk it addresses: under EU AI Act Art. 9 every control must map back to a specific risk, and controls recorded separately from their risks is a recognized compliance failure. It works in one of three types: preventive, detective, or corrective. See risk, control types, evidence.Open full entry →, and that is the question an organization has to resolve before it deploys one.

This article takes the common agentic vocabulary as a starting point and asks the governancegovernanceThe system through which an organization steers itself: corporate governance, risk management, compliance, lines of accountability, risk appetite, and the operating model. It exists across everything the organization does, before and beyond AI. AI governance is this same system extended for AI. See AI governance, governance design, execution level.Open full entry → question the vocabulary skips: when an AI systemAI systemA machine-based system that, for explicit or implicit objectives, infers from input how to generate outputs — predictions, content, recommendations or decisions — that can influence physical or virtual environments. The OECD-style definition followed by the EU AI Act.Open full entry → stops merely producing output and starts taking actions on its own, what happens to control, oversight, evidenceevidenceThe concrete proof that a control is designed, implemented, and working: a test report, an audit trail, an impact assessment, a monitoring log. Each link in the governance chain produces an artifact, and together they are what an organization hands to its own board, a regulator, a customer, or an affected person to show, not say, that a system is governed. Its absence is itself the failure: a risk register without test results, or a mitigation claimed without validation, is a governance gap, not a paperwork one. The closing link of the governance chain. See control, governance.Open full entry →, and accountabilityaccountabilityThe principle that a named human or organization answers for an AI system's outcomes, through ownership, documentation, audit trails and redress — never the system itself.Open full entry →? The short version: autonomy does not create a new kind of governance. It widens the gap between what the system does and who is accountable for it, and that makes the existing governance chaingovernance chainThe traceable line by which a single pillar is held for a single system: principle, then the harm that would breach it, then the risk that harm carries, then the control that reduces the risk (preventive, detective, or corrective), then the residual risk judged against appetite, proven with evidence. The chain is what makes responsible AI accountable rather than aspirational, and what lets an organization move a principle from a policy statement to a working control it can point to. See principle, harm, risk, control, residual risk, evidence.Open full entry → more important, not less.

What changes when a system becomes agentic

A traditional AI system produces an output: a score, a classification, a piece of text. A human reads it and decides what to do. The decision, and the accountability for it, sits clearly with the person.

An agentic system narrows that gap or removes it. It does not just suggest; it acts. In the common vocabulary: tool use means the agent queries the database, sends the email, or updates the record itself; a loop means it repeats the act-check-decide cycle without a human approving each step; orchestration means several of these actions chain together across tools. The point of all this engineering is to let the system work without a person checking every step. That is exactly the capability that makes it useful, and exactly the capability that creates the governance problem.

Because here is what moves: in a traditional system, the human is the control point by default, standing between the output and the action. In an agentic system, that human is no longer automatically in the path. The control point does not disappear; it has to be designed in. If you do not design it, you have a system taking real actions with no one positioned to catch the one you cannot undo.

The control question: guardrails are controls, not features

In agentic vocabulary, guardrails are "the limits on what an agent is allowed to do." That is accurate as far as it goes, but it frames a guardrail as a product feature, a setting. In governance terms, a guardrail is a control: a deliberate constraint that traces back to a specific riskriskIn the EU AI Act's terms, the combination of the probability that a harm occurs and the severity of it if it does. The link between a principle (via the harm that would breach it) and a control (the measure that reduces it). Naming the harm and assessing its risk is required by Art. 9 before any mitigation measure is chosen. See harm, control, residual risk.Open full entry → and produces evidence that it held.

The distinction matters because it changes how you reason about it. A feature is something you switch on. A control is something you can point to and say: this addresses that risk, it is preventive (it stops the action before it happens) rather than detective (it notices afterward), and here is the log that proves it was in force when the agent ran. The "something you can't undo" that everyone worries about with agents is, in governance language, an irreversible-harmharmThe concrete damage an AI system can do that a responsible-AI principle exists to prevent: in the EU AI Act's terms, harm to a person's health, safety, or fundamental rights. Harm is the bridge between an abstract principle and a governable risk; governance becomes operational the moment an organization names the specific harms it wants to prevent. For fairness, a harm is a group receiving systematically worse outcomes because of a characteristic that should not have counted. See principle, risk.Open full entry → risk, and an irreversible-harm risk is precisely the kind that demands a preventive control with evidence behind it, not a setting someone hopes is enabled.

This is the GovCompass model applied to agents: every guardrail is a control, every control traces to a risk and forward to evidence. Agents do not change that logic. They raise the stakes on it, because the actions are real and sometimes irreversible. (For the full chain, see what AI governance is.)

The oversight question: "human in the loop" is not automatically oversight

The vocabulary offers human in the loop: "a person approves before the agent does something that matters." This is the most over-trusted term in the agentic vocabulary, because the phrase describes a position and quietly implies it provides control, when those are not the same thing.

The EU AI Act, in Article 14, requires that high-risk AI systems be designed so they can be effectively overseen by people. The operative word is effectively. Putting a human in the loop satisfies the org chart; it does not satisfy Article 14 unless that human can meaningfully intervene. A person who has to approve forty agent actions an hour, with no real ability to evaluate each one, is not oversight. A person who approves because the interface nudges them toward "yes" and gives them no basis to say "no" is not oversight. That is automation biasautomation biasThe human tendency to over-trust automated outputs — accepting a system's recommendation without genuinely weighing the case, which hollows out human oversight.Open full entry → wearing the costume of control, and the Act names it directly: Article 14 requires that overseers be able to stay aware of the tendency to over-rely on a system's output. It is one of the most common ways agentic deployments fail the effectiveness test.

Meaningful oversight needs four things the phrase "human in the loop" does not guarantee: the authority to override without reprisal, the expertise to judge the agent's action, the information to understand what the agent is about to do, and the time to evaluate it. Strip any one of those and the human is a rubber stamp, and the system is uncontrolled regardless of who is nominally in the loop.

Notably, the Act ties the weight of oversight directly to autonomy. Article 14 requires oversight measures to be commensurate with the risks, the level of autonomy, and the context of use. The law itself says, in other words, that the more autonomous the system, the more oversight it demands, which is exactly the point an agentic deployment cannot afford to miss.

The evidence question: evals are evidence, not just tests

In the vocabulary, evals are "a scored set of tasks you test an agent against, like unit tests for AI." The analogy is good for engineers. The governance reframing is that evals are evidence: not only a development-time check that the agent works, but part of the record that demonstrates, to a regulator or an auditor, that the system did what you claimed and was tested against the risks that matter.

The difference between a test and evidence is durability and purpose. A test tells you, today, whether the thing works. Evidence is what you can produce months later when someone asks you to prove the system was fit for use, behaved within its limits, and was monitored after launch. Agents make this harder, because their behavior depends on context and can drift, so the evidence cannot be a one-time snapshot. It has to be the kind of ongoing record that the governance chain is built to produce.

The accountability question: someone is accountable, regardless of autonomy

This is where agentic AI most tempts people into a category error: the more autonomous the system, the easier it is to talk as if the agent is responsible. It is not. Accountability does not transfer to software.

The EU AI Act is explicit on this for deployers. Article 26 makes the deployerdeployerAn organization using an AI system under its own authority in its activities — carrying operator duties: use per instructions, oversight, input relevance, monitoring, notices.Open full entry → accountable for the high-risk AI system it puts into use, regardless of who built it, including the duty to assign human oversighthuman oversightDesigned-in human ability to monitor, intervene in, override or shut down an AI system — meaningful only when the human has authority, information and time to act.Open full entry → and to retain the system's logs. Autonomy does not dilute that. An agent that acts on its own is still an agent someone chose to deploy, configured, and is accountable for. The harder the system is to trace, the more the burden falls on the deployer to have built the controls, the oversight, and the evidence that show it was under control.

The practical consequence: as a system becomes more agentic, the question "who is accountable when this causes harm" does not get fuzzier; it gets more demanding. You cannot point at the autonomy and shrug. You have to point at the control that should have caught it, the oversight that should have intervened, and the evidence that shows whether either was in place.

Why this makes governance more important, not less

It is tempting to think that a system clever enough to act on its own needs less governing. The opposite is true. Every property that makes an agent useful, acting without a human in the path, chaining tools together, improving with memory, operating in a loop, also widens the distance between the action and the accountable person. Governance is what closes that distance: the control that constrains the action, the oversight that can stop it, the evidence that proves what happened, and the named person who is accountable for it.

Agentic AI does not need a new governance model. It needs the existing one applied with more rigor, because the actions are real, sometimes irreversible, and harder to trace. The vocabulary tells you how agents work. Governance tells you how to stay in command of them, and that is the part that decides whether deploying one is responsible.

Continue reading

• What is AI governance
• Agentic AI: the integrating element