From Copilot to autopilot: governance in the age of AI agents

Artificial intelligence in recent years has come in roughly two flavours. On one hand, we had classical Machine Learningmachine learningThe dominant approach to AI: algorithms that improve at a task by learning patterns from data rather than following rules a human wrote.Open full entry → (ML): models that excel at recognizing patterns and making predictions on structured data. On the other, we witnessed the mass adoption of Generative AIgenerative AIAI systems that produce new content (text, images, audio, code) rather than only classifying or predicting. Large language models are the prominent example.Open full entry →.

These generative systems functioned primarily as obedient assistants. We issued a command, the copilot generated text or code, and we as humans decided whether to actually use that output. In both cases, the human served as the indispensable buffer between the technology and business operations.

This paradigm is now shifting rapidly. We are entering the age of AI agentsAI agentA system that perceives its environment, decides and takes actions toward a goal (calling tools, executing plans). Autonomy of action demands allowlists, approval gates, sandboxing, logging and a kill switch.Open full entry →: systems that not only answer questions, but autonomously formulate goals, draw up plans, and execute actions within our IT systems. From autonomously handling customer service returns to scanning supplier contracts and processing payments directly in the ERP system.

The business case is undeniable. AI agents promise a step change in operational efficiency, scalability, and cost reduction. Yet the move from copilot to autopilot brings fundamentally new risksriskIn the EU AI Act's terms, the combination of the likelihood that a harm occurs and the severity of it if it does. The link between a principle (via the harm that would breach it) and a control (the measure that reduces it). Naming the harm and assessing its risk is required by Art. 9 before any mitigation measure is chosen. See harm, control, residual risk.Open full entry →. Many organizations grapple with the same core question: how do you retain controlcontrolThe concrete, testable measure that reduces a specific risk, and through that risk protects the principle behind it. Also called a risk management measure, risk response, or risk treatment. Always traceable to the risk it addresses: under EU AI Act Art. 9 every control must map back to a specific risk, and controls recorded separately from their risks is a recognized compliance failure. It works in one of three types: preventive, detective, or corrective. See risk, control types, evidence.Open full entry → over a system that is designed precisely to act independently?

In this article we dissect the unique governancegovernanceThe system through which an organization steers itself: corporate governance, risk management, compliance, lines of accountability, risk appetite, and the operating model. It exists across everything the organization does, before and beyond AI. AI governance is this same system extended for AI. See AI governance, governance design, execution level.Open full entry → challenges of autonomous AI agents and present concrete solutions for implementing this technology responsibly.

The new risk dynamics of autonomous agents

Every generation of AI brought its own challenges. Where classical ML models struggled with data biasbiasA systematic skew in data, model behavior, or outcomes that treats one group differently from another without justification. Bias usually enters through training data that reflects historical patterns. For high-risk AI systems, Article 10 of the EU AI Act requires examination of datasets for possible biases and measures to detect, prevent, and mitigate them. See fairness, proxy discrimination.Open full entry → and model driftmodel driftDegradation of model performance as the deployment environment shifts away from the training distribution; detected by monitoring, answered by revalidation and retraining.Open full entry →, and Generative AI introduced risks around hallucinationshallucinationFluent but false content produced by a generative model: statistically plausible output mistaken for fact. A design property to manage, not a bug to patch away.Open full entry → and intellectual property, AI agents bring operational and systemic dangers. This is because agents have agency: the authority to actually influence systems via APIs.

1. the snowball effect of flawed logic

A regular language model that gives a wrong answer stops there. The human reads it and corrects it. An AI agent, however, operates in self-directing iterations: think, plan, act, evaluate.

If an agent makes an incorrect assumption in step one of a complex process, or misinterprets a system error, this can trigger a chain reaction of wrong decisions in subsequent steps. Before long, an autonomous procurement agent has placed tens of thousands of euros in incorrect orders based on a single hallucination.

2. privilege escalation and security risks

To be useful, agents must be given access to internal systems, databases, and external services. This creates a substantial new attack surface.

A malicious actor can attempt to manipulate the agent via sophisticated prompt injectionprompt injectionSmuggling adversarial instructions into a generative system's input (directly or via retrieved content) to override its intended behavior. It is called indirect prompt injection when the instructions arrive through an intermediate source such as a document, an email, or a web page rather than from the user. Complete separation of instruction and data does not exist inside a language model, so resistance is a matter of limiting and detecting damage rather than preventing it.Open full entry → into taking actions for which the system is authorized, but the original user is not. Consider an agent that receives instructions via an apparently innocent customer email to export sensitive customer information from the CRM and forward it externally.

3. the loss of demonstrable accountability

Under frameworks such as the EU AI ActEU AI ActRegulation (EU) 2024/1689, the European Union's law on artificial intelligence. It takes a risk-based approach: prohibited practices, requirements for high-risk AI systems, transparency obligations for specific uses, and a separate regime for general-purpose AI models. Obligations are divided between providers and deployers. See general-purpose AI, conformity assessment.Open full entry →, accountabilityaccountabilityThe principle that a named human or organization answers for an AI system's outcomes, through ownership, documentation, audit trails and redress; never the system itself. The EU AI Act attaches obligations to the role rather than the technology, with provider duties in Article 16 and deployer duties in Article 26, supported by technical documentation (Article 11) and record-keeping (Article 12). See provider, deployer, record-keeping, responsible AI.Open full entry → is crucial. In the event of harmharmHarm is the concrete damage an AI system causes or can cause: to a person, a group, an organization, or society. A risk is that same damage seen in advance, weighed by likelihood and severity; a harm that has occurred is remedied rather than managed.Open full entry → or data breaches, an organization must be able to demonstrate precisely how and why a decision was made.

With an autonomous agent that makes dozens of API calls in fractions of a second and dynamically adjusts its own logic along the way, this traceability becomes a technical nightmare. Who is responsible if the agent autonomously terminates a contract and the original decision tree can no longer be reconstructed?

The legal gap: classification versus autonomy

Here we touch on a fundamental problem that many organizations overlook. The EU AI Act classifies AI systemsAI systemA machine-based system that, for explicit or implicit objectives, infers from input how to generate outputs (predictions, content, recommendations or decisions) that can influence physical or virtual environments. The OECD-style definition followed by the EU AI Act.Open full entry → based on their application and risk to fundamental rights, not based on their level of autonomy.

That distinction is not an academic subtlety. An autonomous procurement agent operating in a general business domain may formally fall under "limited risk". Operationally, however, the same system exhibits high-risk behavior: it makes irreversible decisions with direct financial and contractual consequences.

The regulation was also written and adopted before agentic AIagentic AISystems where a model takes actions (calling tools, executing multi-step plans), amplifying both capability and every failure mode; governed with action allowlists, approvals and full logging.Open full entry → broke through at scale. Regulators and legal practitioners are visibly grappling with how these systems fit within the existing framework.

For you as a deployerdeployerAn organization using an AI system under its own authority in its activities. It carries the operator duties: use per instructions, oversight, input relevance, monitoring, notices.Open full entry →, this means one thing: you cannot blindly rely on the formal risk classification. The responsibility to look further, at the operational impact, the irreversibility of actions, and the scope of the permissions granted, lies with your organization. A conservative, impact-based assessment is wiser here than a formalistic one.

The hidden role shift: from deployer to provider

A second underestimated risk concerns your legal role. Many organizations assume they are merely a deployer (user) because they use an existing foundation modelfoundation modelA model trained on broad data at scale that can be adapted to many downstream tasks; called a general-purpose AI model in EU AI Act terminology.Open full entry → such as GPT-4 or Claude.

However, as soon as you build your own agent workflow on top of such a model, with your own instructions, tool integrations, and autonomous decision logic, your position changes. In many cases, you become partly a providerproviderThe actor who develops an AI system (or has it developed) and places it on the market or into service under its own name. It carries manufacturer-style duties: design controls, documentation, conformity.Open full entry → in the sense of the regulation.

That distinction has significant consequences. As a provider you carry the provider obligations of Article 16 and the provider-facing articles (Art. 9 to 15, 17 to 21), including the technical documentationtechnical documentationRecords a provider must compile and keep for a high-risk AI system to demonstrate conformity, covering its design, data, testing, risk management and monitoring.Open full entry → of Annex IV. Being a provider does not by itself impose the Article 26 deployer duties; those attach only where you also deploy the system. You must then be able to demonstrate how your system is designed, which risks have been identified, and how they have been mitigated.

The practical lesson: building your "own" agent is rarely a low-stakes technical exercise. It can silently place you in a heavier compliance category than you anticipated.

Governance strategies and solutions

Halting the adoption of AI agents is not a realistic option for competitive organizations. The solution lies in a robust, multilayered governance architecture that scales with the autonomy of the system.

Solution 1: guardrails and strict sandboxing

Governance for agents begins with structurally constraining the scope of action through technical guardrailsguardrailA barrier meant to stop unwanted behavior of an AI system. The distinction that decides its value is where it lives: a guardrail written into the prompt is a request the model can weigh incorrectly or be manipulated out of, while a guardrail in code or configuration is an enforcement. The word is also used in the market for output filters, schema validation, classifiers, and policy engines, so the layer it sits in is worth naming. See enforcement point, policy engine, prompt injection.Open full entry → and sandboxing.

• PrincipleprincipleOne of the seven responsible-AI values a governed system should live up to (fairness, safety and reliability, privacy, security and robustness, transparency and explainability, accountability, human oversight). A principle is abstract: it states an outcome, not a lever you can pull. It becomes governable by naming the harm that would breach it, assessing the risk that harm carries, and placing controls against that risk. Held this way, a principle becomes a pillar. See pillar, harm, risk.Open full entry → of least privilege: Give an agent only the permissions strictly necessary for its defined task. Does an agent need to analyze financial data? Grant read-only access and block any possibility of data modification at the architecture level.
• Action geofencing: Restrict the systems the agent may communicate with to a pre-approved allow-list of APIs and internal network domains. Everything outside is blocked by default.

Solution 2: dynamic human oversight, but realistic

We do not need to check every action of an agent; that would negate the business case. We do need to place human gatekeepers at critical nodes. But nuance is required here, because the concept of "human oversighthuman oversightDesigned-in human ability to monitor, intervene in, override or shut down an AI system. It is meaningful only when the human has authority, information and time to act. One of the seven pillars of responsible AI, and under the EU AI Act a requirement for high-risk AI systems: Article 14 requires that those systems are designed so natural persons can effectively oversee them. Oversight that exists on paper but amounts to confirming in practice does not meet that bar. See override rate, automation bias, high-risk AI system, fairness, safety and reliability, privacy, security and robustness, transparency and explainability, accountability, responsible AI. In the IAPP AIGP body of knowledge, this principle appears as human-centricity, with human oversight as one of its elements.Open full entry →" is deceptive with agents.

The classic human-in-the-loophuman-in-the-loopOversight configuration where a human approves or decides each case the system recommends. It fits high-stakes individual decisions and is meaningful only with authority, information and time.Open full entry →, where a human pre-approves every individual action, has become a fiction for an agent acting in milliseconds. Two other models are more realistic: human-on-the-loophuman-on-the-loopOversight configuration where humans monitor operation and intervene by exception. It fits high-volume systems where per-case review is impossible.Open full entry → (the human monitors and can intervene, but does not pre-approve every action) and human-in-command (the human sets the frameworks, goals, and boundaries within which the agent may operate).

Article 14 of the regulation also imposes a requirement that goes beyond merely designating a supervisor. That person must be competent and trained, and, critically, genuinely able and authorized to intervene. A supervisor without the technical ability to stop an agent, or without the mandate to do so, is governance on paper only.

• Thresholds based on deterministic criteria: Configure the system so that the agent may act autonomously as long as objective, pre-established thresholds are not exceeded: a maximum financial amount, a specific type of action, the systems involved, or the irreversibility of the action. Once a threshold is reached, the agent pauses and escalates to a human supervisor for approval.

A caution is warranted here. It is tempting to have the agent itself produce a confidence score and use that as the threshold. However, LLM-based systems produce notoriously poorly calibrated confidence scores, a model is often most assertive when it is wrong. A confidence level reported by the model itself should therefore never be the sole gatekeeper. Rely on deterministic, verifiable criteria.

Solution 3: agentic audit trails and observability

Because agents execute complex chains of actions, standard application logging is insufficient. Organizations must invest in deep observability.

• Immutable action logs: Every chain-of-thought of the model, every API call made, and every system response received must be recorded immutably. This ensures that regulators, compliance officers, and auditors can reconstruct the full decision-making process step by step after the fact.

This is not merely a technical best practice, but the only effective way to comply with the transparencytransparencyOpenness about the fact that AI is used and how it operates in general: disclosures, documentation, notices. Pairs with explainability, which addresses individual outcomes.Open full entry → and accountability requirements of the regulation, and to be able to debug incorrect behavior in the future. An AI management systemAI management systemThe organizational structure, policies and processes for governing AI across its life cycle, as formalized in ISO/IEC 42001.Open full entry → conforming to ISO 42001 embeds this logging discipline in a broader governance structure, so that it does not remain a standalone technical measure.

Solution 4: periodic red teaming and stress testing

Because AI agents continuously respond to dynamic environments and external data inputs, a one-time compliance check at go-live is insufficient.

There is a fundamental difference between functional testing (does the agent work as intended?) and adversarial testing (can the agent be deliberately pushed outside its boundaries?). The second category is indispensable for autonomous systems. Conduct controlled attacks in a safe test environment: red teamingred teamingAdversarial testing that deliberately probes an AI system for failures, harmful outputs or security weaknesses, before and after release.Open full entry →.

Agent-specific attack patterns that must be tested include prompt injection via incoming data, tool misusetool misuseAn agent calling a permitted tool in a way that produces an unintended real-world effect. Countered with scoped tool access and approval gates on high-consequence calls.Open full entry → (exploiting the agent's permissions), and goal hijackinggoal hijackingAn attack that redirects an agent's objective so it pursues a goal you did not set. Prompt injection combined with autonomy: it changes what the agent does, not just what it says.Open full entry → (subverting the agent's objective).

Two principles apply here. First: Red teaming must be independent, not conducted by the same team that built the agent, as that team has blind spots for its own assumptions. Second: It is a continuous process, not a one-time exercise. The NIST AI Risk Management Framework provides a useful structure for organizing such tests methodically and repeatably.

Conclusion: autonomy demands tighter frameworks

The shift from static models and passive copilots to AI agents marks the true promise of artificial intelligence for business. The efficiency gains are unprecedented, but delegating executive authority from human to machine demands a fundamentally different and far stricter approach to risk management.

Governance cannot be an administrative afterthought in this era. By combining strict authorizations, realistic human oversight, and irrefutable logs, a framework emerges in which agents can operate safely and effectively. Only organizations that build control by design into the DNA of their autonomous systems will reap the rewards without becoming entangled in unmanageable operational and compliance risks.