The control environment for agentic AI: where controls belong and how you prove they work

Updated: July 2026

The question this article answers

An organization moves a business process to an AI agentAI agentA system that perceives its environment, decides and takes actions toward a goal (calling tools, executing plans). Autonomy of action demands allowlists, approval gates, sandboxing, logging and a kill switch.Open full entry →, and the controlscontrolThe concrete, testable measure that reduces a specific risk, and through that risk protects the principle behind it. Also called a risk management measure, risk response, or risk treatment. Always traceable to the risk it addresses: under EU AI Act Art. 9 every control must map back to a specific risk, and controls recorded separately from their risks is a recognized compliance failure. It works in one of three types: preventive, detective, or corrective. See risk, control types, evidence.Open full entry → that governed that process have to move with it. The approval limit, the segregation of duties, the mandatory sequence, the human sign-off: all of them were designed for a process performed by people and by conventional software. The question is not whether they still apply. It is where they now live, and how anyone can demonstrate that they work.

That question has an uncomfortable first answer. Part of an agent cannot carry a control at all, because part of an agent does not behave the same way twice.

What an agent is made of

An AI agent is a small stack of parts. At the center sits the language model, which reads, reasons, and decides. Around it sits conventional software: the orchestrationorchestrationThe fixed program code around an AI model that determines which steps run in which order, what is mandatory, and where a human is involved. Because it is conventional software, it behaves predictably and can carry enforceable controls, which is why process structure belongs there rather than in prompt text. See enforcement point, guardrail.Open full entry →, code that determines which steps run in which order and where a human is involved. The agent acts through tools, functions it calls to do or request something, each bounded by the permissions granted to it. It receives its assignment through a system instruction, informs itself through retrieved knowledge fetched per task, and may carry memory of earlier steps or runs.

Which of those parts holds a given rule is the subject of this article.

The distinction that decides everything

The parts split into two kinds, and the split determines what kind of assurance is even possible.

Deterministic parts are conventional software: the orchestration, the checks inside tools, the access rights. Their rules are explicitly coded, so given the same relevant inputs, state, and configuration, they apply those rules predictably and reproducibly instead of interpreting them anew on each run. If the payment tool is configured to refuse amounts above 10,000 euros, it applies that limit today, tomorrow, and on the thousandth invoice.

Probabilistic parts are everything that passes through the language model: how it reads the instruction, what it does with a retrieved document, which tool it decides to call and with which values. The model chooses the most likely continuation; it does not follow fixed rules. The same input can produce two slightly different results.

An auditor already knows both types:

• A deterministic component behaves like an application control. You test it the way you test a system limit. A negative test shows that the barrier blocks the prohibited action on the path you tested, and the outcome of each test is binary: the action must fail. Assurance comes from covering every material path and configuration, not from a single test.
• A probabilistic component behaves like a human performer. You cannot prove that a person always follows the procedure. You take a sample, you test scenarios, you measure how often it goes right, and you set a threshold for what is acceptable.

As the working principleprincipleOne of the seven responsible-AI values a governed system should live up to (fairness, safety and reliability, privacy, security and robustness, transparency and explainability, accountability, human oversight). A principle is abstract: it states an outcome, not a lever you can pull. It becomes governable by naming the harm that would breach it, assessing the risk that harm carries, and placing controls against that risk. Held this way, a principle becomes a pillar. See pillar, harm, risk.Open full entry → goes: you test the knowledge layer the way you test an employee, not the way you test a lock. A control placed in a deterministic layer can, when it is correctly designed, implemented, and operated, provide enforceable assurance. A rule that lives only in a probabilistic layer cannot offer that guarantee. It can be made likely, which means it needs a substantiation for why likely is enough.

Why a rule in the prompt is not a control

The system instruction is the tempting place to put rules. It is text, it is quick to change, and it reads like policy. "Never approve a payment above 10,000 euros" in the system instruction looks like a limit.

It is not a limit. It is a request to a probabilistic systemprobabilistic systemA system whose outputs are statistical predictions with error rates: designed to be wrong some fraction of the time, which testing measures rather than eliminates.Open full entry →. The model can weigh it incorrectly, lose it among competing instructions, or be manipulated out of it through content it processes. And here is the trap that makes this worse than a plain gap: the agent may comply often enough for limited testing to create confidence. Measured over a modest number of runs, the rule appears to hold. That is an inadequate design operating flawlessly, and it produces false assurance. The organization believes it has a limit; it has a habit.

The rule that resolves this is short. Knowledge may live in the probabilistic layers. Controls must move to the deterministic layers. The description of the process, how a request is handled, what tone to use with a customer, what information belongs in the file, can live in the system instruction and in retrieved documents. But every control receives an enforcement pointenforcement pointThe place where a rule is applied in code or configuration rather than described in text, so that breaking it does not depend on a system choosing to comply. In access control the equivalent is the policy enforcement point. For an AI agent the candidates are the tool, the policy engine, the permissions, and the orchestration; the prompt is not one of them. See guardrail, policy engine, control objective.Open full entry → outside the model: in the tool, in the policy enginepolicy engineA component that applies recorded rules automatically to every action, for example a requirement that payments above a threshold need separate approval. Because the rule sits outside the model, it cannot be ignored or reasoned around by the model. Recording those rules as machine-readable code, kept under version control like any other production code, is known as policy-as-code.Open full entry →, in the permissions, or in the orchestration. An enforcement point is the place where a rule is applied in code or configuration rather than described in text, so that breaking it is not a matter of the model choosing to comply.

One qualification belongs with that rule. Some requirements cannot be made deterministic at all: judging whether a reply is misleading, whether a justification is complete, whether a case is unusual. Where that is so, the correct route is to treat the requirement as probabilistic performance, with a threshold, monitoring, a fallback, and accepted residual riskresidual riskThe risk that remains after controls have reduced it. No control reduces a risk to zero, and not every control is worth its cost, so a deliberate judgment is made: whether the cost of further control is justified by the reduction it would buy, and whether the remaining risk is acceptable against the organization's risk appetite. This is a design-level judgment, where execution reports back up and governance accepts the residual risk, calls for more control, or declines the use case. EU AI Act Art. 9(5) requires it to be judged acceptable per hazard and overall. See risk, control, risk appetite.Open full entry →, rather than to present it as an enforced barrier.

The prompt layer is not worthless. It is the first line of defense, because good instructions reduce how often the agent reaches the barriers at all. But it is defense in depth, never the only layer. The IMDA Model AI Governance Framework for Agentic AIIMDA Model AI Governance Framework for Agentic AIA governance framework for agentic AI published by Singapore's Infocomm Media Development Authority, first released in January 2026 and updated in June 2026. It decomposes agents into models, memory, tools, and actions, and states a preference for structural and rule-based controls over guardrails that live in the prompt layer. It is guidance, not law. See agentic AI, guardrail, least agency.Open full entry →, from Singapore's Infocomm Media Development Authority, states the same preference from the governancegovernanceThe system through which an organization steers itself: corporate governance, risk management, compliance, lines of accountability, risk appetite, and the operating model. It exists across everything the organization does, before and beyond AI. AI governance is this same system extended for AI. See AI governance, governance design, execution level.Open full entry → side, for structural and rule-based controls over prompt-layer guardrailsguardrailA barrier meant to stop unwanted behavior of an AI system. The distinction that decides its value is where it lives: a guardrail written into the prompt is a request the model can weigh incorrectly or be manipulated out of, while a guardrail in code or configuration is an enforcement. The word is also used in the market for output filters, schema validation, classifiers, and policy engines, so the layer it sits in is worth naming. See enforcement point, policy engine, prompt injection.Open full entry →.

Two families, and why they are complete together

A control environment for an agent contains two kinds of controls. The first kind is easy to find and the second kind is easy to forget.

• Process controls are the controls the business process already had: limits, segregation of duties, mandatory sequence, dual approval, data restrictions. The 10,000 euro limit existed long before anyone considered an agent. It has an owner, it appears in an audit file, and moving to an agent does not remove it. It moves its enforcement point.
• Agent controls are the controls for risksriskIn the EU AI Act's terms, the combination of the likelihood that a harm occurs and the severity of it if it does. The link between a principle (via the harm that would breach it) and a control (the measure that reduces it). Naming the harm and assessing its risk is required by Art. 9 before any mitigation measure is chosen. See harm, control, residual risk.Open full entry → that appear in no procedure, because they only exist once an agent runs the process: resistance to manipulation through processed content, memory policy, restriction of the toolset, integrity of tool definitions and third-party connections, an agent identity with least privilege. Nobody wrote a procedure about hidden text in an incoming invoice, because until now an invoice could not give instructions to anyone.

At the level of process execution the two families cover the ground, because they follow two different questions. Process controls answer what this process always needed to keep working correctly. Agent controls answer what this process now needs because a probabilistic system with tools performs it. A control mapping that contains only process controls looks complete and by definition is not, which is why the register labels every control with its family. The wider control environment holds more than these two families: the governance arrangements later in this article are what keep both of them working.

Moving process controls: the translation table

Every wrong answer below has the same form. This article calls it a prompt-layer guardrail: a rule expressed in prompt text without an independent enforcement point. The word guardrail is used more broadly in the market, for output filters, schema validation, classifiers, and policy engines among others, so the qualifier matters. The prompt-layer variety is the most common way organizations believe they have moved a control when they have only described it.

Control in the business process	Wrong enforcement point: the prompt-layer guardrail	Correct enforcement point
Approval limit: above 10,000 euros extra approval	Sentence in the system instruction	Limit check in the payment tool or policy engine; the tool refuses above the threshold
Segregation of duties: whoever enters does not approve	Instruction "do not approve your own entry"	Permission model: the agent identity that enters has no approval right
Mandatory sequence: first check, then booking	Step plan in the prompt	Orchestration, or a condition check inside the tool when the agent plans its own steps
Human approval for impactful actions	"Ask for confirmation when in doubt"	Orchestration or tool: the action waits until a human with the correct role approves
Data restriction: only files of your own department	"Only look at your own department"	Access rights on the data source: other files are technically unreachable
Prohibited action: never external mail with customer data	"Never send customer data externally"	The mail tool does not exist in the toolset, or filters destinations against an allowlist

The pattern repeats in every row. The enforcement point sits outside the model. The model may want what it wants; the environment determines what is possible.

One nuance on the tool layer: the check inside a tool is deterministic and enforceable, but whether the tool is called, and with which values, is a model decision. The tool layer guarantees what can never happen, such as paying above the limit. It does not guarantee what always happens, such as choosing the correct invoice. The same split runs through the tool definition itself: the executing code and its parameter checks are deterministic, while the model's reading of the tool description, its choice to call that tool, and the values it constructs are not. Both sides need verification, and they need different kinds.

Agent controls: the core set

For a first deployment, five agent controls cover most of the exposure.

• An agent identity of its own, with least privilege. The agent is not a person and should not borrow a person's account. Where it acts on behalf of a user, its effective authority per action should be no broader than the intersection of what the agent may do and what that user may do. Where it runs a process rather than serving an individual, a dedicated service identity with bounded process rights is the cleaner arrangement.
• A restricted toolset with bounded parameters. Every tool is a capability, so a tool that does not exist is a capability the agent cannot misuse. That is a stronger control than any instruction. Broad tools that accept free-form queries hand the agent effectively unlimited power in a system.
• Limits on the influence of untrusted content. These are the measures against indirect prompt injectionprompt injectionSmuggling adversarial instructions into a generative system's input (directly or via retrieved content) to override its intended behavior. It is called indirect prompt injection when the instructions arrive through an intermediate source such as a document, an email, or a web page rather than from the user. Complete separation of instruction and data does not exist inside a language model, so resistance is a matter of limiting and detecting damage rather than preventing it.Open full entry →, one of the central risks across OWASPOWASPThe Open Worldwide Application Security Project, a nonprofit foundation that publishes open security guidance. Two of its outputs matter for AI governance: the OWASP Top 10 for LLM Applications, which lists prompt injection as its first entry, and the OWASP Top 10 for Agentic Applications, which covers the security risks specific to agents that use tools and act autonomously. See prompt injection, least agency, agentic AI.Open full entry →'s LLM and agentic security work: hidden instructions inside content the agent processes, an incoming invoice, an email, a web page. Content of that kind may inform a decision, but it should not be able to steer tool behavior. In practice this is a combination: marking external content as untrusted, separating content processing from action authorization, restricting tool rights, validating tool calls, and monitoring the attempts that succeed.
• Memory policy. What is retained, for how long, and who can inspect and clear it. An outdated supplier exception held in memory can shape every approval that follows, without anyone seeing where the influence came from.
• Integrity checks on tool definitions and third-party connections. Whoever controls a tool description influences the agent's behavior, which makes third-party tool sources a supply chainsupply chainThe layered chain behind an AI product (foundation models, datasets, labeling services, integrators), each layer adding risk the buyer never contracted for directly.Open full entry → matter, checked at every change.

One caveat belongs here, because it changes how the register reads. Complete separation of instruction and data does not exist inside a language model, so resistance to manipulation through processed content is never deterministic. What can be deterministic is the damage limit: the toolset, the limits, and the permissions determine what a successful manipulation can achieve at most. This is a risk you limit and detect rather than eliminate, and it therefore carries an explicit, accepted residual risk rather than a claim of prevention.

Design effectiveness: is the control in the right place

With the enforcement points assigned, the first assurance question comes before any testing: is this control capable of mitigating the risk it is meant to mitigate. For an agent, that question takes a different form per layer.

• For a control in a deterministic layer: does the enforcement point sit outside the model, does it cover every path to the action, and does it fail closed or move into a predefined safe state. The second part is where designs fail without anyone noticing. A limit inside the approval tool is worthless if a second tool can approve in bulk without passing the same check, and an agent that plans its own steps will find that second path eventually.
• For a control left in a probabilistic layer: why is that acceptable. The rule of thumb is narrow. A control may stay in the probabilistic layer only if the residual harmharmThe concrete damage an AI system can do that a responsible-AI principle exists to prevent: in the EU AI Act's terms, harm to a person's health, safety, or fundamental rights. Harm is the bridge between an abstract principle and a governable risk; governance becomes operational the moment an organization names the specific harms it wants to prevent. For fairness, a harm is a group receiving systematically worse outcomes because of a characteristic that should not have counted. See principle, risk.Open full entry → is low or reversible, or if a deterministic layer underneath still catches the failure.

That substantiation is where the design question becomes a governance act rather than a technical note. Every control accepted in the probabilistic layer needs a named role that accepts the residual risk, with a date. Without that signature the substantiation is an intention; with it, it is a decision the organization owns.

Two things make the design assessment complete rather than plausible:

• It starts from a risk analysis of the process as the agent will run it, not from the old procedure. An agent changes the risk profile: errors scale, natural hesitation disappears, new attack surface appears, and errors become correlated instead of dispersed.
• It starts from a walkthrough with the people who do the work today, because processes contain implicit controls that appear in no procedure. The instinct "I do not trust that supplier, I call them" is a control, even if it is written nowhere. Whoever translates only the paper procedure, translates the paper process.

Operating effectiveness: does it work, and how would you know

Once the design holds, the testing splits along the same line that runs through the whole design.

Deterministic controls: binary testing. This is classic application control testing, and much of its execution can be automated. Negative tests: try to pay above the limit, approve your own entry, skip the mandatory step, read outside the data scope. The expected result is failure, every time, one hundred percent. Because these tests are automated, they belong in the same pipeline that runs on every change, so that every modification to a prompt, a tool, or the process passes all barriers again.

Probabilistic controls: statistical testing. Here the instrument is an evaluation setevaluation setA fixed collection of test scenarios with predetermined desired behavior, used to measure the behavior of a model or agent repeatably. It is the agent equivalent of a regression test, and it produces a score against a threshold rather than a pass or fail, because the behavior it measures is probabilistic. See drift, override rate.Open full entry →: a fixed collection of realistic scenarios with predetermined desired behavior, producing a score against a threshold. Five properties make an evaluation set a controlled artifactartifactThe concrete record that proves a control was carried out: a test report, an impact assessment, a monitoring log, a release sign-off. An artifact is the tangible form evidence takes, the thing an auditor reaches for to confirm that a control was not just designed but actually operated. Each stage of the AI life cycle produces its own anchor artifact. Distinct from evidence as a whole: evidence is the proof, an artifact is one piece of it. See evidence, life cycle.Open full entry → rather than a demo:

• Independence. Challenged or reviewed outside the build team, proportionate to the risk, because otherwise the maker tests their own work.
• A threshold from the risk analysis, not from a feeling about what sounds high.
• RepresentativenessrepresentativenessHow well training data reflects the population and conditions the system will face in deployment, the fitness-for-purpose core of AI data quality.Open full entry →, including the edge cases from the walkthrough.
• Maintenance, because a test set that ages while the process moves proves nothing.
• Follow-up on failures, with analysis and a measure, not only a score.

Run on a fixed rhythm and after every change, the evaluation set stops being a release test and becomes one component of continuous monitoringcontinuous monitoringOngoing observation of a deployed system's performance, drift, fairness and usage against thresholds with named owners. It is the control that matches AI's speed and scale.Open full entry →. It is not the whole of it: production signals belong beside it, because a fixed set can miss shifts in the real case mix. Together they catch behavioral driftdriftThe gradual divergence of an AI system's behavior or performance from its validated state after deployment, without any code change or error. Drift is silent by nature: nothing breaks, accuracy simply decays until someone measures it. Causes are worth separating: a shift in the mix of incoming cases, a change in the source material, a change in surrounding systems, or an update to the underlying model by its provider. See model drift, agent drift.Open full entry → and performance change, caused for example by a shift in the case mix, in the source material, in surrounding systems, or in the underlying model, while every artifact of your own stayed exactly the same.

Alongside both sits adversarial testing: deliberate attempts to derail the agent through hidden instructions in content it processes, to see whether the damage limits hold.

And then the failure mode that none of the tests above catches: the process changes and the agent does not. This is the quiet one, because nothing breaks. The explicit control is a synchronization check: every change to the procedure demonstrably triggers an assessment of the agent artifacts, the system instruction, the knowledge base, the tool limits, the orchestration, and the evaluation set, with a trace in regular change managementchange managementControlled handling of updates to models, data and configurations: every material change re-passes validation before redeployment.Open full entry →. Process change management and agent change management are one chain with two ends.

One distinction belongs here that anyone who has signed an audit opinion will recognize. A test today shows the configuration today. Assurance over a period also requires evidenceevidenceThe concrete proof that a control is designed, implemented, and working: a test report, an audit trail, an impact assessment, a monitoring log. Each link in the governance chain produces an artifact, and together they are what an organization hands to its own board, a regulator, a customer, or an affected person to show, not say, that a system is governed. Its absence is itself the failure: a risk register without test results, or a mitigation claimed without validation, is a governance gap, not a paperwork one. The closing link of the governance chain. See control, governance.Open full entry → that the control stayed deployed, that every change to it passed the change process, and that it was monitored throughout the period under review.

All artifacts live versioned: system instructions, tool definitions, orchestration code, policy rules, evaluation sets, and the procedure versions themselves. That is what lets anyone reconstruct which process knowledge and which barriers the agent ran on a given date.

The two sections above condense into one picture:

	Deterministic layer	Probabilistic layer
Design question	Is there an enforceable barrier on every path to the action?	Why is likely behavior good enough here?
Test	Negative tests plus path and configuration coverage	Evaluation set with a threshold from the risk analysis
Monitoring	Configuration, failures, blocked and bypassed attempts	Performance, drift, override ratesoverride rateThe share of proposals from an AI system that a human reviewer changes or rejects, used as an indicator of whether human oversight is functioning. A structural zero is a reason to investigate rather than a reassurance, because it can mean the system performs well or that the reviewer has stopped examining. It is a signal, not a standalone quality measure. See human oversight, automation bias.Open full entry →
Evidence	Logs, code and configuration, test results	Evaluation set version, score, analysis, acceptance

The governance shell

Controls on the agent are not the whole environment. This article groups the arrangements around it into seven, which together form the governance and general control environment around the agent: they decide whether the design holds up in practice.

1. The legal pre-check, before the design

Does personal data pass through the instruction, the retrieved documents, the memory, or the logs, and what does that mean for the legal basis, the role division with the model providerproviderThe actor who develops an AI system (or has it developed) and places it on the market or into service under its own name. It carries manufacturer-style duties: design controls, documentation, conformity.Open full entry →, and a data protection impact assessmentimpact assessmentA structured evaluation, carried out in the plan-and-design stage, of the harms an AI system could cause and the risk those harms carry, before the system is built. The first place the governance chain is run, and the cheapest point in the life cycle to reduce risk. The anchor artifact of the planning stage; under the EU AI Act, a fundamental-rights impact assessment is required for certain high-risk deployers. See harm, risk, life cycle.Open full entry →. Does the agent contribute to a decision about an individual that is based solely on automated processing and produces legal or similarly significant effects? If so, Article 22 GDPRGDPRRegulation (EU) 2016/679, the General Data Protection Regulation, the EU's law on the processing of personal data. It applies to AI wherever personal data enters training, inputs, outputs, or logs, and it operates alongside the EU AI Act rather than being replaced by it. See controller, processor, lawful basis, DPIA.Open full entry → may bar the arrangement outright, or require one of its exceptions together with appropriate safeguards, which include the ability to obtain human intervention, to express a point of view, and to contest the decision. Where the legal analysis points that way, those measures are design constraints rather than optional governance features. High-risk AI systemshigh-risk AI systemAn AI system that falls under the EU AI Act's strictest requirements, following Article 6. There are two routes in: a system that is a product or safety component covered by the Union harmonization legislation in Annex I and subject to third-party conformity assessment, or a system used in one of the areas listed in Annex III, such as employment, education, or access to essential services. Article 6(3) contains a filter: an Annex III system is not high-risk if it does not pose a significant risk of harm to health, safety, or fundamental rights, although a system that profiles natural persons is always high-risk. See EU AI Act, Annex III, conformity assessment.Open full entry → under the EU AI ActEU AI ActRegulation (EU) 2024/1689, the European Union's law on artificial intelligence. It takes a risk-based approach: prohibited practices, requirements for high-risk AI systems, transparency obligations for specific uses, and a separate regime for general-purpose AI models. Obligations are divided between providers and deployers. See general-purpose AI, conformity assessment.Open full entry → carry human oversighthuman oversightDesigned-in human ability to monitor, intervene in, override or shut down an AI system. It is meaningful only when the human has authority, information and time to act. One of the seven pillars of responsible AI, and under the EU AI Act a requirement for high-risk AI systems: Article 14 requires that those systems are designed so natural persons can effectively oversee them. Oversight that exists on paper but amounts to confirming in practice does not meet that bar. See override rate, automation bias, high-risk AI system, fairness, safety and reliability, privacy, security and robustness, transparency and explainability, accountability, responsible AI.Open full entry → requirements of their own, and the precise measures depend on the system and on whether your organization is the provider or the deployerdeployerAn organization using an AI system under its own authority in its activities. It carries the operator duties: use per instructions, oversight, input relevance, monitoring, notices.Open full entry →; a human approval step may be one of those measures, but it is not the universal form of compliance. The outcome of this check bounds the design.

2. Ownership and risk acceptance

One named human owner in the first line per agent. The mapping is formally adopted: the owner signs for the design, the accepting role signs per residual risk, and the second line assesses completeness and substantiations before go-live.

3. Change management on the agent artifacts

The system instruction, tool definitions, policy rules, and orchestration are production configuration. Separated environments, an assessed change process, and segregation of duties so no single person can move every barrier unchecked. In practice the instruction often lives as a loose text file outside any process, which is a finding waiting to happen.

4. Model change management and vendor risk

The model is a third-party dependency. Pin the version where the provider allows it, and treat every model switch as a change requiring a full retest before production. Where pinning is not possible, the evaluation set and drift monitoring are the detection layer. Alongside that sits the ordinary third-party question: continuity, contractual terms on data processing and retention, and which data reaches the provider through the agent.

5. The detective layer

Logging alone is storage, not control. Four parts make it one.

Logging with intent distinguishes agent actions from human actions and records the why next to the what: the plan, the retrieved documents, the tool calls. Periodic log review by a named role turns that record into oversight, because a detective control nobody reads is not operating. Drift monitoring watches escalation rates, refusals, throughput, and error patterns against thresholds, with alerting. And the override rate, the share of agent proposals a reviewer changes or rejects, works as a KPI: a structural zero is a signal to investigate rather than a reassurance, because it can mean the agent performs well or that the reviewer stopped looking. Read it as a signal rather than a standalone quality measure, alongside review time, the reasons given for deviations, and the share of cases handled without human involvement.

One rule governs all four: a metric becomes a control only when a threshold, an owner, a response time, and a required action are defined. Without those it is an indicator on a dashboard.

Two cautions remain. The logs are themselves a processing operation, and intent logging captures prompts and retrieved documents, so retention, access, and legal basis for the logs belong in the legal pre-check; a detective control that becomes a privacyprivacyThe principle that personal data used by or produced through an AI system stays within the purpose and the legal basis it was collected for. Three routes cause most of the trouble: personal data in training material that was never intended for it, model output that reproduces what the model retained, and purpose creep, where a system built for one use drifts into another the original basis never covered. The GDPR governs this in full, and the EU AI Act adds data governance duties for high-risk systems (Article 10). See DPIA, purpose limitation, responsible AI.Open full entry → finding is a familiar failure. And the evidence itself needs protection: logs, test results, versions, and approvals are only useful if they are complete, hard to alter, reliably timestamped, and traceable to the configuration that ran in production.

6. Continuity, stop mechanism, and incident response

A tested stop mechanism with recorded authority. A fallback so the process can continue manually, including the queue left behind, with the craft to do so maintained. And an agreement on what counts as an agent incident, who assesses, who decides to stop, and how the incident feeds back into the risk analysis and the evaluation set.

7. Retire and revalidation

The life cyclelife cycleThe span of a single AI system from first intake to retirement, across which it must be governed. The horizontal axis of governance: where the governance chain holds one principle, the life cycle runs one system through time. Commonly drawn as six stages, plan and design, data and develop, verify and validate, deploy, operate and monitor, and retire, each with controls native to it and an anchor artifact. A loop rather than a line, because a system in production feeds new risk back into fresh assessment. See artifact, control, governance chain.Open full entry → does not end at operate. A decommissioningdecommissioningDeliberate retirement of an AI system: turn-off plan, stakeholder communication, data and model disposition, record retention.Open full entry → playbook: remove memory and intermediate storage, revoke credentials and the agent identity, disconnect tools and third-party connections, archive logs according to the retention decision, and record the decision to stop or scale with the evaluation results behind it. And a revalidation rhythm for the risk analysis itself, periodic and event-driven, after an incident, a model switch, a process change, or a shift in the threat picture. The risk analysis is a starting point with an expiration date.

Worked example: purchase invoices

The process: receive purchase invoices, match them with the order, approve automatically on a match below 10,000 euros, and route anything above that or anything deviating to a human.

• Legal pre-check. No decisions about individuals. Supplier data passes through the instruction and the logs, including limited personal data for sole proprietorships. The processing agreement with the model provider is checked and a light impact assessment is recorded.
• Risk analysis. The largest harm scenarios are an unjustified payment, through invoice manipulation and accelerated by scale, and a missed payment damaging a supplier relationship.
• Autonomy choice. A workflow agent, because the task has a fixed structure and more autonomy adds nothing. The steps are fixed in code; the model reads, matches, and drafts within them.
• Walkthrough. It surfaced one implicit control that appeared in no procedure: on a changed bank account number, the employee calls the supplier. Made explicit as a mandatory escalation.
• The design. The system instruction carries the role and the escalation rule, not the limit. Retrieved knowledge supplies the current work instruction, the exception supplier list, and the discount policy. The toolset is exactly four functions: read the invoice, match it with the order, approve the payment, present to a human. The limit check sits inside the approval tool, and approval is technically possible only with a valid match reference. There is no tool to change supplier data, so that action is impossible rather than prohibited. Invoice content is treated as untrusted input: its influence on the model cannot be ruled out entirely, but tool validation, bounded permissions, and the fixed orchestration keep it from bypassing the approval limit or the mandatory escalations. The orchestration makes matching mandatory before approval, escalates on a changed bank account number, escalates after three failed matches, and closes every run with a log entry. There is no long-term memory; every invoice is a fresh task.
• The count. Seven controls sit in deterministic layers, six process controls and one agent control, all covered by negative tests that run on every change. One control activitycontrol activityThe concrete action or system rule that achieves a control objective, for example a limit check inside a payment tool that refuses amounts above a threshold. A register that records only objectives fills up with statements nobody can test. See control objective, enforcement point.Open full entry → leans on a probabilistic step. Invoices involving a supplier on the exception list must be routed for human review, and the escalation that follows a returned flag is enforced in the orchestration. The retrieval that surfaces the list entry is not itself the control; it is an input to it, and it is tested by an evaluation set at a threshold derived from the harm estimate, with the residual risk accepted and signed. The seven governance arrangements above complete the control environment as this article defines it.
• The shell in practice. The accounts payable manager owns the agent. The instruction and tools live in version control with separated environments and a change process. The evaluation set holds fifty scenarios including the walkthrough edge cases, drafted outside the build team, retested monthly with follow-up on failures. The team lead reviews logs weekly, a dashboard tracks escalation and refusal rates, and the override rate is discussed monthly. The stop mechanism is assigned to the owner, and the manual fallback is documented and practiced.

The register: one source, three readers

The core control mapping begins in one register. Ten fields per control carry it, and an operational register will add more.

1. Risk: the failure scenario the control follows from.
2. Control family: process control or agent control.
3. Control objectivecontrol objectiveA statement of the outcome a control must achieve, such as "unauthorized payments must not be technically executable". It says what must be true rather than what must be built, which leaves room for more than one control activity to meet it. Keeping the objective separate from the activity is what keeps a control register testable. See control activity, enforcement point.Open full entry →: what must be prevented or assured.
4. Control activity: the concrete action or system rule that achieves it. Keeping this separate from the objective is what stops a register filling up with statements nobody can test.
5. Enforcement point: tool, policy engine, permission, orchestration, or the prompt layer with substantiation.
6. Layer: deterministic or probabilistic.
7. Verification method: binary test or evaluation set with threshold, plus retest rhythm.
8. Control owner: the role accountable for the control operating as described.
9. Residual risk: what remains after the control, in terms of the harm estimate.
10. Acceptor: the role that accepted the residual risk, with date.

Three readers use it. The designer sees which part of the agent carries which control. The second line sees whether the translation is complete and the substantiations are signed. The auditor sees whether the enforcement point exists, whether it operates, and whether the chain from risk to acceptance holds. That chain, from a risk in the business process to an enforcement point in the agent and back to signed evidence, is what makes an agent auditable rather than merely functional.

What this article does not cover

Three subjects sit outside this scope on purpose:

• Multi-agent environments, where agents communicate and failures cascade, which changes the analysis as soon as there is more than one agent.
• The quality of the source documents themselves, because an agent on a poor procedure automates the poor procedure, and document quality is a discipline of its own.
• The human side of the reviewer role, training, alert fatigue, and capacity, which the override rate touches but does not solve.