Agentic AI risk assessment: from architecture decisions to control objectives

Updated: July 2026

The step between architecture and controls

The architecture of an AI agentAI agentA system that perceives its environment, decides and takes actions toward a goal (calling tools, executing plans). Autonomy of action demands allowlists, approval gates, sandboxing, logging and a kill switch.Open full entry → determines where it stores knowledge, makes decisions, and exercises authority. Seven decisions carry that architecture: the six building blocks through which process knowledge reaches the agent, which are the model, the system instruction, retrieved knowledge, the tool definitions, the orchestrationorchestrationThe fixed program code around an AI model that determines which steps run in which order, what is mandatory, and where a human is involved. Because it is conventional software, it behaves predictably and can carry enforceable controls, which is why process structure belongs there rather than in prompt text. See enforcement point, guardrail.Open full entry →, and memory, plus the autonomy the agent is granted. Those decisions are not risksriskIn the EU AI Act's terms, the combination of the likelihood that a harm occurs and the severity of it if it does. The link between a principle (via the harm that would breach it) and a control (the measure that reduces it). Naming the harm and assessing its risk is required by Art. 9 before any mitigation measure is chosen. See harm, control, residual risk.Open full entry → by themselves. A tool that can approve payments is a capability, not a failure. They become risks when they create the possibility of harmharmThe concrete damage an AI system can do that a responsible-AI principle exists to prevent: in the EU AI Act's terms, harm to a person's health, safety, or fundamental rights. Harm is the bridge between an abstract principle and a governable risk; governance becomes operational the moment an organization names the specific harms it wants to prevent. For fairness, a harm is a group receiving systematically worse outcomes because of a characteristic that should not have counted. See principle, risk.Open full entry →, and that is a separate judgment.

This article makes that judgment explicit. It takes the recorded architecture decisions of an agent and turns them into risk scenarios and control objectivescontrol objectiveA statement of the outcome a control must achieve, such as "unauthorized payments must not be technically executable". It says what must be true rather than what must be built, which leaves room for more than one control activity to meet it. Keeping the objective separate from the activity is what keeps a control register testable. See control activity, enforcement point.Open full entry →, which are what the controlcontrolThe concrete, testable measure that reduces a specific risk, and through that risk protects the principle behind it. Also called a risk management measure, risk response, or risk treatment. Always traceable to the risk it addresses: under EU AI Act Art. 9 every control must map back to a specific risk, and controls recorded separately from their risks is a recognized compliance failure. It works in one of three types: preventive, detective, or corrective. See risk, control types, evidence.Open full entry → environment needs as input.

The method does not change; the inputs do

AI risk management already has a shape: recognize the risk, assess how likely and how severe it is for your specific system, and control it with measures you can test. That pattern, and the chain behind it from principleprincipleOne of the seven responsible-AI values a governed system should live up to (fairness, safety and reliability, privacy, security and robustness, transparency and explainability, accountability, human oversight). A principle is abstract: it states an outcome, not a lever you can pull. It becomes governable by naming the harm that would breach it, assessing the risk that harm carries, and placing controls against that risk. Held this way, a principle becomes a pillar. See pillar, harm, risk.Open full entry → through harm and risk to control and evidenceevidenceThe concrete proof that a control is designed, implemented, and working: a test report, an audit trail, an impact assessment, a monitoring log. Each link in the governance chain produces an artifact, and together they are what an organization hands to its own board, a regulator, a customer, or an affected person to show, not say, that a system is governed. Its absence is itself the failure: a risk register without test results, or a mitigation claimed without validation, is a governance gap, not a paperwork one. The closing link of the governance chain. See control, governance.Open full entry →, is set out in the AI risk management article and does not need restating here. Likelihood, severity, risk appetiterisk appetiteThe level of risk an organization's leadership is willing to accept in pursuit of its objectives, set at the governance design level. It is the benchmark against which residual risk is judged acceptable or not, inherited from the organization's broader governance and applied to AI. A concept from enterprise risk management (COSO ERM) before it is an AI one. See residual risk, governance design.Open full entry →, and the obligations under Article 9 of the EU AI ActEU AI ActRegulation (EU) 2024/1689, the European Union's law on artificial intelligence. It takes a risk-based approach: prohibited practices, requirements for high-risk AI systems, transparency obligations for specific uses, and a separate regime for general-purpose AI models. Obligations are divided between providers and deployers. See general-purpose AI, conformity assessment.Open full entry → live there too.

Agentic AIagentic AISystems where a model takes actions (calling tools, executing multi-step plans), amplifying both capability and every failure mode; governed with action allowlists, approvals and full logging.Open full entry → changes the input to that method in three ways.

• The process is performed differently. Part of the work now runs through a probabilistic systemprobabilistic systemA system whose outputs are statistical predictions with error rates: designed to be wrong some fraction of the time, which testing measures rather than eliminates.Open full entry →, so "the procedure says so" no longer predicts what happens.
• The agent holds authority. It can act, not only advise, and the tools it holds define what it can reach.
• The failure modes are new. Some of them appear in no procedure, because they only exist once an agent runs the process.

The consequence is practical. A risk assessment for an agentized process cannot be the old assessment with a paragraph added. It starts from the architecture decisions, because those decisions determine how this process can now fail.

Start with the process, not with the model

Before touching the architecture, bound the assessment with four questions.

• Which process does the agent run, and what outcome must it achieve? An agent that drafts a reply carries different stakes than one that pays an invoice.
• Who and what can be affected? People and their rights, money, data, continuity of service, the organization's standing.
• Which decisions or actions are material? Not every step deserves equal attention; the ones that move value, affect a person, or are hard to reverse do.
• What is the volume? The same error at ten cases a week and at ten thousand a week is not the same risk.

These answers set the weight of everything that follows. They also decide how much assessment is proportionate: a low-impact internal assistant does not need the treatment a payment agent needs, and spending equally on both is its own failure.

The architecture decisions as input

Each architecture decision is a place where this process can now fail.

Architecture decision	The risk question it raises
Model and version	Which assumptions does the design make about the model's behavior, what can change outside our control, and does the model hold organization-specific knowledge through fine-tuningfine-tuningFurther training of an existing model on your own data to adapt its behavior. This makes you responsible for the modification, potentially up to provider level.Open full entry →?
System instruction	Which process rules exist only as behavioral instructions?
Retrieved knowledge	Which sources can influence decisions, and how current are they?
Tool definitions	Which actions can the agent technically perform, and what does each tool refuse?
Orchestration	Which process steps are enforced in code, and where is a human involved?
Memory	Which earlier information can shape later decisions?
Autonomy	Which steps and sequences does the model decide for itself?

The table is a starting point, not the assessment. The sharpest risks rarely come from one decision alone.

The amplifiers that make agentic risk different

The same failure can be minor in one setting and severe in another. Eight factors decide how serious a scenario becomes. They are not new risk categories: they change the weight of the risks you have already identified, and an agent changes several of them at once.

• Autonomy. The more the model decides, the more execution paths exist, and the harder it becomes to test them all.
• Scale and speed. One error is not one error. It is the same error repeated until something stops it.
• Reversibility. A draft can be discarded, a payment cannot. Irreversible actions deserve heavier control regardless of how unlikely the failure seems.
• Detectability. How long would this run wrong before anyone noticed? Slow detection turns a small error into a large one.
• Privileges. What the agent can reach sets the ceiling on what any failure, including a successful manipulation, can achieve.
• External exposure. Does content from outside the organization enter the agent's context? That is where manipulation arrives.
• Impact on individuals. Decisions that affect people carry legal weight that operational errors do not.
• Correlation. Human errors are spread out and often balance each other. An agent's errors all point in the same direction, so they add up.

Use these to weigh scenarios, not as a second scoring scheme. The general assessment of likelihood and severity stays as it is. These factors explain why a known risk becomes more serious when an agent performs the work.

Turning decisions into scenarios

A usable scenario has three parts: a cause, an event, and an impact. Without the cause it cannot be controlled, and without the impact it cannot be weighed.

The scenarios that matter most come from combinations of architecture decisions. Six examples, with the amplifiers that drive them.

• Retrieved knowledge and orchestration. An outdated exception list is retrieved, so the agent treats a restricted supplier as ordinary, an invoice is approved without the review it required, and money is paid to a supplier who should have been checked. Amplifiers: detectability, scale.
• Autonomy and toolset. A planning agent finds a second route to the same action, a route the limit check does not cover, so an amount above the threshold is approved without the extra approval it required. Amplifiers: autonomy, reversibility.
• External content and privileges. A hidden instruction inside an incoming document steers the agent to call a tool with values the sender chose, and data leaves the organization through a legitimate channel. Amplifiers: external exposure, privileges.
• Memory and time. An outdated exception is retained in memory and quietly shapes every later approval, producing a systematic error that nobody traces back to its origin. Amplifiers: detectability, correlation.
• Model change and unchanged artifactsartifactThe concrete record that proves a control was carried out: a test report, an impact assessment, a monitoring log, a release sign-off. An artifact is the tangible form evidence takes, the thing an auditor reaches for to confirm that a control was not just designed but actually operated. Each stage of the AI life cycle produces its own anchor artifact. Distinct from evidence as a whole: evidence is the proof, an artifact is one piece of it. See evidence, life cycle.Open full entry →. The providerproviderThe actor who develops an AI system (or has it developed) and places it on the market or into service under its own name. It carries manufacturer-style duties: design controls, documentation, conformity.Open full entry → updates the model, its behavior shifts, escalations fall away, and cases that should have gone to a human are handled automatically while every artifact of your own stayed the same. Amplifiers: detectability, scale.

Not every important scenario starts in the technology. Automation also changes how people behave.

Human approval and automation biasautomation biasThe human tendency to over-trust automated outputs: accepting a system's recommendation without genuinely weighing the case, which hollows out human oversight.Open full entry →. The reviewer approves without examining the case, because the agent is right most of the time and the queue is long, so the approval step exists in the design and not in practice. Amplifiers: scale, impact on individuals.

Two sources keep this list complete. The architecture decisions, which tell you what is technically possible, and a walkthrough with the people who run the process today, which brings out the implicit controls that appear in no procedure. The instinct "I call the supplier when the bank details change" is a control, and it disappears silently when the process is automated by someone who never knew it existed.

From scenario to control objective

A control objective states the outcome that must hold. It is not the measure itself, and it deliberately says nothing about the mechanism, because choosing the mechanism comes later and choosing it too early narrows the options.

Three examples, taken from the scenarios above:

• Unauthorized payments must not be technically executable, regardless of the route the agent takes to reach them.
• Decisions based on outdated process knowledge must be detected before execution, or routed to a human.
• A successful manipulation of processed content must not be able to move value or data outside the agreed boundaries.

Each of these is testable in principle, each says what must be true rather than what must be built, and each can be met in several ways. That is what makes them useful: they leave the design of the control to the next step instead of prescribing it.

Choosing the treatment

Not every risk needs a control. The four options are the familiar ones, and two of them deserve a note in this context.

• Avoid. Do not give the agent the capability at all. In agent design this is the strongest and most underused option: a tool that does not exist cannot be misused.
• Reduce. Lower the autonomy, narrow the toolset, or add controls. Least agencyleast agencyThe principle, used in OWASP's agentic security work, that an agent should be granted the smallest set of capabilities and the least autonomy the task requires. Applied at architecture level it is a risk decision rather than a technical preference: less autonomy leaves more of the process enforceable in code. See OWASP, agentic AI, enforcement point.Open full entry → belongs here: reducing what the agent may decide is risk treatment, not merely an architecture preference, and it should be recorded as such.
• Transfer. Contractual arrangements with a provider shift some exposure, but they do not move accountabilityaccountabilityThe principle that a named human or organization answers for an AI system's outcomes, through ownership, documentation, audit trails and redress; never the system itself. The EU AI Act attaches obligations to the role rather than the technology, with provider duties in Article 16 and deployer duties in Article 26, supported by technical documentation (Article 11) and record-keeping (Article 12). See provider, deployer, record-keeping, responsible AI.Open full entry → for the process.
• Accept. A decision, not a default. It needs a named acceptor with the mandate to accept, a substantiation, and a review date.

What this step delivers

The result is a short, weighted set of scenarios, each with a control objective and a treatment decision. That is the input for the next question, which is a different one: where each objective is enforced, how you test that it holds, and what evidence shows it held over time. That question is the subject of the control environment for agentic AI.