Pillar 02 of seven
Safety & reliability
Safety and reliability mean the system performs as intended across the conditions it will meet, and fails in a safe, predictable way when it reaches its limits.
What it means
Safety and reliability is the property that an AI system performs as intended within its operating conditions, fails in predictable and contained ways outside them, and does not cause harm through its normal operation. The EU AI Act addresses this through the Art. 15 requirements on accuracy, robustness, and the obligation that high-risk systems achieve and maintain an appropriate level of performance across their lifecycle. Reliability is the temporal dimension of safety: a system that is accurate on the day it ships but degrades silently over the following year is not reliable, regardless of its launch metrics.
The distinction between the two halves matters operationally. Safety is about the consequences of the system's outputs being wrong. Reliability is about how often, and under what conditions, they are wrong. A system can be highly reliable and still unsafe if the rare failures are catastrophic, and it can be tolerably safe while unreliable if the failures are frequent but individually minor. The control set has to address both the failure rate and the failure consequence.
Why it matters
For systems embedded in regulated products, safety failures carry product-liability and sectoral-regulatory consequences in addition to the EU AI Act. For standalone high-risk systems, the most common real-world failure is silent performance degradation, where a model that was validated at deployment slowly drifts as the world changes around it, and no one notices until the accumulated errors surface as a complaint, an audit finding, or an incident.
Governing safety and reliability
The governing idea is that performance is not a launch property but a lifecycle property. The controls below treat the deployment metric as a baseline to be defended, not an achievement to be filed.
| Control layer | Control |
|---|---|
| Preventive | Define acceptable performance thresholds and failure tolerances before deployment, derived from the use case and the provider's Art. 11 technical documentation, and record them in your own control documentation. Conduct robustness testing across edge cases and out-of-distribution inputs, not only the expected operating range. Establish the operating conditions under which the system is validated, and design the deployment so the system is not used outside them. |
| Detective | Monitor live performance against the deployment baseline at a defined frequency, with alerting when accuracy or error rates cross the documented threshold (Art. 26.5 monitoring, feeding Art. 72 post-market monitoring). Detect concept drift and data drift through scheduled comparison of input distributions and outcome distributions against the validated baseline. Capture and review near-misses, not only realized failures. |
| Corrective | Maintain a defined response for threshold breaches: rollback to a prior model version, fallback to a manual process, or controlled suspension. Operate a kill switch that an oversight function can invoke (linking to Art. 14). Report a serious incident to the provider under Art. 26(5), the deployer's route into the Art. 73 regime, and feed every incident back into the risk-management loop so the next iteration is safer. |
- PrincipleDependable, bounded operation
- HarmMalfunction that harms health, safety, or property
- RiskLikelihood of failure, combined with its severity
- ControlPerformance thresholds, testing, and safe fallback
- EvidenceTest logs, monitoring data, and incident records
* Article references are verified against the consolidated text of Regulation (EU) 2024/1689 and, where cited, Regulation (EU) 2016/679. The pillar itself is a general responsible-AI principle; these articles are where it anchors in EU law.
Reliability is proven by testing against the conditions the system will meet, and safety by having a defined, safe response when it reaches its limits.
Define performance thresholds and the conditions under which the system must fall back or hand to a human.
Test against edge conditions before release and monitor for drift and degradation in operation, with a rehearsed fallback.
An agent chains actions, so one wrong step can cascade into the next before a person notices. Reliability is no longer per-answer, it is the reliability of a whole sequence with real-world effects.
Define blast-radius limits per agent: what it may touch, up to what scope, and when it must hand back to a human.
Run agents behind circuit-breakers that halt the chain on anomalous behavior, and rehearse the stop path under load.