Safety & reliability

What it means

Safety and reliability is the property that an AI system performs as intended within its operating conditions, fails in predictable and contained ways outside them, and does not cause harm through its normal operation. The EU AI Act addresses this through the Art. 15 requirements on accuracy, robustness, and the obligation that high-risk systems achieve and maintain an appropriate level of performance across their lifecycle. Reliability is the temporal dimension of safety: a system that is accurate on the day it ships but degrades silently over the following year is not reliable, regardless of its launch metrics.

The distinction between the two halves matters operationally. Safety is about the consequences of the system's outputs being wrong. Reliability is about how often, and under what conditions, they are wrong. A system can be highly reliable and still unsafe if the rare failures are catastrophic, and it can be tolerably safe while unreliable if the failures are frequent but individually minor. The control set has to address both the failure rate and the failure consequence.

Why it matters

For systems embedded in regulated products, safety failures carry product-liability and sectoral-regulatory consequences in addition to the EU AI Act. For standalone high-risk systems, the most common real-world failure is silent performance degradation, where a model that was validated at deployment slowly drifts as the world changes around it, and no one notices until the accumulated errors surface as a complaint, an audit finding, or an incident.

Governing safety and reliability

The governing idea is that performance is not a launch property but a lifecycle property. The controls below treat the deployment metric as a baseline to be defended, not an achievement to be filed.

Control layer	Control
Preventive	Define acceptable performance thresholds and failure tolerances before deployment, derived from the use case and documented in the Art. 11 technical documentation. Conduct robustness testing across edge cases and out-of-distribution inputs, not only the expected operating range. Establish the operating conditions under which the system is validated, and design the deployment so the system is not used outside them.
Detective	Monitor live performance against the deployment baseline at a defined frequency, with alerting when accuracy or error rates cross the documented threshold (Art. 26.5 monitoring, feeding Art. 72 post-market monitoring). Detect concept drift and data drift through scheduled comparison of input distributions and outcome distributions against the validated baseline. Capture and review near-misses, not only realised failures.
Corrective	Maintain a defined response for threshold breaches: rollback to a prior model version, fallback to a manual process, or controlled suspension. Operate a kill switch that an oversight function can invoke (linking to Art. 14). Report serious incidents under Art. 73 and feed every incident back into the risk management system under Art. 9 so the next iteration is safer.