When Artificial Intelligence Leaves the Test Environment: What Happens in the Real World

IA FORUM MEMBER INSIGHTS: ARTICLE

By Lisa Matthews, Former Senior Director, Cybersecurity Compliance, ALLY

Over the past couple of years, I’ve had dozens of conversations with regulators, executives, risk leaders, engineers, and policy teams across the financial services industry around one common question:

What actually happens to AI safety once systems leave controlled environments and enter real-world production?

Most AI safety conversations around generative, agentic and world-like models still focus on model design: alignment and behavioral techniques, evaluation frameworks, and training safeguards. Only recently has Information Security entered the picture regarding detection of malicious activity within the models.  All of this is crucial, but it addresses only part of the challenge.

As AI systems move rapidly into production environments, a new risk dimension is emerging - one driven not by model design alone, but by how AI interacts with complex institutional, technical and cultural systems over time.

Some of the highest-impact AI risks emerge after deployment. Future AI failures will not originate solely from training data, model architecture, or evaluation gaps. They will emerge from the interaction between AI systems and the environments in which they are deployed. This includes organizational incentives, human over-reliance and trust creep (a colleague calls this the deification of AI), multi-vendor ecosystems and increasingly, autonomous agentic workflows.

To address this shift, organizations will need to move beyond traditional model-centric risk assessments toward a new class of real-world AI system assessments, which may include:

• Institutional drift assessment

• Human over-reliance assessment

• Failure containment readiness assessment

• AI ecosystem and third-party interaction assessment

In parallel, regulators and Boards are beginning to shift expectations toward lifecycle accountability, operational resiliency, and demonstrable real-world control over AI-enabled systems. While this series focuses primarily on safety during real-world AI deployment, operational resiliency - including the ability to maintain service integrity, recover from AI-driven failures, and prevent cascading system impacts - is an equally critical and rapidly emerging focus area that will be explored separately, particularly as safety and resiliency expectations increasingly converge in AI-enabled systems.

Over the next 3 - 5 years, AI incidents are likely to become more distributed and ecosystem-driven, particularly as agentic AI and world-model-like systems become more widely deployed across multi-company and multi-platform environments.

The defining question of the following years will not be whether organizations can build powerful AI systems. It will be whether they can safety operate them - at scale, over time, and under real world pressures. 

This shift is driving the emergence of a new cross-functional capability that sits at the intersection of research, engineering, risk, security and operations.

Author Disclaimer: The views and opinions expressed herein are those of the Author alone and are shared in a personal capacity, in accordance with the Chatham House Rule. They do not reflect the official views or positions of the Author’s employer, organization, or any affiliated entity.