Autonomous AI creates a new enterprise risk: When systems fail, no one knows why

As enterprises accelerate the deployment of autonomous artificial intelligence, a new and potentially serious governance gap is emerging. While organisations are investing heavily in AI agents to automate workflows and decision-making, many are losing visibility over accountability, particularly in complex, multi-agent environments.

New research from Kore.ai highlights the scale of the issue. In a survey of more than 400 IT and business leaders from large enterprises, 70 percent of respondents said they could detect when something had gone wrong but could not identify which AI agent was responsible. This finding underscores a key challenge in the evolution of enterprise AI. This where detection is improving, but attribution is not.

The rise of multi-agent complexity

The move towards multi-agent AI systems represents a natural progression of enterprise automation. Rather than relying on single models or bots, organisations are deploying networks of specialised agents, each responsible for discrete tasks such as customer interaction, data processing, or decision orchestration.

However, as these systems scale, their interactions become increasingly opaque. When one agent triggers an action that cascades across others, tracing the origin of an issue becomes difficult.

This is reflected in the survey findings. While most organisations have detection capabilities, the inability to isolate the source of a failure suggests that enterprises are operating AI ecosystems they cannot fully explain.

The research also highlights an important trust deficit. More than half (53 percent) of organisations admit they are running AI agents they do not fully understand. This creates a paradox. Enterprises are relying on AI to deliver efficiency and innovation, yet at the same time, they lack confidence in how these systems behave under real-world conditions.

In highly regulated sectors this raises immediate concerns. If organisations cannot explain how decisions are made, or why failures occur, regulatory compliance and risk management become significantly more complex.

Manual intervention remains the safety net

Despite the promise of autonomy, human oversight remains critical. The survey shows that 79 percent of enterprises have had to reverse autonomous AI actions manually, indicating that full automation remains some distance away. More significantly, these reversals are not trivial. Some 93 percent of respondents reported that corrective actions were costly and disruptive.

This suggests that organisations are effectively absorbing a form of operational drag, where the benefits of automation are offset by the cost of intervention when things go wrong.

From a systems perspective, this is not unusual. Complex systems tend to fail in unpredictable ways. What is changing, however, is the speed and scale at which those failures can propagate when driven by autonomous AI.

Detection delays and customer-led discovery

The findings also reveal variability in how quickly organisations can identify issues. Around half of respondents said failures are detected within one to four hours, while a third require four to eight hours. Detection methods are similarly uneven. For example, 39 percent say they rely on real-time dashboards, whereas 29 percent use automated alerts, and 17 percent depend on log analysis.

Perhaps most concerning, 15 percent of enterprises rely on end users to report problems, effectively outsourcing detection to customers. This introduces reputational risk. When customers identify failures before the organisation does, trust can erode rapidly, particularly in customer-facing services such as banking, retail, or digital platforms.

Raj Koneru, CEO and founder of Kore.ai, describes this phenomenon as a form of “governance debt.” The concept reflects how delays in identifying and resolving issues allow risks and costs to accumulate over time. In traditional IT systems, governance structures evolved alongside the technology. For AI, particularly autonomous AI, deployment is often outpacing governance frameworks.

Visibility is not accountability

A key insight from the research is that observability alone is insufficient. Many organisations have invested in monitoring tools that provide visibility into system activity. However, these tools do not necessarily answer the critical question: who made the decision?

As Kore.ai’s Chief Marketing Officer, Peter Mullen, notes, there is a distinction between visibility and accountability. An AI agent that can be observed but not governed remains a risk. This distinction is particularly relevant as enterprises move towards AI-driven decision-making and introduce automated workflows with minimal human oversight. Without clear accountability mechanisms, these systems can become black boxes, visible, but not controllable.

The findings point to a broader shift in how organisations must approach AI. Governance can no longer be treated as an afterthought. Instead, it must be embedded into the design and deployment of systems from the outset.

As organisations scale their use of AI, they must also confront the challenges of control, trust, and accountability. Perhaps deploying AI agents is no longer the hard part; it may be that governing them is.