Checking, checking: Is human oversight of AI becoming a false comfort?

Artificial intelligence is rapidly becoming embedded within business decision-making. From forecasting sales and identifying business opportunities to recommending actions and automating workflows, AI is increasingly influencing how organisations operate. Yet as AI adoption accelerates, an important question is emerging: how effective is human oversight when the people supervising AI systems may themselves be relying on incomplete or subjective information?

New research from Salesloft, based on a survey of 500 U.S. sales and revenue leaders, suggests this question deserves greater attention. The findings reveal a paradox at the heart of many enterprise AI strategies. While organisations often cite “human in the loop” governance as a safeguard against poor AI decisions, the underlying quality of human judgement may not always be as robust as assumed.

AI is everywhere, but maturity remains limited

The study found that AI usage has become universal among surveyed revenue organisations. Every respondent reported using AI somewhere within the revenue process. However, widespread adoption should not be confused with maturity. Only 20.6 percent of respondents described their AI strategy as production-ready and delivering measurable business outcomes. Meanwhile, 28.2 percent remain in the experimentation phase, still exploring how AI can be incorporated into operational workflows.

This distinction is important. Many organisations have successfully deployed AI tools, yet relatively few appear to have fully integrated these technologies into repeatable and measurable business processes.

As Steve Cox, CEO of Salesloft, noted: “Revenue teams don’t have an AI access problem anymore. The bigger question is what they’re getting from it.” Increasingly, the challenge is not whether organisations possess AI capabilities, but whether those capabilities are producing reliable and meaningful results.

One of the most interesting findings concerns the growing popularity of so-called “guided autonomy” models. According to the survey, 38.4 percent of organisations favour an approach whereby AI systems can make recommendations or take actions while humans retain oversight and approval authority.

Many AI governance frameworks recommend maintaining human review of automated decisions, particularly when those decisions carry operational, financial or regulatory consequences. Human involvement is intended to reduce risk by providing a final layer of scrutiny before action is taken. Yet the survey data also raises a potentially uncomfortable question. If human reviewers are working with flawed information, can they realistically provide effective oversight?

The data quality challenge

The report highlights a significant issue affecting many organisations: the reliability of underlying business data. More than half of respondents (55.6 percent) reported that information entered into customer relationship management (CRM) systems is based primarily on subjective seller reporting rather than independent verification. This creates a challenge extending well beyond sales performance.

Artificial intelligence systems rely on data for training, analysis and decision support. Human managers likewise depend upon the same information when reviewing AI outputs. If both AI and human supervisors are drawing conclusions from incomplete or subjective inputs, then the supposed safeguard of human oversight may be weaker than many organisations realise. In such circumstances, a human reviewer may simply confirm an AI recommendation rather than independently challenge it. The danger is not that AI becomes too autonomous. Rather, it is that both human and machine arrive at the same conclusion because both are relying on the same imperfect source material.

The survey also revealed limitations in organisational visibility. Although 89 percent of respondents believe managers assess performance objectively and more than half report receiving coaching at least every two weeks, only around one-third can immediately determine why a deal has stalled. A further 41 percent either take time to identify the cause or lack sufficient visibility altogether. An additional 27 percent can see whether opportunities are won or lost but cannot explain what occurred between individual stages of the sales process.

For AI systems, this type of incomplete visibility creates obvious problems. Machine learning models are most effective when they can identify patterns from comprehensive and reliable data. For human decision-makers, limited visibility can create an equally serious challenge. Without understanding the factors that influence outcomes, oversight risks becoming reactive rather than analytical. The combination of AI recommendations and incomplete business visibility may therefore create an illusion of control rather than genuine understanding.

Technology is not the primary constraint

Perhaps the most significant finding is that technology itself is no longer the principal obstacle. Modern organisations have unprecedented access to software, analytics platforms and AI tools. Revenue teams generate more information today than at any point in history. Yet data abundance does not automatically lead to better decisions. Many organisations continue to struggle with connecting information, identifying emerging risks and translating insights into effective action.

In some respects, the challenge resembles issues observed in other sectors undergoing AI transformation. Healthcare providers may possess extensive patient data but still face diagnostic complexity. Financial institutions may process billions of transactions while struggling to identify emerging fraud patterns. Manufacturers may collect vast amounts of operational information without fully understanding the causes of process variation. Across industries, the problem increasingly centres on interpretation rather than acquisition.

The findings may prompt organisations to reconsider how they define AI governance. Much discussion surrounding responsible AI focuses on maintaining human oversight, ensuring transparency and limiting autonomous decision-making. However, the effectiveness of human oversight depends not only on the presence of a reviewer but also on the quality of the information available to that reviewer. A poorly informed human does not automatically provide a stronger control than a well-designed AI system.

Consequently, organisations should place greater emphasis on data quality, data validation and independent verification mechanisms. Rather than asking whether humans remain involved, leaders may need to ask whether humans are sufficiently informed to make meaningful interventions. In many cases, better governance may require improving the quality of business data before increasing the sophistication of AI systems.