The questions every buyer needs to ask an AI vendor

Artificial intelligence is no longer confined to chatbots or predictive analytics, organisations are now being offered autonomous AI agents capable of completing tasks, making decisions and interacting with software systems on behalf of humans. The promise is: lower costs, faster workflows and enhanced productivity. Yet amid the excitement, a practical question remains largely unanswered: how can buyers distinguish genuine AI capability from a carefully scripted demonstration?

The AI market is rapidly becoming crowded. Every technology conference features vendors claiming their products can transform business operations. Marketing material highlights impressive use cases, and demonstrations often appear seamless. However, a polished presentation can conceal important limitations. As organisations invest increasing amounts in AI-powered tools, buyers need to move beyond the sales pitch and ask harder questions.

The demo problem

Technology demonstrations have always been designed to show products in their best light. Artificial intelligence is no different. The difference is that AI can appear surprisingly capable during a controlled demonstration while struggling with real-world complexity. Many AI systems are shown performing narrowly defined tasks under carefully prepared conditions. The workflow is selected beforehand, the data cleaned in advance and the outcome largely predictable.

This does not necessarily mean vendors are being deceptive. Instead, it reflects the reality that AI performance often varies significantly depending on the environment in which it is deployed. The challenge for buyers is determining whether a successful demo represents genuine operational capability or merely a favourable scenario.

According to Gartner’s analysis of generative AI, many organisations remain in the early stages of understanding how generative AI and agentic systems behave once introduced into complex business environments. Likewise, the annual Stanford AI Index Report continues to document gaps between laboratory performance and real-world deployment. Therefore, rigorous questioning has become an essential part of procurement.

Can the agent do everything I can do?

Perhaps the most important question a purchaser can ask is deceptively simple: “Can the AI agent do everything I can do in the software, or only what I have just been shown?“

Many demonstrations focus on a small number of highly polished activities. A sales representative may show the system creating a report, updating a customer record or scheduling a meeting. The interface appears intuitive and the task is completed successfully. However, real work rarely follows a perfect sequence. Business processes often involve exceptions, unexpected decisions, incomplete information and unusual combinations of tasks. Users frequently move between screens, switch priorities and encounter unforeseen circumstances. If an AI agent only performs well within a narrow subset of functions, the organisation may discover significant limitations after deployment.

Buyers should therefore request a capability map showing exactly what the software can and cannot do. They should also ask vendors to define any restrictions relating to permissions, workflows, integrations and escalation procedures.

A useful question is: “What percentage of the application’s functionality can your agent access and perform autonomously?”

The answer may reveal a significant gap between marketing claims and operational reality.

It needs to be considered that ‘real’ organisations rarely possess perfect data. Customer records contain errors. Product catalogues include inconsistencies. Historical databases feature missing fields. Different departments may use conflicting naming conventions. This “messiness” is one of the biggest challenges facing enterprise AI adoption.

Research published in the Sloane Review has repeatedly highlighted the importance of data quality in determining AI success. Similarly, the AI Governance Alliance emphasises that effective AI deployment depends upon trustworthy and well-managed data ecosystems. Yet demonstrations frequently rely on pristine datasets.

A sensible buyer should therefore ask: “Can the system operate successfully when the data are incomplete, duplicated, contradictory or poorly formatted?”

The answer is critical because few production environments resemble the carefully curated examples shown during demonstrations. Vendors should be able to discuss how their systems manage uncertainty, missing information and conflicting inputs. More importantly, they should be willing to demonstrate this capability. An AI platform that performs brilliantly with ideal data but struggles with everyday operational information is unlikely to deliver the promised return on investment.

One of the key tests is quite straightforward: Ask the vendor to abandon the prepared demonstration and carry out an unscripted task. This may involve a workflow, process or scenario selected by the prospective customer during the meeting itself. The objective is not to embarrass anyone. Rather, it allows buyers to observe how the technology behaves outside a rehearsed environment.

Recent research from the Stanford Institute for Human-Centered Artificial Intelligence has shown that AI systems often exhibit markedly different performance when transitioning from controlled testing environments into real operational settings. Similarly, analysis from https://www.idc.com/promo/future-of-ai suggests that many organisations encounter unexpected challenges when AI systems are exposed to diverse operational demands.

AI systems vary considerably in their ability to adapt to unfamiliar circumstances. Some can generalise effectively while others depend heavily on predefined workflows, finely tuned prompts and extensive configuration. An unscripted demonstration allows buyers to determine whether the system genuinely understands changing objectives, copes effectively with ambiguity, requests clarification when necessary, recovers gracefully from errors and completes tasks without requiring hidden support from engineers in the background. These capabilities often provide a much more realistic indication of future performance than any carefully prepared presentation.

If preparation takes months, what does that tell you?

Perhaps the most revealing question concerns preparation. Many advanced demonstrations require substantial effort behind the scenes. Data may be cleaned, integrations configured, workflows optimised and prompt libraries refined. In some cases, technical teams spend weeks or even months preparing a demonstration environment.

That raises another critical question: “If it takes months to prepare a successful demonstration, what does that imply about deployment in a live environment?”

A robust AI solution should not require heroic levels of preparation before delivering value. The experience of many organisations documented in McKinsey’s annual report, suggests that implementation complexity remains one of the most significant barriers to obtaining measurable value from AI investments. The research consistently shows that while adoption rates are increasing, relatively few organisations are fully scaling AI across business operations.

Similar findings emerge from Deloitte’s research. In its report, Deloitte notes that organisations continue to face challenges related to governance, workforce readiness, integration complexity and data management. The business case for careful vendor evaluation is reinforced by PwC’s study, which estimated that AI could contribute trillions of dollars to the global economy. However, achieving that value depends on successful implementation rather than simply purchasing AI technology.

Rather than focusing solely on the demonstration itself, buyers should ask how long the demonstration environment took to build, how many engineers were involved, how much manual tuning was required, whether custom integrations had to be developed and what ongoing maintenance the system will require after deployment. Transparency in these areas often provides a much clearer understanding of total ownership costs than the demonstration itself.

Artificial intelligence undoubtedly offers transformative opportunities. Agentic AI systems may eventually automate complex knowledge work, improve decision-making and streamline organisational operations. However, enthusiasm should never replace due diligence. Successful procurement depends upon understanding capability, reliability and limitations. Buyers should focus less on impressive presentations and more on evidence of performance under realistic conditions. This approach is consistent with the recommendations of the NIST AI Risk Management Framework and emerging international standards such as ISO 81230, both of which emphasise transparency, accountability and trustworthy AI governance. The strongest vendors will welcome difficult questions because they are confident in the robustness of their technology.