New benchmark exposes AI’s difficulty with reading construction blueprints

Artificial intelligence has transformed industries ranging from customer service to software development, with systems such as ChatGPT and Claude demonstrating increasingly sophisticated capabilities. Yet a new benchmark developed by construction technology firm Togal.AI suggests that the industry’s most valuable documents, construction blueprints, remain a formidable challenge for even the most advanced AI systems.

The benchmark, known as Construction’s Last Exam (CLE), has been designed to test whether AI models can perform the kinds of drawing analysis tasks that contractors, estimators and quantity surveyors undertake every day. According to Togal.AI, leading AI systems, including ChatGPT and Claude, achieved accuracy scores ranging from 2% to 34% on the company’s benchmark.

While the findings are noteworthy, the benchmark has been created by Togal.AI itself and the reported results have not yet been independently validated through peer-reviewed research or third-party testing. Nevertheless, the exercise highlights an important issue for the construction sector: despite rapid progress in generative AI, there remains a substantial gap between conversational intelligence and the precise visual understanding required to interpret technical building plans.

The benchmark subjects AI models to 33 real-world construction tasks involving blueprint interpretation, quantity take-offs and measurement exercises. Rather than answering questions or generating text, the systems must analyse genuine construction drawings, identify relevant components and calculate measurements needed for project estimation. One example involves identifying and measuring every balcony within a building plan and comparing the results against verified measurements.

Such tasks are central to the construction bidding process. Before work begins, contractors must accurately determine material quantities, floor areas, façade dimensions and other project requirements. Even relatively small errors can have significant commercial implications. According to Togal.AI, the benchmark was specifically designed to evaluate whether today’s frontier AI models can reliably perform these activities. The company’s work reflects a wider trend across artificial intelligence, namely the development of specialised benchmarks to assess AI performance within specific professional domains rather than relying solely on general-purpose tests.

Why blueprint interpretation is so difficult

The challenge stems from the unique nature of construction drawings. Large language models have largely been trained to process vast amounts of text, allowing them to generate natural language responses, write code and summarise information. While many newer systems also include image-processing capabilities, blueprint interpretation requires a combination of computer vision, spatial reasoning, engineering knowledge and mathematical precision.

Construction drawings contain layers of complexity that differ markedly from ordinary photographs or diagrams. A single sheet may incorporate symbols, scales, elevations, annotations, reference points and discipline-specific conventions that professionals learn to interpret over many years. This presents a significant hurdle for general-purpose AI.

A model may successfully describe what appears on a page while still failing to produce the precise measurements necessary for practical use. In construction, precision is critical because inaccurate calculations can influence procurement decisions, scheduling and project costs. Industry analysts have long argued that productivity improvements in construction have lagged behind many other sectors. According to research from consultancy McKinsey & Company, construction remains one of the least digitised major industries despite increasing adoption of digital technologies. As a result, there is significant interest in AI tools capable of automating labour-intensive activities such as quantity take-offs and plan review.I

Togal.AI argues that solving blueprint interpretation requires specialised technology rather than relying solely on general-purpose models. The company says it has spent several years developing machine-vision systems trained specifically to analyse construction plans, using a proprietary dataset comprising tens of millions of drawings. This contrasts with the broad training strategies used by developers of large language models such as OpenAI and Anthropic.

The distinction reflects a broader evolution within artificial intelligence. Increasingly, organisations are discovering that strong performance on generic benchmarks does not automatically translate into expertise within specialised industries. Healthcare, legal services, financial analysis and scientific research have all seen the emergence of domain-specific AI systems designed to complement or outperform general-purpose models in narrowly defined tasks.

In many cases, the future may not involve a single AI system doing everything. Instead, organisations may deploy combinations of foundation models and specialist applications tailored to particular workflows. The construction sector appears to be following the same path.

Security concerns remain a significant issue

Beyond technical performance, blueprint analysis raises important questions around data security. Construction drawings often contain commercially sensitive information, including building specifications, client details, site layouts and proprietary project information. Organisations must therefore consider not only whether an AI model can understand a drawing but also how that information is stored, processed and protected.

Patrick E. Murphy, founder and chief executive of Togal.AI, believes this issue deserves greater scrutiny. “When you hand your blueprints to a piece of software, you’re handing over everything about a project. Most AI tools can’t tell you where that data goes or who sees it, and in this industry, that’s not a small thing,” Murphy said.

He added: “Frontier models still struggle to read a blueprint accurately, let alone protect it.” These concerns mirror wider discussions about AI governance across business and government. As organisations increasingly integrate generative AI into operational processes, questions relating to data sovereignty, compliance and intellectual property protection continue to grow in importance.

Construction’s Last Exam joins a growing list of industry and research benchmarks created to measure AI capabilities more rigorously. Among the most prominent examples is Humanity’s Last Exam, a project designed by hundreds of subject matter experts to test advanced AI systems on difficult academic questions spanning mathematics, science and the humanities.

In software engineering, SWE-bench evaluates whether AI can resolve genuine coding issues taken from real GitHub repositories. Meanwhile, the ARC-AGI benchmark focuses on measuring abstract reasoning abilities that many researchers view as important indicators of progress towards more general intelligence. These initiatives share a common objective: determining whether impressive demonstrations of AI capability translate into meaningful performance in complex real-world tasks.

Construction’s Last Exam extends that philosophy into the built environment.

What it means for the industry

The benchmark’s results should not be interpreted as evidence that AI has failed in construction. Rather, they underline the fact that many practical construction workflows demand a level of precision that remains challenging for today’s most advanced systems. For contractors and developers, the message is clear. AI tools may already offer value in document management, project coordination, data extraction and administrative support. However, when it comes to high-stakes blueprint interpretation and quantity estimation, human oversight remains essential.

More broadly, the benchmark serves as a reminder that artificial intelligence is advancing unevenly. Exceptional performance in language generation does not guarantee corresponding competence in every specialist domain. Whether the figures reported by Togal.AI ultimately withstand independent scrutiny remains to be seen. What is already clear, however, is that blueprint interpretation represents one of the next major frontiers for artificial intelligence. Until that challenge is solved, the construction industry’s most important documents are likely to remain firmly in human hands.