What Canadian journalists are learning about AI from each other

Many journalists spend their careers trying to beat each other to a story. On Wednesday evening in Toronto, a room full of them handed over how they use AI in their newsrooms: the tools they’ve built, how they check the AI’s work, and what they’d do differently next time.

“It’s been a few months, which is roughly a geological era in AI time,” said Rignam Wangkhang, co-founder, Canadian AI Journalism Network, of the time since the group last met in the spring.

About 50 people from Canadian newsrooms, large and small, came to the network’s quarterly forum, hosted at the Toronto Star‘s offices at The Well. They came for the same reason leaders in every other industry are comparing notes right now: The tools change monthly, every organization is learning as it goes, and the fastest way to find out what works is to hear it from someone who has tried it.

Digital Journal is one of those newsrooms. We cover technology for the leaders responsible for it inside Canadian organizations, so we use the tools ourselves. AI helps us find sources, verify facts and research stories, along with a good share of what the presenters showed that night, and we build and test prototypes of our own. Our journalists write, edit and check every story. Like everyone else in the room, we’re a small team using AI to take on bigger work, and I came with questions of my own.

Wangkhang and Nikita Roy started the network a year ago so people working on AI in Canadian media had a place to share what they’re learning. This forum was its biggest yet.

Roy, who also founded Newsroom Robots Lab, an AI training, strategy and product development company for media organizations, set out what she sees coming. AI assistants are moving from chatbots that answer questions to personal agents that act on a person’s behalf, always on and connected to their data. If people get their news through those agents, the agent sits between the newsroom and its readers. She also pointed to the money: Postmedia, Torstar, The Globe and Mail and The Canadian Press sued Anthropic on Oct. 6 over the use of their journalism to build Claude, and she said Google has started paying some news outlets to take part in its AI features.

“What happens to Canadian news media when it seems like Silicon Valley tech companies are determining, and being, that information intermediary between us and our citizens in the audience that we’re catering to?” asked Roy.

Any organization whose customers might soon reach it through someone else’s AI agent faces the same question.

Three sessions followed, from a large newsroom, a two-person newsroom and a veteran technology reporter. Between them, they covered how to build with AI, how to check its work and how to explain it to the public.

How the Toronto Star used AI to sort 4,000 mayoral decisions

Kate Allen and Nathan Pilla of the Toronto Star started their AI talk with a question from political reporter David Rider. Ontario had extended strong mayor powers, which let mayors veto council decisions and pass some bylaws with a minority of council support, to nearly half the province’s municipalities on the promise of faster housing. Rider wanted to know whether the powers were being used to build homes.

Answering him meant reading every strong mayor decision in the province. The challenge is that municipalities post them however they like, as PDFs, Word files, spreadsheets or images, and the team had to ask some for decisions that were never posted. The final set ran past 4,000 documents from 216 municipalities, some a single page long and some hundreds.

Pilla, the Toronto Star’s AI tools and data lead, built the classifier in Python on Google’s Gemini. Each document went to the model on its own so it couldn’t confuse one decision with another. The instructions grew past 10,000 words, with the full legislation, definitions for every question and rules for edge cases, and the team rewrote them repeatedly as strange documents turned up.

The model answered four questions about each document:

• Was a strong mayor power used at all?
• What was the decision about?
• Which power was used?
• Did the decision directly support building homes?

Answers had to come from a fixed list, which Pilla said “reduces hallucinations to very close to zero,” because anything outside the list is rejected automatically. The model also had to explain every answer and list any reason it might be wrong, which showed the team where its rules had holes and which documents needed a human look.

Classifying every document cost less than $60 in AI fees. The team had ordered in Egyptian sandwiches for the reporters who checked the AI’s work, and Pilla joked that lunch was the bigger expense.

“I think the Egyptian sandwiches cost more than the analysis did,” said Pilla, laughing.

The headline finding was that only 2% of decisions directly supported building homes. In one municipality, a mayor used the powers to veto a decision allowing sixplexes. And two mayors used them on pickleball courts (one to approve a new one and one to cancel another).

Those sandwiches fed the reporters checking the AI’s work, and that checking is where the time went.

To know whether the AI’s answers could be trusted, the team first needed to know how people would answer the same questions. Allen, the Toronto Star’s AI reporting lead, gathered 10 reporters with experience covering municipal politics, and three of them answered the same four questions on each of 65 documents. The plan was to treat their shared answers as the correct ones and grade the AI against them. She called it a human verification party.

“Unfortunately, this did not work, and the humans failed the human verification party,” said Allen.

The reporters disagreed on almost every document, she said, and in nearly half the cases they couldn’t agree on whether a power had been used at all. Over three months, the AI changed one answer, once. That proved it was consistent. Proving it was right took a second round, in which Allen, Pilla and Rider wrote themselves a rulebook, coded a fresh sample and argued until they agreed. Graded against that standard, the AI made one error, and a second apparent error turned out to be the humans’ mistake.

Allen then read every decision the AI coded as housing and had a different chatbot look for decisions that could have gone either way. The team reclassified a handful of decisions as housing, even ones only loosely tied to it, to give mayors as much credit as possible.

“Basically, end of story is that the 2% never changed,” she said.

The team had its answer by late last year and published in May. Allen said the months in between taught her what the tool was for.

“We thought this was going to be a time-saving tool because we couldn’t read 4,000 documents, but it took forever. And actually, I think it was an accuracy tool,” said Allen.

Without AI, she said, the Toronto Star would have assigned reporters to code the documents by hand and trusted their answers. The verification party showed they would all have answered differently.

I asked two questions from the floor.

The first was whether the team checked accuracy along the way, since Digital Journal has seen and heard of AI drawing its own inferences over thousands of documents. Pilla said they ran the documents in batches, tested different models and settings, and got the same answers across far more than three passes.

The second was about how mayors wrote their strong mayor decisions. AI models rely heavily on wording, so I asked whether a carefully worded decision could slip through, with the model missing something important because of how it was phrased.

“It could. It definitely could,” said Pilla.

He said the model sometimes caught mayors’ mistakes, such as citing the wrong section of the legislation, and flagged them in its notes. The model reads only its instructions and the document, with no internet access, so at some point it takes a mayor at their word. Allen said she tried tracing every housing decision to an actual zoning approval and found it impossible.

As for learning, Allen said if she were doing it again she’d keep the verification group small with explicit instructions, and she wouldn’t email every municipal clerk to confirm decision counts, which caused chaos and showed the Toronto Star knew more about the decisions than some clerks did. She would still verify.

When it came time to publish the story, the Toronto Star chose radical transparency and published a long, separate story on its methodology. Allen said she saw no complaints about the use of AI. Executive editor Grant Ellis said the newsroom picked a core Toronto Star subject over something flashier because it would build skills it could reuse. Pilla said the same system can classify video of council meetings, podcast audio or images by changing the prompt and the questions.

Any organization running AI over its own records can borrow the Toronto Star’s method: one document per request, answers from a fixed list, a reason for every answer and a human standard to grade against before publishing.

The reporters’ disagreement came down to the documents. Allen said the decisions were so confusing that experienced municipal reporters read them differently, and it took three journalists working from a shared rulebook to set the standard the AI was measured against. The Toronto Star also shared what went wrong along the way, from the failed first round of verification to the emails to every clerk, which is the part most organizations leave out when they talk about AI.

How Indicator runs investigations with AI agents and a two-person team

Craig Silverman runs Indicator, a two-person newsroom that investigates digital deception. He was previously a national reporter at ProPublica and the founding editor of BuzzFeed Canada. He covers the harms of AI and uses it a lot.

“I’m going to talk about using AI to assist with investigations, and assist is kind of the key word there,” said Craig Silverman, co-founder of Indicator.

His rules are short. AI doesn’t write Indicator’s articles or do its deep analysis. It gathers, reviews, monitors and helps with fact-checking, and every investigation ends with a methodology section explaining how AI was used.

“So we don’t ever have it do things where we can’t check it,” said Silverman.

He started by giving the AI context. He moved 15 years of clips and thousands of Evernote notes into Obsidian, a free notes app, and connected it to Claude. He and the AI both write into the same set of notes, backed up to GitHub with version history so he can roll back when the AI wrecks something. The AI can read his newsletter platform, payment system and a separate email account full of newsletters, and it can write only to the notes.

A file of standing instructions tells it to stop describing findings as significant or notable, a habit he said drove him crazy. It weighs evidence using analysis of competing hypotheses, an intelligence-community method, along with a framework Silverman drafted for rating digital evidence. A running file for each project records where he left off, so he can pick a story back up a week later.

For research, he uses Spotlight, an agent system built by Tom Vaillant, founder of the investigative technology studio Buried Signals, who partners with Indicator on its AI tools. In Spotlight, Silverman writes a prompt, the agent proposes a plan, and he approves it before it runs. A second agent acts as a fact-checker on the first agent’s findings, and the results come back to him. On an upcoming investigation into explicit ads on Facebook and Instagram, Spotlight ran for three hours, found 90 sources, gathered 451 files and returned 44 findings, including leads from Chinese-language sources he hadn’t found himself. Its findings come with links he can follow and verify.

Silverman said what makes the agents useful is skill files, step-by-step instructions for specific jobs such as finding who owns a website or tracing an email address.

“You can’t just expect the agent to know how to do this stuff,” said Silverman.

His checking habits work outside journalism too. He uses AI to build banks of keywords, then asks it for searches that would turn up evidence against his own thesis. When he asks a model to review work, he uses neutral wording so it doesn’t simply confirm what he wants. He has one model check another’s output and doesn’t say where it came from, because he said Gemini will tear apart anything it knows came from Claude.

“Just be a journalist and ask, well, how do you know that? What’s the link? Where did that come from?” said Silverman.

Silverman wants Indicator to own its tools. He expects AI subscription prices to double or triple within a year or two, so the team is training open models that run on a laptop with 32 GB of memory, which also keeps sensitive material on its own machines.

“We can find stuff and do things we couldn’t do before because we didn’t have the knowledge or we didn’t have the resources,” said Silverman.

Nora Young on covering AI when the technology keeps changing

Nora Young has covered technology since the 1990s. She created Spark, the CBC Radio technology show that ran for 17 seasons, spent two years as senior technology reporter for CBC National News and retired from the CBC on Oct. 1. She spoke about covering AI rather than building with it.

Young treats AI as a foundational technology like electricity or the steam engine, and borrows Princeton researchers Arvind Narayanan and Sayash Kapoor’s term “normal technology.” In their sense, that means a major technology in line with past ones, discussed without the superintelligence talk she sees as a distraction from AI’s effects on people’s lives. She credits Canadian computer scientist Daniel Wigdor with an analogy: looking at ChatGPT and calling it AI is like looking at a blinking light bulb and calling it electricity.

Her lessons from covering the internet are candid: The early web’s oddness left reporters, herself included, naive about the dangers of social media. Technical experts got credibility on subjects far outside their field. And the applications built on the internet were hard to predict.

What’s new with AI, she said, is the speed of adoption, a chat interface that’s easy to use and hides how it works, and a technology that keeps changing underneath the people explaining it. That makes coverage hard to keep current. What a reporter told the audience six months ago may no longer be true, and she said reporters have to persuade their editors that a story already done needs doing again because the technology has moved on.

Her most practical point came from history. Steam engines took off once factories were redesigned around them, and she asked what the AI version of that redesign will be.

“Technological change regularly outstrips the regulatory, moral, and organizational environment around it,” said Young.

She also wants journalists to stop describing AI in human terms. Phrases like “going rogue” have become shorthand, she said, and they get in the way of the public understanding what these systems do.

Why the Canadian AI Journalism Network wants newsrooms sharing more

Grant Ellis, the Toronto Star’s executive editor, used part of his talk to argue that Canadian newsrooms should share far more of what they learn about AI. He said the Toronto Star’s legal and regulatory teams already work with peers across the country and around the world, and he wants newsrooms to do the same, competing on stories but sharing standards, skills and technology. He said Canada lacks the philanthropic and government funding that backs that kind of shared work in the U.S. and Europe.

“This group is the sum total of people who are gathering about AI and journalism right now. It should be hundreds and hundreds more,” said Ellis.

Other industries already work this way. Technology leaders in banking, energy and health compare notes at peer forums because what someone in the same seat learned last quarter is the fastest guide to what to try next. The room on Wednesday was the newsroom version: a $60 classification run, a verification panel that couldn’t agree with itself, a two-person newsroom’s research agents and a veteran reporter’s reminder that explanations of AI expire quickly.

The Canadian AI Journalism Network describes itself as an industry network committed to fostering the responsible use and development of AI technologies in journalism across Canada, through open knowledge sharing and peer-to-peer conversations. Its next forum, in November, takes on the question Roy raised: what happens to the news when audiences get it through AI.