OTTAWA — Artificial Intelligence Minister Evan Solomon says he used a Google AI tool to brief himself on a previous Liberal bill that would have regulated the sector he’s responsible for, and praised it for its accuracy.

When The Logic created briefings using the same steps he described, however, the results misled about a core feature of the bill twice in three tries.

Solomon enthused about Google’s NotebookLM in an interview with Maclean’s posted Monday, answering a question about how he uses AI in his free time.

“I used AI today,” Solomon replied. “I had to get a briefing on Bill C-27, a piece of legislation from a few years ago that has to do with privacy and data. I uploaded it to Google NotebookLM, asked the software to create a podcast and listened to it on the 15-minute drive to my constituency office.”

He played some of the audio and went on: “I’ve sent that to tons of my staff. It’s pretty good, right? … No hallucinations, either.”

“Hallucinations,” the term for when a generative AI model makes things up and presents them as factual, are large language models’ Achilles heel. Because they can present true and false things with equal conviction and authority, it can be hard for those without expertise in a given subject to tell which is which.

Google warns in a small but ever-present banner across the bottom of NotebookLM windows that its tool “can be inaccurate,” adding: “Please double-check its responses.”

The Logic asked Solomon’s office for a copy of the audio file that he used to brief himself.

“I don’t have it and I’m sure it’s deleted after he used it but I’m sure the same podcast is generated with a similar prompt so you could most likely replicate it,” press secretary Sofia Ouslis responded.

In a follow-up exchange, Ouslis sent a statement from Solomon himself: “Like millions of Canadians, I’ve experimented with free AI tools such as NotebookLM to see how they summarize complex information. But government policy is never based on an AI summary—it’s based on rigorous human briefings from officials and experts. 

“I talk openly about my own experiments because it shows both the promise and the pitfalls of these tools,” the statement continued, “and why Canada’s approach to AI is grounded in trust, safeguards and human oversight.”

Creating an audio briefing of the type Solomon used is easy. NotebookLM allows users to upload files and then type queries about their contents or use various summarizing functions. Generating a podcast-style audio file—with a host-voice and a reporter-voice chatting back and forth for 15 to 20 minutes—takes one click and a few minutes’ wait.

These are not, of course, two actual people having a conversation, but one machine-driven summary presented as a dialogue. (As convincing as the AI-generated voices are, occasionally one of them might sputter some bit of code like “hashtag-tag-tag-outro” at the end of an otherwise normal sentence.) 

Despite Solomon’s spokesperson’s suggestion, NotebookLM isn’t like a pocket calculator where pushing the same buttons in the same order will get you the same results. Each time you run a summary tool, it produces different outcomes—sometimes accurate, sometimes with hallucinations.

Justin Trudeau’s Liberals introduced Bill C-27 in 2022. They never passed it, but it’s important context for the work Solomon is now responsible for, as Canada’s first dedicated AI minister under Prime Minister Mark Carney.

It was a complicated and expansive bill that is not easy to summarize quickly.

The first two sections of the legislation would have overhauled federal law on how the private sector protects personal information and created a new tribunal to enforce penalties for violations of that new Consumer Privacy Protection Act (CPPA).

A third section of the bill would have created the Artificial Intelligence and Data Act (AIDA), a first legislative attempt to regulate how the private sector uses AI.

(Many leaders of AI-oriented businesses didn’t like it; nor, for different reasons, did many digital rights advocates.)

Critically for someone who needs to understand the bill, the privacy tribunal would have had no authority to enforce the AI law.

In fact, the different chunks of the legislation were so unconnected that the House of Commons Speaker made an unusual ruling that they should be voted on separately. The privacy law and the new tribunal had nothing to do with the artificial intelligence law other than sharing a “broad theme,” the ruling said.

To see what Solomon might have learned from NotebookLM, The Logic uploaded the text of Bill C-27 and made a podcast-style summary.

After a chatty and essentially accurate account of most of the bill, including noting the AI law would give an unspecified minister new enforcement powers, the summary got a key element wrong.

“How do CPPA, AIDA and this new tribunal structure fit together?” the imaginary host says toward the end of the file. “What’s the overall goal for Canada’s digital space?”

“I think the interconnectedness is really the core idea here,” the reporter-voice replies. The new privacy law would deal with privacy and the AI law would deal with AI, but: “Layered over both, you have the privacy commissioner doing investigations and audits, and the new tribunal, providing that appeal mechanism and the power to impose meaningful penalties under CPPA. It’s meant to be a cohesive system.”

This might sound reasonable but it’s incorrect. The mention of penalties being imposed specifically under CPPA, the privacy law, was right. But the overall message was wrong. The privacy commissioner and the new tribunal would not have been “layered over” both the new privacy and AI laws.

The Logic tried creating two more “podcasts” based only on the text of Bill C-27.

The second nailed the role of the privacy tribunal and (unlike the other two) correctly mentioned the possibility of a new AI commissioner who might oversee enforcement of the AI law.

The third audio summary made the same error as the first, if not as egregiously, implicitly lumping the privacy and AI components together again.

“We have all these new rules,” the host-voice says. “CPPA for data privacy, AIDA for AI. It really begs the question: who’s the referee in all of this? And maybe more importantly, for you the listener, where do you go if you have a concern, if you think your rights under these new laws have been violated?”

“Enforcement is critical,” the reporter-voice says. “Under the CPPA, the privacy commissioner of Canada continues to play a central role. That office isn’t going away. They still have powers to investigate complaints, conduct audits of organizations’ privacy practices. But now—a key change—they recommend penalties to a brand new body.”

“A new body,” the host-voice interjects.

“Yes, the Personal Information and Data Protection Tribunal,” the reporter-voice resumes. “This is newly established by this legislation. Its job includes hearing appeals about the commissioner’s findings or orders under CPPA, and crucially, it’s the tribunal that will actually impose those significant penalties we talked about.”

Although everything the reporter-voice says is accurate in isolation, it’s a flawed answer to the host-voice’s question about enforcing both laws.

Ouslis told The Logic that Solomon had “received multiple human briefings on Bill C-27 and its history.”

“Those began well before the Maclean’s interview and briefings continue regularly as the file evolves,” she wrote. “The experiment with NotebookLM is not a substitute for official briefings—it was part of his broader effort and experimentation to understand how Canadians themselves might use AI to digest complex legislation.”

Asked whether Solomon still believes that NotebookLM does not hallucinate, Ouslis wrote that he believes that “AI outputs must always be tested against trusted human sources.”