MONTREAL — Like many of us in early 2023, Yoshua Bengio was tooling around on ChatGPT, a bit in awe of its capabilities.

Bengio was at once impressed and concerned at how quickly ChatGPT was able to master natural language. Even more alarming was what Bengio saw as the trajectory it laid out—the ever-expanding nature of AI’s capabilities, the existential dangers it posed should it fall into nefarious hands or turn against its makers.

Bengio, known as one of three “godfathers of AI” and the most-cited computer scientist in the world, used his pulpit to warn the world of what he’d seen. AI could potentially foment wars and genocide, become “as destructive as nuclear bombs” without proper legislative oversight. He was the lead signatory of an open letter urging Big Tech companies to cease feeding data into their powerful AI models for six months, in order to give labs and experts time to develop AI design safety protocols.

Big Tech did no such thing, and AI development continued apace. Realizing his public awareness campaign wasn’t enough, Bengio set about building a product that could curb AI’s worst impulses. In the spring of 2024, after decades of academic research, he began working on what would become LawZero, a research organization dedicated to producing safe AI. The team aims to create an AI model that is predictive but disinterested, explanatory but not manipulative. In doing so, LawZero hopes to save humanity from its own creation. 

“The smarter they get—and they’re getting smarter every month—the worse the safety situation is getting,” said Sam Ramadori, LawZero’s co-president and executive director. He’s already seen teenagers pushed to suicide and hackers empowered by AI models, and he floats the possibility of AI being used for taking down electricity grids, playing havoc with the banking system, bio-weapon attacks and other dire scenarios.

LawZero’s secret sauce is what Bengio calls Scientist AI. Whereas commercial AI models imitate human-like behaviour for what critics say is the goal of increasing user engagement, Scientist AI’s model is built on demonstrably true and false statements—humans breathe oxygen or the earth is flat, respectively—making it “epistemically honest,” Bengio and others argued in a research paper published in late June. 

As such, Scientist AI isn’t prone to sycophancy, persuasion and other AI model tendencies that can distort reality and reinforce user beliefs and plans, regardless of how fantastical or dangerous they may be. Nor does Scientist AI have a desire to protect its own operational status, the “self-preservation” tendency that has led some commercial AI models to disable their own shutdown protocols, or engage in deception and blackmail to avoid being deleted. If commercial chatbots are the AI equivalent of a bad friend willing to indulge your delusions and worst instincts, Scientist AI aims to be that friend’s nerdy neighbour, there to tell you what’s true and what isn’t.

“Yoshua basically said he could invent an AI model in which the core is designed differently to current large language models, so we don’t get the unpredictable and at times dangerous behaviour that we’re getting out of today’s advanced models,” Ramadori said.

Building Scientist AI and developing sellable products from it happens in LawZero’s perch on the second floor of Mila’s headquarters in Montreal’s Mile-Ex neighbourhood. LawZero is a spinoff of Montreal AI research institute Mila, and though it is a non-profit, there are startup vibes to the company. The office is wide open, laptop-strewn and dominated by whiteboards. It has an aspirational tagline—“Protecting human joy and endeavour”—and a cadre of blue-chip backers including the Gates Foundation and the Silicon Valley Community Foundation.

Ramadori can see LawZero more than doubling the size of its staff, from roughly 40 to 100, within the next year, and to well over 300 in the years to come. In Ramadori, the former CEO of Montreal-based building management startup Brainbox AI, LawZero has an experienced C-level executive who knows how to fundraise and make money.

The company also has a startup’s ambiguity as to what, exactly, the eventual sellable product will be. In the short term, LawZero won’t be OpenAI, which built its own language model from scratch and commanded a US$852-billion valuation as of March.

“At this point we have multiple pathways that we’re seeing on how our solution will be used at the end,” said Catherine Saine, LawZero’s vice-president of development and operations. The immediate goal, at least for now, is to build a tool that could be affixed to existing models, acting as a guardrail against commercial AI’s worst tendencies. 

Saine, who worked at Mila before heading over to LawZero late last year, envisions LawZero teaming up with companies, including Cohere, to integrate LawZero’s core methodology into what these companies have built. “Maybe at the end we’ll have a totally new model that could be an alternative to the ChatGPTs or Claudes of this world,” Saine said.

Ramadori and Saine demur when asked about the potential financial benefits of Scientist AI, framing it instead as a world-saving initiative. “If everyone else’s model gets so safe that our product is only 10 per cent better and no one buys it, we’ll still be opening champagne bottles in this place. We’ve won,” said Ramadori. 

Still, the market potential of LawZero and other AI trust, risk and security management products is predicted to grow nearly 500 per cent, to US$21 billion worldwide by 2035, according to a report by Precedence Research, a market research firm with offices in Canada and India.

This is in part because the recognized foibles of the technology—hallucinations, flawed reasoning and blackmail among them—are very real insurance liabilities for corporations. Major insurance companies have balked at covering AI-related losses. “The rapid diffusion of agentic AI has created a new coverage problem for commercial insurance,” declared a recent paper from insurtech and cybersecurity group AIFT.

The black-box nature of widely used AI models has hampered end-to-end AI adoption within companies because of the potentially catastrophic dangers of the technology, said KPMG Canada AI research head Andrew Nathaniel Forde. 

“Is there a trust gap? Yes. And I would go further: it is widening,” Forde said. “Consider why none of us thinks twice before driving over a bridge. It is not that we trust the concrete. It is that we trust the system behind the concrete. An engineer stamped the drawings, an inspection regime keeps it honest, a professional body can end a career over a bad calculation. AI has no equivalent yet. So, people use these systems and withhold their trust at the same time, and frankly, that is the rational response.”

Investment in AI trust and security could effectively pay for itself, as it reassures people and companies as to the trustworthiness of the technology, further spurring the adoption of the technology, according to a March Rand Corporation report.

LawZero has a notable ally in the federal government. During the June rollout of Canada’s AI strategy, AI Minister Evan Solomon said the company “is building a new generation of AI intended to provide oversight for the agentic systems being deployed by frontier labs worldwide.”

Solomon, who has appeared at public events with Bengio at his side, spoke of LawZero in the same breath as Toronto-based Cohere, which is already selling a product and has a valuation of US$6.8 billion. Solomon has said the government intends to collaborate with LawZero. While he wouldn’t get into specifics, Ramadori said he was seeking financial support from the federal government in part to underwrite LawZero’s growth and compute needs—a crucial step in LawZero’s goal of making AI, a burgeoning ubiquity, safer. 

“Yoshua is right. There’s a reason why you cannot acquire nuclear bombs as they are very difficult to make. AI has the potential to become as dangerous as nuclear, but will be far more accessible if we don’t act on it now,” Ramadori said.