LONDON — Deep learning pioneer Yoshua Bengio said he supports the creation of a Canadian AI safety institute, as he embarks on a six-month sprint to deliver a landmark assessment of advancements in the field, designed to help the world’s policymakers better appreciate what he believes are the potentially catastrophic risks of the technology.

At the AI Safety Summit this week at Bletchley Park, the U.K. and U.S. both announced plans to create organizations that will test artificial intelligence models and examine the technology’s potentially harmful capabilities. Canada’s Innovation Minister François-Philippe Champagne said Thursday he’s also considering setting up such a body.

In an interview with The Logic in London on Friday, Bengio—scientific director of the Montreal-based AI institute Mila and co-chair of the federal government’s AI advisory council—expressed support for the initiative. “I think we need that,” he said. “We’re lucky in Canada to have an amazing talent pool that could contribute to [solving] this global problem.”

Meanwhile, he has his own task. At the close of the summit on Thursday, British Prime Minister Rishi Sunak announced that Bengio, a celebrated AI researcher and lately one of the most prominent voices warning of its potential dangers, will lead the writing of a “state-of-the-science” report. The paper will detail the capabilities of so-called frontier systems, the multi-purpose models that power applications like ChatGPT. 

The two-day event saw 29 governments, including the U.S. and China, sign the Bletchley Declaration, agreeing to cooperate more closely on AI safety. The summit helped improve policymakers’ grasp of the problem and create some alignment on the need to solve it, according to Bengio.  

“There are a lot more governments—and especially among the important ones—that understand much better the risk [of AI] and have an informal commitment to do something,” he said, citing the need for both research resources and domestic rules. “They won’t be able to regulate something … that they don’t master.”

Bengio’s report is meant to summarize the scientific literature on the technology, helping governments understand it. “There’s a rapid increase in the number of papers being written and submitted in machine learning on the topic of AI safety [and] governance,” said Bengio, crediting the increased research output to public discussions, media coverage and open letters about the technology’s risks. He’s signed at least three notable ones, calling for a pause to the development of more powerful models; for mitigating extinction risks to be made more of a priority; and for more spending on safety. 

To write the report, Bengio expects to call on experts in AI and other fields, including ethics, national security, biosafety, chemical weapons, and the social and political sciences. “Some of the questions are about the impact of technology on society,” he said, while he’s also expressed concern about malicious actors using AI to generate disinformation or new munitions. However, the report will not make recommendations to participating governments.

Some AI scholars have said warnings focused on the technology’s catastrophic risks—such as those expressed by Bengio and fellow Canadian AI pioneer Geoffrey Hinton—are a distraction from present concerns about bias, discrimination and labour displacement. At an event hosted by Tortoise Media in London on Friday, Bengio said governments should be trying to address both sets of issues.

The initiative Bengio will lead is based on the Intergovernmental Panel on Climate Change (IPCC), established in 1988. Countries with large resource sectors and industry firms have reportedly lobbied the UN to water down the IPCC’s conclusions. Bengio told The Logic he’s worried his work could face similar pressures, but said the tight timeline on which he’s working could limit the potential for interference.

Eight major tech firms—including Amazon Web Services, Anthropic, Google, Meta, Microsoft and OpenAI—agreed at the summit to give the U.K. and partner countries access to their models for testing. But Bengio said governments should require that AI companies demonstrate to government-commissioned independent scientists that their products are safe, rather than making authorities search for misuse. “In order to get your licence, you have to show that your system is OK,” he said. “And if you don’t behave properly—and you’re going to be monitored—then we take out your licence.”  

In an interview with The Logic on Thursday, Champagne did not rule out such a licensing regime, but said he would watch whether allies adopt one, and that any measure would need to be coordinated internationally.