In late May, retired AI researcher Ramón Brena was checking his email in his home office in San Luis Potosí, Mexico, when a message popped up that got his attention. 

“Invitation to join Hinton, Bengio & Amodei,” the subject line said. It came from Max Tegmark, president of the Future of Life Institute, a Narberth, Penn.-based non-profit. Brena read on to find out what he was being invited to do.

Two months earlier, he’d joined one of those boldface names—Canadian “AI godfather” Yoshua Bengio—in signing an open letter from the institute calling on AI labs to pause the training of more powerful AI systems out of concern for where they might lead. Now, the Future of Life Institute was leaning on the power of Bengio’s involvement—along with fellow British-Canadian AI pioneer Geoffrey Hinton and Anthropic CEO Dario Amodei—to persuade him to sign something more.

“We think it is essential to also mainstream and legitimize debate about AI’s most severe risks,” the email said. Attached was a “new and concise” 22-word Statement on AI Risk from the Center for AI Safety that read: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

You may have heard of that 22-word statement; it attracted international attention and headlines. Ever since the launch of OpenAI’s ChatGPT in November 2022, AI has become a topic of dinner table conversation—and so have the ominous warnings from big-name scientists that the apocalypse may be nigh.

Last week’s drama at OpenAI, in which CEO Sam Altman was fired and rehired over the course of four days over reported tensions with safety-minded board members—including Canadian chief scientist Ilya Sutskever—highlights the stakes at play.

Canada has played a key role in the development of AI, and hopes to capitalize on the economic opportunity it offers. But for better or worse, in Hinton, Bengio and now Sutskever, Canada has also produced the most prominent global voices warning of the technology’s potential dangers. 

All these headlines about AI risk are related to a coordinated and well-funded global effort to convince lawmakers and the public that AI poses an existential threat to humanity. It’s a movement of people and institutions whose worldview has been influenced by a shared set of ideas and who are often interconnected.

For example, the Future of Life Institute and the Center for AI Safety have a common significant donor, the U.S. granting organization Open Philanthropy. Open Philanthropy has also donated US$30 million to OpenAI and its former CEO, Holden Karnofsky, was once a board member at the startup.

Those who hold this shared set of ideas, who say they are concerned with AI’s existential risks, often call themselves longtermists or effective altruists. A more recent, and more pejorative, descriptor is the term TESCREAL, an acronym coined by philosopher Émile Torres and former Google AI researcher Timnit Gebru to describe a bundle of philosophies: transhumanism, extropianism, singularitarianism, cosmism, rationalism, effective altruism and longtermism.

One of the major beliefs the TESCREALists have in common, Torres says, is a heaven or hell view of technology—that it will eventually transform humanity, either for better or worse. Get it right and we can collectively shed the prisons of our human bodies and, in the words of futurist Ray Kurzweil, “multiply our effective intelligence a billion-fold by merging with the intelligence we have created.” Get it wrong and, if we’re lucky, we could end up with superhuman AI treating us like pet Labradors. If we’re unlucky, we could accidentally be wiped out if that AI takes a well-meaning human instruction like “manufacture as many paperclips as possible” a little too literally. 

“It’s either going to be complete annihilation, or utopia,” Torres said, summarizing the TESCREAList tension between longing for and fearing superintelligent AI. “Therefore, we should be careful in creating AGI so we get it right and get utopia—rather than end up just going extinct.”

Hinton and Open Philanthropy did not respond to requests for comment and the Center for AI Safety did not provide an on-the-record response. Bengio said in an emailed statement that he believes his work signing and promoting various open letters on AI risk “​​is essential to protect democracy, society, humanity and our shared future against AI’s immediate and potential threats,” but he is not affiliated with the Future of Life Institute or the Center for AI Safety and does not identify as an effective altruist.

Anthony Aguirre, executive director of the Future of Life Institute, said in an emailed statement that the organization has not received funding from Open Philanthropy since 2020 and is not affiliated with effective altruism. The organization reached out to some people in its network about signing the Center for AI Safety Statement because “the statement reflected our shared goal of reducing the risks of the out-of-control, unchecked race to develop increasingly powerful advanced AI systems,” he said.

Brena, the retired AI researcher, said he doesn’t think it’s productive to put too much focus on “the idea that AI is going to wipe humanity from Earth,” an idea he considers “anthropomorphic” and not well founded in science. His other, more immediate concerns about artificial intelligence—like the potential for job loss and for tech companies to take people’s data without consent—are the reason he signed the original open letter calling for a pause on the development of more powerful AI systems. He gave some thought to signing the second statement but was uneasy about it. He eventually decided not to.

“I stayed with the feeling that something was off about this one,” he said.

In addition to his concern that focusing on the extinction risk from AI distracted from the technology’s present day harms, he felt the comparison of AI risk to pandemics was manipulative, given the COVID-19 restrictions and waves of deaths the world had recently endured.

Brena was not aware that the Future of Life Institute and Center for AI Safety have both received funding from Open Philanthropy, which is in turn mainly funded by billionaire Facebook co-founder Dustin Moskovitz and his wife, Cari Tuna. The links between the organizations promoting an AI doom message deserve more scrutiny, he said.

“Following the money, where the support is actually coming from?” he said. “I think that’s a very clever idea.”

---

In the grey October drizzle, AI Governance and Safety Canada executive director Wyatt Tessari L’Allié stepped onto a platform on Parliament Hill to speak at the AI Safety and Ethics Rally. He had to make himself heard over the din of another, a larger, unaffiliated demonstration happening at the same time. “Hello, everyone,” he joked into a microphone. “I made the very wise decision of bringing paper notes on a rainy day.”

Only about 20 people in total attended the rally, one of a handful held at cities around the world that day to call for a ban on the creation of superintelligent AI. L’Allié said his Ottawa-based organization isn’t itself calling for a pause on AI development, but he shares the concerns of those who are.

L’Allié, 37, used to think climate change was the defining issue of our time, leading him to run for the federal Green Party and speak at environmental rallies. In 2015, he started reading books that convinced him that AI risk, not climate change, is his generation’s defining issue.

One of the books he read was Global Catastrophic Risks, co-edited by University of Oxford philosopher Nick Bostrom. The book includes an essay by Eliezer Yudkowsky, a co-founder of the Berkeley, Calif.-based Machine Intelligence Research Institute, which argues assuming superintelligent AI will be friendly to humans “implies an obvious path to global catastrophe.” Yudkowsky makes the case that it’s imperative to figure out how to make sure AI is aligned with human values before we develop it.

Bostrom and Yudkowsky are foundational—and controversial—thinkers in the movement to influence public opinion and policy on AI existential risk, with strong links to the so-called TESCREAL bundle of philosophies.

In January, Bostrom’s views attracted scrutiny after Torres publicized a racist email he had sent to a listserv in the ‘90s, in which he used a racial slur and said he believes Black people are less intelligent than white people. Bostrom published a response in which he apologized, called the email “disgusting,” said his use of a racial slur was “repulsive” and that the email does not accurately represent his past or current views, but did not explicitly disavow the idea that some racial groups are less intelligent than others.

Both Yudkowsky and Bostrom are transhumanists, a philosophy that advocates for the use of technology to transcend the limits of our bodies. Transhumanists often hold a quasi-religious belief in a coming end time brought on by the singularity, or the merging of human intelligence and artificial superintelligence.

Yudkowsky is perhaps better known as creator of the blog LessWrong, “a community dedicated to improving our reasoning and decision-making.” LessWrong is affiliated with the rationalist movement—which attempts to remove bias from thinking and see the world as it actually is—and the effective altruism community, which encourages people to orient their careers and charitable giving towards doing the maximum amount of good in the world.

Prominent effective altruist William MacAskill popularized the concept of longtermism in his 2022 book What We Owe the Future, which argues we should focus on actions with the greatest positive impact on distant generations—including preventing human extinction at the hands of AI. Under his influence, effective altruists have mostly shifted their focus from improving health in impoverished nations to AI safety.

These philosophies have drawn considerable criticism. The transhumanist project of enhancing human intelligence and eliminating disabilities has been called eugenicist—a charge that has also been levied at Bostrom for fretting about less intelligent people outbreeding the smart. Many have held up the downfall of effective altruist Sam Bankman-Fried, who claimed to have launched the failed crypto-trading platform FTX in order to make as much money as possible and give it away, as an example of how the philosophy taken to extremes can end up looking a lot like “the ends justify the means.”

Criticism of effective altruism has heated up again in the wake of the OpenAI boardroom drama. In an opinion piece for The Information, OpenAI investor Vinod Khosla said the firm’s “board members’ religion of ‘effective altruism’ and its misapplication could have set back the world’s path to the tremendous benefits of artificial intelligence.” Aidan Gomez, CEO of Toronto AI startup Cohere, as well as Shopify CEO Tobi Lütke, Coinbase CEO Brian Armstrong and prominent tech reporter Kara Swisher have also criticized the philosophy in the last week.

Critics have also noted that focusing on people in the far-off future has the convenient side effect of giving those in positions of wealth and privilege an excuse to avoid doing something about the problems faced by those less fortunate today. After The New York Times published an article in May in which Hinton said he had left his job at Google to speak freely about the dangers of AI, some of his former colleagues expressed outrage that he hadn’t spoken up when Google AI ethics researchers lost their jobs after publishing a paper about the harms the technology poses to marginalized communities, or when employees organized a walkout over the company’s contracts with the U.S. Department of Defense.  

Despite the critics, this stew of ideas is everywhere in the world of big tech. Prominent tech investor Peter Thiel and Russian-Canadian Ethereum co-founder Vitalik Buterin have expressed thoughts consistent with aspects of rationalism, transhumanism or longtermism.

Two other noted Canadians—Elon Musk and his former partner Claire Boucher, known professionally as the musician Grimes—reportedly met after tweeting the same pun referencing a LessWrong post about AI risk. Musk and Buterin have made significant donations to the Future of Life Institute, the organization behind the open letter calling for a six-month pause on AI development.

The boardroom coup at OpenAI that captivated the tech world’s attention last week appears to have been rooted in these philosophies as well.

Tasha McCauley and Helen Toner, two of the board members involved in firing Altman who have since been replaced, have links to organizations that are backed by or explicitly affiliated with effective altruism, while Sutskever has expressed views about AI risk that are consistent with those held by effective altruists and longtermists. Sutskever had spearheaded a technical breakthrough (a model called Q*, or “Q-Star”) prior to the boardroom coup, The Information reported, raising concerns among safety-focused staff about whether the company had sufficient safeguards in place.

Even Altman, who appears to have been on the other side of the effective altruist-linked campaign to reduce AI risk during the debacle, has expressed views consistent with other letters in the TESCREAL bundle. Altman was an effusive fan of the now-defunct rationalist blog Slate Star Codex and is one of 25 people who paid US$10,000 to join a waitlist for a transhumanist startup that proposes to euthanize customers and preserve their brains, in the hopes of one day uploading their consciousnesses into the cloud.

This year, these philosophies have begun to receive mainstream attention from lawmakers and the public, in the form of concern about AI’s apocalyptic potential.

Most of the organizations behind the U.K.’s high-profile AI Safety Summit held in early November have links to effective altruism. Open Philanthropy, the effective altruist grantmaker that funds the organizations behind the two prominent AI risk letters, is also bankrolling the salaries of tech fellows working in the U.S. Senate and other government departments. In Canada, Open Philanthropy has funded at least six grants to researchers at Canadian universities and institutions working on projects related to AI ethics and alignment with human values.

At the AI Safety Summit, Canada’s Innovation Minister François-Philippe Champagne said he’s considering joining the U.K. and the U.S. in creating AI safety organizations. In an interview with The Logic, Champagne said “everyone acknowledged” AI’s potential existential risk to humanity at the summit, “and we are determined to take action.”

Kyunghyun Cho, an AI researcher and an associate professor at New York University, said he believes the abundant funding from Effective Altruist organizations is influencing junior researchers’ decisions about whether to enter the field and what type of research to pursue.

“Academics are always craving funding,” he said. “The more junior faculty members moving into this field of AI, they probably have been influenced somewhat by the abundance of funding.”

L’Allié has himself been the beneficiary of the effective altruism movement’s deep pockets. In 2022, he received two grants of US$87,000 and US$17,000 from the Long-Term Future Fund, an EA-linked endowment that “aims to positively influence the long-term trajectory of civilization” and receives funding from Open Philanthropy. The money helped get the organization off the ground, though L’Allié noted the grants were for individuals and not AIGS Canada, which he said is funded by small and medium-sized donors and does not accept foreign funding.

Asked if he’s heard of the acronym TESCREAL and what he thinks about criticisms of effective altruism and related philosophies, L’Allié laughed.

“I think it’s entirely valid to question the motives of people,” he said. “Big Tech has a lot of problems, and we should be very on our guard.” But he doesn’t see much to worry about: “The people that I’ve met in the EA movement, for the most part, they’re well-meaning nerds.”

L’Allié said he’s excited about the momentum that’s developed behind the concept of AI safety this year, a formerly niche and technical issue. The open letters, signed by scientific luminaries such as Bengio and Hinton, played a huge role in changing the conversation, he said.

“It’s huge. It’s just logically gratifying to know that we weren’t crazy.”

---

Well-meaning or not, Richard Sutton—one of three Canadian names typically described as a “pioneer” or “godfather” of AI, along with Hinton and Bengio—has some serious concerns about the growing movement sounding the alarm about AI’s hypothetical risks to humanity.

Sutton said Bengio approached him about signing the Future of Life statement calling for a six-month pause on AI development. His answer was a resounding no.

“I wasn’t interested at all. I guess I was not approached on the second one because it was clear what my position was going to be,” Sutton said. He believes the statements are part of a coordinated public opinion push: “A concerted effort in a very deliberate way to change the Overton window. And I’m very unhappy with it.”

Sutton has put considerable effort into understanding the increasingly influential conversation around AI risk, reading books by Bostrom, Yudkowsky and Future of Life Institute president Max Tegmark (who in June shared a stage with Bengio at a June Munk Debate event where they argued in favour of the resolution “AI research and development poses an existential threat”). He has come away unimpressed.

Sutton said he found those authors tend to present hypothetical scenarios where runaway AI could lead to disaster that he doesn’t find particularly plausible.

“I look for it, and I do not see an argument,” he said. “Instead, there just seems to be people manipulating. They’re using all the levers to generate fear.”

It’s unclear whether Hinton and Bengio share the worldview of Bostrom, Yudkowsky, Tegmark and others, or just their concern about AI risks. In a blog post, Bengio described a book by computer scientist Stuart Russell—whose Center for Human-Compatible AI is funded by Open Philanthropy and Effective Altruism Funds, the same bundle of grantmaking reserves linked to the movement that provided funding to Ottawa’s L’Allié—as being a turning point in his “awareness of a possible existential risk for humanity if we do not maintain control over superhuman AI systems.” Bengio’s statement to The Logic did not directly address a question asking what impact, if any, TESCREAList thinking has had on his views.

Sutton said he’s dismayed by Bengio and Hinton’s willingness to lend credence to the global campaign on AI risk.

“Yoshua and Geoff have gone to the side of fear, and I’m just extremely disappointed in both of them,” he said. “They also do not provide any worthwhile arguments, in my opinion.”

Other figures in Canada’s AI community are similarly concerned about the credibility and mainstream attention the movement is receiving, thanks in part to Bengio and Hinton. Amir Hajian, vice-president of data science at Toronto-based Arteria AI, said the success of the AI doom message may help companies with a lead—such as OpenAI, whose executives signed the Center for AI Safety statement—keep their dominant position.

“It looks like a way for Big Tech to build a moat around itself,” he said. Stringent regulations about safety and alignment are expensive to follow, he said—“you’re eliminating everyone else who doesn’t have big money, like us.”

Nicole Janssen, co-CEO of Edmonton-based applied AI company AltaML, said Canadian policymakers don’t seem to be fixating on the doomsday message, to her relief.

“Canada’s government recognizes that this is the biggest opportunity we will have as a country in our lifetime,” she said. “If we do not grab ahold of this opportunity, we will not compete.”

Bengio and Hinton certainly do have the ears of lawmakers in Canada and abroad, however. At the U.K. AI Safety Summit, Champagne cited Bengio’s recent appointment to chair an international study of the potential risks and promise of frontier AI systems as an example of how Canada is “front and centre when it comes to thought leadership” on the issue. 

Bengio and Hinton have also continued to lend the power of their names to open letters on AI risks. Both scientists signed one in late October and Bengio signed another in advance of the U.K. AI Safety Summit.

The first letter, “Managing AI Risks in an Era of Rapid Progress,” laid out the heaven-and-hell case once again. Get AI right, and we could end sickness, save the environment and improve our quality of life; get it wrong, and we could end up unleashing a malevolent superintelligence that could take control of our financial systems, militaries and supply chains.

The all-powerful AI they describe may be around the corner, or it may never arrive. But the movement working to convince us to fear it is very much here. The world will have to rely on human—not artificial—intelligence to decide whether to dismiss it, or to start building bunkers.

With files from David Kawai in Ottawa