MONTREAL — The next advances in the capabilities of AI could well be made by AI itself, as tech giants and startups automate more machine learning research and development.

Many firms are aiming for so-called recursive self-improvement (RSI), a process in which AI models design and train new and improved versions of themselves. Developers have seen performance gains from using their current systems to do engineering tasks and run machine learning experiments.

Now, they’re building toward full RSI, which would require AI to effectively follow the scientific method of coming up with an idea, implementing it, analyzing the results and refining its approach until it makes a breakthrough. Google has been “finding areas where we’re able to do the full loop, and that’s amazing,” Tulsee Doshi, senior director and head of product for the firm’s Gemini models, told The Logic.

Since AI models are fundamentally a kind of software, improvements in their programming capabilities could also make them better at developing their successors. AI firms and all kinds of other businesses are using AI agents to write an increasing share of their new code—75 per cent at Google, and over 80 per cent at Anthropic, as of this spring.

Doshi’s team has worked to improve Gemini’s performance at the various tasks that human developers do, like building fresh features, restructuring existing computer scripts and reviewing and testing new code. Some Google AI staff have always been “very code-pilled,” she said. For example, Montreal-based research scientist Danny Tarlow, who previously led programming-related work at Google DeepMind, predicted that coding capabilities would drive improvements in other model capabilities.

Last month, the firm released a new Gemini model designed for voice-based interaction. Researchers used AI to draft and improve the system instructions for the product, hidden rules it consults before tackling users’ questions or tasks. Google’s RSI system helped it “quickly iterate on finding an optimal prompt that would meet our user experience goals,” Doshi said. The technology has had an even greater impact on more fundamental parts of the development of Gemini models, she added, though she declined to disclose details for competitive reasons.

Still, no one has yet achieved true recursive self-improvement, according to Hugo Larochelle, scientific director of the Mila AI institute in Montreal. “You give one mandate to an AI and it builds the next one—it still seems fairly far from that.”

So far, most gains have come in what Larochelle calls “day science”—work-hours tasks like coding and tweaking experiments, trying out variations, checking results and writing up findings. AI agents can’t yet do the dream-hours “night science” of thinking about which machine learning problems remain unsolved and deciding which to tackle next, said Larochelle, whose own work focuses on AI for AI research.

Still, French challenger Mistral is betting on autonomous AI research as it rolls out new models to rival Anthropic and OpenAI. “Recursive self-improvement is one of the next paradigm changes that will allow us to accelerate and close even further and faster the gap with our competitors,” said Pierre Stock, vice-president for science, citing early success in using AI agents to fully implement new research findings.

Meanwhile, star researchers have launched a new cohort of startups to develop recursive self-improvement. In August, Google DeepMind chief scientist Jeff Dean departed with three of the firm’s most prominent engineers to found Discovery Loop, which is trying to automate the scientific method. Toronto-based Radical Ventures co-led the startup’s initial funding round.

San Francisco-based Recursive, as the name suggests, is also trying to achieve RSI. Founders include Jeff Clune, a University of British Columbia professor and former OpenAI researcher; several of his students have also joined the firm. Clune previously co-authored an August 2024 paper detailing how an AI scientist could generate and test “novel research ideas,” producing results good enough to meet the acceptance criteria for academic conferences.

Other well-funded startups in the field include Ineffable Intelligence, founded by David Silver, a longtime Google DeepMind executive and an alumnus of the University of Alberta’s reinforcement-learning school of AI; and Ricursive Intelligence, which was also started by former Google DeepMind researchers and counts Radical as an investor.

Some prominent AI researchers have raised concerns about the impact that RSI might have on human control over the technology. Automating machine learning R&D could lead to an “intelligence explosion,” accelerating the benefits of the technology but posing “extreme risks,” a headline-grabbing paper warned last month. The article’s co-authors included Clune, as well as Canadian AI pioneers Geoffrey Hinton and Yoshua Bengio, plus top executives from Anthropic, OpenAI and Microsoft.

“If an AI is going to be designing the next version of AI—because humans aren’t able to do it [or] it’s too complicated—we better be sure that that AI doesn’t have some hidden bad goal,” Bengio told The Logic. LawZero, his non-profit AI research lab, recently landed $300 million from the Canadian and German governments to develop its own system called Scientist AI that could eventually act as a check on other advanced AI models.

Night science should remain the job of human AI researchers, according to Larochelle. “This is where we decide where we spend our resources,” he said, adding that “it’s much more legitimate for a person to be taking that decision.”

Still, human AI researchers are developing a technology that could automate the work of other humans, so they must be willing to automate parts of their own, according to Larochelle. “We should be the canary in the coal mines here,” he said. “We have to go there first, learn some lessons, and share it with other parts of society.”

Several of the firms focused on recursive self-improvement see machine learning as the first of many scientific domains in which they hope to automate innovation. Ricursive is initially applying the technology to chip design, while Discovery Loop has signalled it could work on drug development, energy economics or cybersecurity.

Doshi says improving Gemini’s coding ability is also just a starting point. “What RSI is about is leveraging AI to improve AI to improve a field,” she said, citing the physical sciences and professional sectors like legal or finance. “There are many different domains in which this can actually play out.”