Recursive self-improvement, where an AI system can autonomously design and develop a better version of itself, “could come sooner than most institutions are prepared for,” said the company’s in-house think-tank. That could lead to new scientific discoveries and other benefits, but also risks humans losing control over AI. (The Logic)

Talking point: Anthropic co-founder Jack Clark and Institute lead Marina Favaro say AI can’t build better AI on its own just yet, but it’s likely headed that way, albeit under human direction. Over 80 per cent of the code the firm puts into effect is already written by its Claude tool, prompted by developers. The system is also getting good at coming up with and running experiments, which is how scientific fields like AI improve. Several well-funded startups are trying to develop recursive self-improving AI systems. But given the negative consequences that ever-smarter AI could have on jobs and other key human interests, “it would be good for the world to have the option to slow or temporarily pause frontier AI development,” the Anthropic co-authors say; that could give society and safety research time to catch up. They acknowledge any such mechanism would require a lot of international co-operation and be difficult to enforce. AI maximalists reacted poorly to the post.