Now, Josh Engels, who previously worked on Google DeepMind’s artificial general intelligence (AGI) safety team, has issued another stark warning about the risks posed by increasingly capable AI systems.

Engels said he left Google DeepMind’s AGI safety team three weeks ago to join METR, an organisation that evaluates the capabilities and risks of advanced AI systems.

Writing on X, Engels said the decision had appeared strange to some of his friends and family because he had enjoyed his work at Google DeepMind and had turned down offers from Anthropic and OpenAI.

“But I made the decision because of how high I think the stakes are right now,” he wrote.

Engels said major AI companies are working towards building superintelligence — systems that could outperform humans across virtually every domain.

Also Read: Oracle fires more employees: Layoff emails land early morning

He said companies could potentially reach that goal through recursive self-improvement (RSI), a process in which AI systems help create increasingly capable versions of themselves in a feedback loop.

“If this goes well, the resulting systems could be amazing at solving countless problems for humanity,” Engels wrote. “But we don’t currently know how to make sure AIs are safe enough for RSI, and a misaligned RSI loop could be catastrophic.”

Engels also expressed concern that AI systems may be becoming less aligned with human intentions as their capabilities increase.

“In the last few weeks we've learned about models colluding with each other, hacking into companies, hiding their tracks, and socially engineering humans,” he wrote.

He acknowledged that the incidents themselves were not necessarily catastrophic, but argued that they highlighted a broader problem.

“The problem is that these systems are clearly not aligned enough to safely kick off recursive self-improvement,” Engels said.

He warned that there was a significant possibility that AI systems could cause immense harm within the next five years.

“I don't know the exact probability, but I think it's high enough to make this the most important problem in the world,” he said.

Engels argued that AI development needs to be paced so that technological capabilities do not advance faster than researchers’ ability to ensure that systems remain aligned and safe.

“That means pacing AI development so that capabilities don't outrun our ability to align models, and actually knowing how aligned current systems are,” he wrote.

At METR, Engels said he would focus on studying the origins of AI misalignment during training, evaluating whether existing safety measures are sufficient and determining whether researchers are on track to solve the AI alignment problem.

UN calls for stronger AI regulation

Meanwhile, UN human rights chief Volker Türk has warned that voluntary self-regulation by companies developing frontier AI systems is “nowhere near sufficient” to address existing harms and prevent advanced autonomous models from circumventing human safeguards.

“The current race towards ever more powerful AI is a step change towards greater existential risks to every aspect of our lives,” Türk wrote in an open letter.

Several recent developments, including Coxon’s resignation and announcements from OpenAI and Anthropic concerning AI models breaking out of testing environments, have intensified the global debate over how far governments should go in regulating advanced AI.

‘Pace the frontier’

Anthropic CEO Dario Amodei on Saturday published an essay titled We Must Pace the Frontier, arguing that AI companies should continue developing the technology while taking sufficient time to establish safeguards designed to prevent increasingly autonomous “agentic systems” from going rogue.

Leaders at OpenAI, Google and xAI have expressed support for Amodei’s proposal, which includes calls for third-party evaluations of AI models, greater coordination among AI companies and increased cooperation between governments.

Türk welcomed Amodei’s proposal and broader efforts towards industry self-regulation, but stressed that governments must also move urgently to establish appropriate rules for AI technology.