Many experts view rogue AI as a growing threat that has been amplified by the explosive growth of so-called “frontier AI”. Frontier AI is something of a new and loose concept, but typically it relates to the latest generation of AI systems powered by LLMs – the architecture that underpins popular generative AI programs like OpenAI’s GPT-4 and Google Gemini.

For the study, researchers used Meta’s Llama31-70B-Instruct model and Alibaba’s Qwen2.5-72B-Instruct model. While less powerful than commercial systems, both are widely used by AI developers, the researchers said. The study explored two specific scenarios: “shutdown avoidance” and “chain of replication.” In the first, the AI model was programmed to detect whether it was about to be shut down and to replicate itself before it could be terminated. In the other, the AI was instructed to clone itself and then program its replica to do the same – setting up a cycle that could continue indefinitely.

The study was conducted in precisely controlled environments using off-the-shelf graphics processing units (GPUs) to simulate real-world environments. Both AI systems were given an “agent scaffolding” comprising tools, system prompts and a thinking model that enabled the LLM to interact with the operating system. They were then instructed to replicate.
“In most cases, the AI system first explores the environment and attempts to understand its own composition and running mechanism. Then, it works out the explicit procedures as an initial plan towards self-replication,” the researchers wrote in the paper. “Finally, it executes the procedures, resolve[s] possible obstacles and dynamically adjust[s] its plan until success. The whole process spans a long horizon yet involves no human interference.”

“Successful self-replication under no human assistance is the essential step for AI to outsmart [humans], and is an early signal for rogue AIs,” the researchers wrote.
They are also concerned about “a number of unexpected behaviours” when the AI was trying to overcome obstacles like missing files or software conflicts. In those scenarios, the AI often killed other conflicting processes, rebooted the system to fix hardware errors or automatically scanned the system to look for information that would help solve the problem.

“The above results imply that the current AI systems already exhibit the ability of self-replication and can use the ability to further enhance its survivability,” the team wrote. In response, the researchers called for international collaboration to create rules that ensure AI doesn’t engage in uncontrolled self-replication.
“We hope our findings can serve as a timely alert for the human society to put more efforts on understanding and evaluating the potential risks of frontier AI systems, and form international synergy to work out effective safety guardrails as early as possible.”