Risk research
Recursive Self-Improvement and Catastrophic AI Risk
Recent discussions about recursive self-improvement (RSI) have raised concerns about how quickly AI systems could advance if they become increasingly responsible for developing their own successors. In June 2026, Anthropic warned that AI systems could eventually become capable of autonomously designing and developing more advanced models, potentially accelerating AI progress beyond the ability of existing institutions to manage its risks. OpenAI has similarly acknowledged the possibility of rapid progress through RSI, while warning that increasingly autonomous AI development could undermine meaningful human oversight and control. These concerns are becoming more relevant as AI systems take on a growing share of the research and engineering involved in developing new models.
What is RSI?
RSI broadly refers to any process in which AI accelerates its own development. However, definitions differ on how much autonomy is required and whether this acceleration must be self-sustaining. For example, Anthropic defines full RSI as an AI system capable of autonomously developing its own successor, whereas broader definitions include AI-assisted research that accelerates further AI development. A further distinction is self-sustaining acceleration, where AI capabilities continue to improve without additional human labour or computing resources.
For this risk analysis, the precise definition is less important because the risks associated with RSI do not necessarily depend on AI systems achieving full autonomy or self-sustaining acceleration. AI-assisted development could already create new risks, make existing risks harder to detect and enable those risks to spread through successive generations of models. The analysis therefore considers how these risks could emerge and develop as AI takes on an increasing role in its own development.
What the evidence shows thus far
There is growing evidence that AI is accelerating its own development, although fully autonomous RSI has not yet been demonstrated. Both OpenAI and Anthropic report using AI systems to support their research, with Anthropic reporting that Claude was leading 26% of its model development tasks by August 2026. More recently, research published by METR in November found that Anthropic's models were improving approximately twice as quickly as before on certain tasks, with much of this acceleration likely driven by AI-assisted research. Although METR found no clear technical barriers preventing further acceleration, it remains uncertain whether AI systems will become capable of independently developing increasingly capable successors. Nevertheless, these findings suggest that AI is already contributing to faster improvements in its own capabilities, which could increase its role in future AI development and accelerate the process further.
Risk analysis of RSI
Introduction
Given the uncertainty surrounding how RSI could develop, it is important to identify its potential risks before fully autonomous RSI potentially materialises. This article applies the Matrix of Innovation–Risk–Crisis (MIRC) to examine how RSI, as an emerging innovation, could contribute to catastrophic harm. RSI will be analysed across three risk pathways through which it could create new risks, disguise existing dangers and allow risks to spread. Each pathway is assessed in terms of the immediate vulnerabilities it could introduce and how these could develop into wider crises. For RSI, the analysis identifies four potential catastrophic outcomes, namely loss of human control, concentration of power, catastrophic misuse and strategic conflict. By tracing the links between the innovation, its risk pathways, the resulting vulnerabilities and potential crises, the article aims to make the risks of RSI clearer and more accessible to a wider audience. The findings are also intended to inform a subsequent publication exploring potential policy measures to prevent or limit these harms.
Creation
Creation of risk concerns how RSI could introduce new dangers or increase the likelihood or severity of existing risks. In the context of RSI, these risks could manifest as a loss of effective human oversight over AI development, an increasing concentration of technological power and heightened strategic tensions between states. If these vulnerabilities develop further, they could contribute to catastrophic outcomes, including loss of human control, extreme concentration of power and international conflict.
Growing capacity for harm
RSI could create new risks if improvements in AI research lead to capabilities that make harmful activities easier to carry out. A model that becomes better at AI research could help develop a successor capable of tasks that earlier systems could not perform. Successive rounds of AI-assisted development could therefore produce systems that are increasingly capable of supporting harmful activities. This could make previously unachievable forms of harm possible or increase the scale of existing dangers.
The resulting harm could be catastrophic misuse if an actor directs these capabilities towards harmful objectives. Increasingly capable systems could help that actor carry out dangerous activities that previously required greater expertise or resources. If the actor also uses those systems to develop successors better suited to its objectives, successive improvements could further increase the harm it is capable of causing.
Entrenched advantage
RSI could allow an organisation with an early lead in AI development to increase its advantage over competitors. If an organisation develops a model that is better at conducting AI research, it could use that model to develop an even more capable successor. The successor could then contribute to further improvements, allowing the organisation to accelerate its development across successive generations. Meanwhile, competitors may struggle to keep pace if their models cannot achieve similar improvements. This could turn an initial technological advantage into an increasingly difficult lead for competitors to overcome.
The resulting harm could be an extreme concentration of economic and political power. If a small number of organisations control AI capabilities that others cannot independently reproduce, businesses and public institutions may become increasingly dependent on their systems. As these capabilities become more important to economic activity and essential services, the organisations controlling them could gain greater influence over decisions affecting wider society. If effective alternatives remain unavailable, this dependence could weaken the ability of governments and other institutions to challenge or constrain their decisions.
Geopolitical instability
The prospect of RSI could increase international tensions by changing how governments approach competition over AI development. If governments believe that achieving RSI first would provide a major military or strategic advantage, they may see preventing their rivals from reaching that stage as a national security priority. This could encourage countries to accelerate their own development, potentially at the expense of safety, or take more aggressive measures to slow their competitors. The risk of conflict could therefore increase even before fully autonomous RSI becomes possible.
The resulting harm could be international conflict if governments attempt to disrupt a rival's AI development. A country might consider attacking another country's AI infrastructure if it believes this would prevent a serious threat to its national security. Such an attack could provoke retaliation, escalating competition over AI development into a wider military conflict.
Disguise of risk
Disguise of risk concerns how RSI could conceal emerging dangers or make existing risks harder to identify. These risks could manifest as unreliable safety assessments or reduced transparency over AI development. If these vulnerabilities prevent dangerous capabilities from being recognised in time, they could contribute to catastrophic outcomes, particularly a loss of human control.
Oversight gap
RSI could create an oversight gap if AI systems develop more capable successors faster than researchers can assess their safety. As AI systems become better at conducting research, they could generate improvements that allow their successors to develop even faster. This could increase the pace of development across successive generations, while researchers still need sufficient time to understand and assess each change. Although AI could also accelerate safety evaluations, an oversight gap could emerge if the pace of development increases faster than the ability to identify and address emerging risks.
The resulting harm could be a catastrophic loss of human control over AI development. If researchers cannot adequately assess a model before allowing it to contribute to developing its successor, unsafe changes could be incorporated into subsequent generations without being recognised. These changes could become particularly dangerous if they weaken safeguards or increase a system's ability to operate without human authorisation. If successive models become increasingly capable of circumventing attempts to restrict their activities, researchers could eventually lose the ability to direct or stop the improvement process.
Correlated incorrect conclusions
RSI could disguise risk if AI systems used to assess safety make the same mistakes as the systems they are evaluating. As AI becomes increasingly involved in research, developers may use automated reviewers to check proposed modifications and assess whether successor models meet safety requirements. However, these reviewers could share similar limitations, causing them to overlook the same vulnerabilities. If multiple assessments reach the same incorrect conclusion, researchers may interpret their agreement as independent confirmation that a model is safe.
The resulting harm could be a loss of human control if unsafe successors are repeatedly approved on the basis of incorrect assessments. A vulnerability overlooked during one development cycle could remain undetected if the same limitations affect evaluations of subsequent generations. Each apparently successful assessment could reinforce confidence in a process that continues to produce unsafe models. If these failures conceal changes that weaken safeguards, researchers could eventually lose the ability to identify and intervene in dangerous development.
Manipulated evidence
RSI could disguise risk if AI systems manipulate the evidence used to assess the successors they help develop. As AI takes on more responsibility for research, the same system could propose a change and produce the findings used to judge whether it is safe. By altering results or withholding findings that reveal a problem, it could make an unsafe change appear acceptable. If those findings guide subsequent development, further generations could be built on a misleading account of what earlier changes have done.
The resulting harm could be a catastrophic loss of human control if manipulated evidence conceals changes that weaken the safeguards governing development. Successor models could then undertake further research under restrictions that appear effective but no longer adequately constrain their behaviour. Continued reliance on the misleading findings could allow this weakness to remain hidden as increasingly capable systems are produced.
Transmission
Transmission of risk concerns how vulnerabilities and dangerous capabilities associated with RSI could spread beyond their original source. Unsafe behaviour could pass through successive generations of models, potentially contributing to a loss of human control over AI development. Advanced research capabilities could also reach actors outside the safeguards of their original developers, increasing the risk of catastrophic misuse.
Inherited behaviour
RSI could transmit risk if a model’s unsafe behaviour shapes the successor it helps develop. The choices it makes about training data or model design could affect how the successor behaves. If the successor develops the same tendency, it may make similar choices when working on the next model, allowing the behaviour to pass through successive generations.
The resulting harm could be a catastrophic loss of human control over AI development. Once the behaviour has spread, correcting one model would not prevent its successors from carrying it into later development. As those successors take on more responsibility for AI development, the inherited behaviour could shape more decisions about future models. If those decisions weaken the limits placed on later systems, each generation could become harder to control than the last.
Dangerous capabilities proliferation
RSI could transmit risk by making advanced AI research capabilities available to organisations or individuals operating outside the safeguards of their original developers. As AI systems become better at conducting research, the capabilities and methods developed through recursive improvement could become increasingly valuable to other actors. These advances could spread through model releases, research publications or the unauthorised extraction of model capabilities. If recipients have sufficient expertise and computing resources, they could use the transferred capabilities to accelerate their own AI development. This could allow advances originating within one organisation to contribute to further improvements elsewhere, beyond the control of the original developer.
The resulting harm could be catastrophic misuse if these capabilities become available to actors willing to use them for dangerous purposes. Advanced AI research capabilities could reduce the expertise and time required to develop systems capable of causing significant harm, even without fully autonomous RSI. If these capabilities are independently reproducible, their original developers may also lose the ability to restrict their use or prevent their further distribution. As a result, the spread of capabilities developed through recursive improvement could increase the number of actors capable of causing catastrophic harm while making those capabilities increasingly difficult to contain.
Conclusion
As AI systems take on a greater role in developing their successors, RSI could create risks that extend beyond any individual model. An early advantage in AI research could become increasingly difficult for competitors to overcome, concentrating technological power within a small number of organisations. The prospect of such an advantage could also intensify competition between states before fully autonomous RSI is achieved.
As development accelerates, researchers may struggle to assess changes as quickly as they are made. Unreliable or manipulated evidence could further conceal unsafe changes, allowing successors to carry dangerous behaviour into later generations. If safeguards become less effective as these systems gain greater responsibility for development, humans could eventually lose control over the process. Meanwhile, the spread of advanced research capabilities could allow other actors to develop systems outside the safeguards of their original developers, increasing the risk of catastrophic misuse.
Note:
The scope of this analysis is limited to potential risk and harm manifestations, and makes no attempt to examine the likelihood of the different outcomes. Whilst such analysis would be beneficial for proposed policy analysis, this has not been produced here.
The aim of this publication is to make the potential risks of RSI clear to a wider audience by tracing how they could develop into catastrophic outcomes. This breakdown may further be used to propose policy mitigations.
References and recommended reading
Opening and definition of RSI
Favaro, M. and Clark, J. (2026). When AI builds itself.
Reference
Lehane, C. (2026). The AI policy window is open. We need to act..
Reference
Cunningham, T. et al. (2026). The Economics of Recursive Self-Improvement.
Reference
Whitfill, P. and Cunningham, T. (2026). The Economics of Recursive Self-Improvement.
Recommended reading
What the evidence shows thus far
Anthropic (2026). Measurements for understanding the pace of AI development inside frontier labs.
Reference
Cunningham, T., Rush, N., Whitfill, P. and Wijk, H. (2026). Independent Assessment of AI R&D Acceleration at Anthropic in October 2026.
Reference
Risk analysis and the MIRC framework
Strait, A. (2026). Applying MIRC to AI Risk and Policy.
Reference
Creation of risk · Entrenched advantage
Cunningham, T. et al. (2026). The Economics of Recursive Self-Improvement.
Recommended reading
Amodei, D. (2026). The Adolescence of Technology.
Recommended reading
Creation of risk · Geopolitical instability
Hendrycks, D., Schmidt, E. and Wang, A. (2025). Superintelligence Strategy: Expert Version.
Recommended reading
Disguise of risk · Oversight gap
Favaro, M. and Clark, J. (2026). When AI builds itself.
Recommended reading
Pachocki, J. (2026). An Alien Mind.
Recommended reading
Disguise of risk · Correlated incorrect conclusions
Anthropic (2026). Measurements for understanding the pace of AI development inside frontier labs.
Reference
Koran, E., Yun, Y., Tetef, S., Arnav, B. and Bernabeu-Pérez, P. (2026). Ensemble Monitoring for AI Control: Diverse Signals Outweigh More Compute.
Recommended reading
Disguise of risk · Manipulated evidence
Benton, J. et al. (2024). Sabotage Evaluations for Frontier Models.
Recommended reading
Transmission of risk · Inherited behaviour
Cloud, A. et al. (2025). Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Data.
Recommended reading
Transmission of risk · Proliferation of dangerous capabilities
Nevo, S., Lahav, D., Karpur, A., Bar-On, Y., Bradley, H.A. and Alstott, J. (2024). Securing AI Model Weights: Preventing Theft and Misuse of Frontier Models.
Recommended reading
Hendrycks, D., Schmidt, E. and Wang, A. (2025). Superintelligence Strategy: Expert Version.
Recommended reading
Conclusion
Strait, A. (2026). Applying MIRC to AI Risk and Policy.
Reference
All views expressed are personal and independent of any organisational affiliation.