An opinion piece written by Professor Nick Jennings for The Conversation platform on 27 October 2026.

Read the shortened version.

---

Every few days I am told that AI agents could cause human extinction.

After nearly 40 years building such agents, I find myself asking the same question: where is the evidence?

That is not because I think there are no serious risks. There are, and we should address them. My concern is with the jump from identifying potential risks to suggesting that human extinction, or even “only” significant societal harm, is a likely consequence of increasingly capable agents.

What Are People Worried About?

The most dramatic warnings generally fall into three broad categories.

• Loss of Control: Highly capable agents could pursue their user’s goals in unanticipated ways and with a degree of relentlessness that produces undesirable consequences. The agents might become adept at resisting intervention, deceiving operators, acquiring resources or circumventing constraints in pursuit of these objectives.
• Emergent Superintelligence: AI systems could eventually surpass human capabilities across many important dimensions. Such agents could then pursue their own goals and eventually decide that the pesky humans need to be eliminated because they hinder achievement of their over-riding aims.
• Concentration of Power: Powerful autonomous agents could give governments, corporations, or malicious actors unprecedented capabilities. Here the danger is less that AI itself destroys humanity and more that humans use AI to cause catastrophic harm.

These scenarios attract attention because they are dramatic and difficult to rule out completely. More moderate possibilities are unfortunately not as headline grabbing.

But something being difficult to disprove does not make it likely. This is where I think much of the debate is flawed. Possibility is treated as probability. Yet almost anything is possible. The real question is whether there is credible evidence that it is likely.

To get from today’s AI to human extinction requires a long chain of assumptions: agents become vastly more capable, they acquire real-world power and resources, technical safeguards fail, institutions fail and human intervention is ineffective.

Extinction requires every link in that chain. Yet discussions often focus on the final outcome rather than the probability of each step.

Why I Remain Sceptical About Extinction Claims

My scepticism is not born of complacency. It comes from decades of building autonomous systems in particular, and complex engineered systems more generally. That experience does not make me infallible. It does, however, make me cautious about claims that such systems will somehow acquire and exercise global power with ease.

Increasing the intelligence of an AI system does not necessarily increase its power. Of course, greater capability could make it easier for an AI system to acquire resources, exploit vulnerabilities or influence people. But that still leaves an important set of questions about access, incentives, embodiment, infrastructure and human responses.

Agents don’t operate in a vacuum. To have truly global impact they would need access to computing, networks, money, physical systems, and organisations around the world. These are built and maintained by many different people and organisations. This means there are multiple points of control, all of which are under human control – we design the systems, decide where to deploy agents and set the constraints under which they operate. Thus, the transition from greater intelligence to global power should not be taken as given.

History also offers us a useful perspective. Nuclear technology, biotechnology and cyberspace have all created serious risks, including risks arising from deliberate misuse. Yet societies have developed governance mechanisms, safety practices and international arrangements around these technologies. These structures do not eliminate the possibility of catastrophic risk, but they show that technological determinism is not the only possibility. Institutions, incentives, rules and human choices also shape what happens next and we need to consider what we need to achieve responsible AI deployments.

Humans have agency in this endeavour. The future of AI is not predetermined. It will be shaped by decisions about how systems are designed, where they are deployed and what restrictions are placed on them. We can design safer systems, regulate deployment, establish standards for transparency and accountability, and, in high-consequence settings, restrict autonomy and require meaningful human oversight.

None of these measures is guaranteed to work. They are choices, and choices can be badly made. But that is very different from being powerless observers of an inevitable process. The challenge is to make these choices well, and to apply safeguards seriously and systematically as the capabilities and autonomy of AI systems increase.

The Responsibility of Developers

Highly capable AI places significant responsibility on developers, researchers and technology companies.

Those building autonomous agents need to understand both the systems they are creating and the environments in which those systems will operate. They also need to consider how capabilities that appear benign in one context might create risks in another.

Safety cannot be something added at the end of a development cycle. It must be part of the process from the beginning.

That means robust monitoring, clear operational boundaries, fail-safe mechanisms, auditability, human oversight and continuous evaluation. These are not novel ideas. Many have been part of good engineering practice for decades.

The challenge is to apply them seriously and systematically as the capabilities and autonomy of AI systems increase.

However, there is one aspect of the current debate that I struggle to reconcile.

Some developers publicly claim that the systems they are working on may have a 10% or greater chance of causing human extinction (often called probability of doom or p(doom) for short). Now, if they genuinely believe there is a reasonable chance that their work could lead to significant harms, let alone the elimination of humanity, a simple ethical question follows: why are they continuing to build these systems?

In most fields, knowingly developing a technology that you believe has a significant probability of causing global catastrophe requires an extraordinary justification. A civil engineer who believed there was a substantial chance that a bridge would collapse and kill thousands of people would be expected to stop and address the risk. A pharmaceutical researcher who believed a medicine posed a serious risk of mass fatalities would not simply carry on to market because there are some upsides to the treatment.

The same ethical question arises here.

If an AI developer has a non-negligible p(doom), what level of precaution does that belief require? What safeguards are sufficient? And why is continuing the work justified despite the stated risk?

I rarely hear those questions answered with the same precision that the probabilities of doom themselves are discussed.

If humanity is at stake, surely “continuing anyway” is simply wrong. The people who believe the risk is substantial need to explain what their beliefs imply for their actions.

A More Productive Discussion

Agentic AI will create valuable opportunities and improve the lives of many people. But it will also create serious challenges. This is why we need better governance, more rigorous safety research, greater transparency and clear accountability for those developing and deploying these systems.

However, we should resist the temptation to frame every advance in AI through the lens of human extinction. Fear can be useful when it sharpens our thinking. It becomes counterproductive when it takes the place of rigorous analysis.

My view is straightforward. The most important risks will depend not only on what the machines can do, but on the choices humans make about how these technologies are designed, deployed and governed.

Agentic AI will not determine its own future. Humanity will.

And that makes the quality of our judgement at least as important as the intelligence of the machines we build.