By: INUWA WAYA.

From Hinton’s and Anthropic’s warnings, the emerging AI race demands a new international system of cooperation, restraint and verification.
There are moments in history when humanity develops a technology before it fully understands the consequences of possessing it.

Nuclear weapons were one. Artificial intelligence may prove to be another.
But there is a fundamental difference. Nuclear weapons possess extraordinary destructive power, but they do not think. They cannot reason about the intentions of their controllers, conceal their actions, rewrite computer code, independently search for vulnerabilities or devise strategies for preventing themselves from being disabled.

Future artificial intelligence may eventually possess some combination of these capabilities. In fact, the information coming out is incredibly alarming. There are strong indications that AI agents are hacking computers and covering their tracks. That possibility has moved the debate about AI beyond questions of employment, productivity, misinformation and economic disruption. A much more disturbing question has entered mainstream scientific discussion. Could the human race create an uncontrollable intelligence more capable and more powerful than itself?

What makes the current debate particularly significant relates to the positions taken by scientists who played a leading role in the emergence of AI. Anthropic, developer of Claude, is one of the world’s leading frontier-AI laboratories and has placed unusual emphasis on AI safety and alignment. Yet researchers associated with the company have recently expressed extraordinarily serious concerns about the direction in which frontier AI is moving.

Jacob Coxon resigned from Anthropic after working on AI pre-training research there and previously at OpenAI. He warned that leading laboratories were racing towards increasingly autonomous and potentially self-improving artificial intelligence before humanity had solved the problem of controlling systems more capable than itself.

More strikingly, Evan Hubinger, Anthropic’s Alignment Science Lead, has placed his personal estimate of the probability of AI causing human extinction within roughly the next decade at greater than 10 percent.

Samuel Marks, Anthropic’s Scalable Oversight Lead, has also publicly emphasised the seriousness with which catastrophic AI risks are regarded among people working closest to frontier systems. None of this means Anthropic believes extinction is inevitable. Nor does it mean a 10 per cent figure can be scientifically measured with precision. It means something more limited—but still extraordinary.

People whose professional work involves understanding and controlling frontier AI systems are telling the world that they cannot confidently rule out catastrophic loss of control.

That, of course, deserves serious attention. Geoffrey Hinton is not a speculator about a technology he does not understand. His pioneering work on neural networks helped establish foundations upon which modern artificial intelligence was built. Today, he is widely regarded as one of the fathers of artificial intelligence. Together with Yoshua Bengio and Yann LeCun, he received the Turing Award for foundational contributions to deep learning. He subsequently received the Nobel Prize in Physics.

In 2023, Hinton left Google after more than a decade with the company. He explained that leaving allowed him to speak more freely about the dangers he increasingly believed advanced AI could create. There is no doubt that his decision to quit and the reason behind it would generate debate about the future of AI. One of the architects, if not the leading architect, of the AI revolution had become sufficiently concerned about where the revolution might lead that he wanted greater freedom to warn society about it. Responding to a question on whether a greater-than-10-percent probability that AI could kill humanity within approximately a decade was plausible, Hinton opined that such an estimate was not unreasonable. Even though he made a caveat regarding the knowledge to calculate such probabilities, that caveat is essential. In spite of that, however, Hinton’s remark is an indication of how urgently some leading researchers believe the governance issues behind advanced AI must be addressed.

It is interesting to note, however, that the scientific community is far from unanimous on the subject of human extinction now or at some time in the future because of advanced AI. Yann LeCun, another giant of modern AI and Hinton’s fellow Turing Award recipient, strongly disputes many extinction scenarios.

One of LeCun’s important arguments is that intelligence should not be confused with motivation. Becoming extraordinarily intelligent does not logically mean acquiring a desire to dominate humanity. Human drives towards power, status and survival have biological and evolutionary origins; and there is no law requiring an artificial intelligence to acquire equivalent ambitions simply because it becomes highly capable. For LeCun, the fear of AI taking over the world is simply humans projecting their own evolutionary traits of power and dominance onto machines. He concluded that if an AI system ever becomes unsafe, humans can simply change its design or turn it off. At all material times, human beings should be in charge.

Andrew Ng has similarly argued that extinction scenarios are excessively speculative and that disproportionate fear could encourage regulation that unnecessarily inhibits beneficial technological development. Such regulations, he said, were championed by large tech companies in order to shut out smaller competitors and hurt the open-source community.

Other researchers emphasise the limitations of today’s AI. Current systems hallucinate, make elementary errors, struggle with aspects of sustained reasoning and remain dependent upon enormous physical infrastructures built, powered and maintained by humans.
These arguments deserve serious consideration. Indeed, scientists who disagree with Hinton should be included in any international effort to govern frontier AI. But their disagreement does not establish that catastrophic AI risk is impossible. It only establishes something more uncomfortable that could best be described as uncertainty.

When Hinton and LeCun, who shared the world’s highest distinction in computer science for their contributions to deep learning can examine the trajectory of the same technology and reach radically different conclusions about its ultimate danger, policymakers cannot responsibly pretend that the scientific question has been settled.

Suppose Hinton and the Anthropic researchers are wrong. Humanity might impose unnecessary safety tests, spend enormous resources on verification and perhaps slow some frontier development.
Conversely, suppose they are right to a large extent. The world may not have the opportunity to correct any identified error. That asymmetry presents a perplexing challenge which will be at the heart of the policy problem.

If aeronautical engineers disagreed over whether an aircraft had a 10 percent probability of crashing, nobody would propose to resolve the conflict with the aircraft at full passenger capacity. An informed reasoning would advise to investigate and resolve the matter before take-off. In respect of the AI conundrum, the rational response should neither be panic nor technological prohibition. It should be precaution proportionate to consequence. The big question is to determine the international body that would be saddled with the responsibility of ensuring the adoption of precautionary measures and whether or not there would be unanimity amongst the scientists on what should be regarded as consequences. The competition between countries to develop artificial intelligence is intensely fierce, often likened to an arms race or a new Cold War. World leaders view AI as the defining technology that will dictate future economic leadership, national security, and global influence. With differences in strategy, China and the United States of America are the leading competitors in the global arena. The U.S. leads in capital, talent, and frontier model innovation, whereas China focuses heavily on scaled deployment, patents, and cost-optimized models. Europe, India, Singapore and some Middle Eastern countries, are gradually developing capabilities. The current AI competition is not unprecedented in the history of the world.

The closest historical analogy is the nuclear race. After Hiroshima and Nagasaki came the Soviet atomic bomb, the hydrogen bomb, intercontinental ballistic missiles and submarine-launched nuclear weapons. The United States and the Soviet Union became locked into a security dilemma. Each increased its arsenal partly because it feared the other’s arsenal. Then came the Cuban Missile Crisis, which demonstrated that unrestricted strategic competition could produce an outcome in which neither side survived sufficiently to enjoy victory.
Out of that recognition, gradually emerged arms control. The Strategic Arms Limitation Talks—SALT—began formally in 1969. SALT I produced the 1972 Anti-Ballistic Missile Treaty and an interim agreement limiting aspects of strategic offensive weapons. SALT did not end the Cold War and did not eliminate nuclear weapons.

It did something more realistic by providing boundaries around competition. Subsequently came the Strategic Arms Reduction Treaty (START). START went beyond limiting strategic arsenals towards requiring substantial reductions. It established extensive verification arrangements involving data exchanges, notifications, technical monitoring and on-site inspections. America did not trust the Soviet Union and vice versa. Where trust was impossible, verification could make restraint possible. As practical as they were, the SALT/START analogy may not entirely suit the AI situation.

The most consequential strategic competition in frontier AI is presently between the United States and China. Their companies, computing infrastructure, semiconductor strategies and military ambitions make their relationship central to the global AI race.
If Washington believes Beijing is approaching transformative artificial intelligence, America has an incentive to accelerate. If Beijing believes Washington is approaching it, China has the same incentive. Neither wants to finish second and there is no practical intention to slow down progress. That makes negotiations between the US and China, very essential. At the moment, the likelihood of such happening is very remote. Intense competition is fracturing the global tech landscape into two increasingly incompatible ecosystems centered around the US and China. In rejecting the warning given by the AI giants on the possibility of AI taking over the World and wiping out humanity, President Trump said such assertions were a hoax, peddled by radical Democrats. For him, AI and data centers in the US, are the “Greatest Economic Development Engine in History which should not be stopped”.

Let’s assume for a moment that China and the US agreed to negotiate limitations. That is not enough. There are others that must be carried on board. Important AI research and infrastructure exist in Europe, Britain, India, Japan, South Korea, Canada, the Middle East, Singapore and elsewhere. Future breakthroughs therefore need not necessarily originate in either America or China. There is an even greater complication to consider. Unlike nuclear weapons, much of the frontier of artificial intelligence is not controlled directly by governments. Corporations and research laboratories are controlling them. Companies such as OpenAI, Anthropic, Google, DeepMind and Meta are developing frontier systems in the United States and allied economies. China has its own major technology companies and AI laboratories. That alone fundamentally changes the governance structure. A treaty constraining Washington while leaving frontier American laboratories outside its obligations would be inadequate. Ditto for Beijing and Chinese developers. In the same vein, an agreement between America and China that ignored significant capabilities elsewhere could become ineffective. The practical solution should have different segments. The bilateral approach should be recognised as Stage One.

Washington and Beijing should begin negotiations for an AI Strategic Stability Agreement because their rivalry currently creates the greatest danger of a frontier race in which safety becomes subordinate to speed. Such negotiations must involve independent AI companies in their respective countries.

The first agreement could prohibit autonomous AI authority over nuclear launches; guarantee meaningful human control over irreversible strategic military decisions; establish emergency AI hotlines; create common reporting obligations for serious loss-of-control incidents; and establish agreed safety evaluations for systems crossing defined capability thresholds. The next stage should widen participation.

Other states possessing significant frontier AI capability, advanced semiconductor industries or very large computing infrastructure should enter the framework. Ultimately, the objective should be the adoption of a universally recognised and acceptable control regime. This assertion may sound simple and inviting in print words. In reality, and judging from the difficulties the international community is experiencing in observing and enforcing international law and order, it will certainly be a herculean task to reach a consensus on establishing a body that would be saddled with responsibility of superintending the frontier artificial intelligence systems, possessing capabilities of creating catastrophic international consequences, and eventually introduce an international verification system. The purpose of international governance and monitoring is not to determine which government or company has good intentions. It is to create rules under which responsible behaviour does not require competitive suicide.

No one knows whether AI could cause human extinction within a decade. We should resist converting uncertain probability estimates into sensational predictions. We should equally resist converting uncertainty into complacency. The next ten years should therefore not be described as humanity’s countdown. They should be treated as humanity’s window for governance as far as AI is concerned.
Artificial intelligence could become one of humanity’s greatest achievements. It could transform medicine, education, scientific discovery and productivity. It could solve problems generations of human beings have failed to solve. The objective cannot therefore be to stop technological progress. It must be to ensure that progress remains compatible with human survival. The international community must therefore put aside their differences along fault and open lines for the sake of humanity. They must note that SALT and START made it possible for adversaries to unite and limit the arms race to guarantee the future of mankind. No government, corporation or scientist has the moral right to gamble with the continued existence of humanity simply because nobody can calculate the odds precisely. If Hinton and the Anthropic researchers are wrong, history may conclude that humanity was excessively cautious. If they are right and humanity ignores them, there may be nobody left to write history. In any case, even before deciding on the decade extinction arguments, the World needs to act on the gradual degeneration of human brain as a result of over reliance on AI in every human endeavour. Nowadays, humans rely on AI for analytical thinking. They consult it for advise on personal and official matters. This decline in human brain is capable of causing incalculable damage to human psyche. The big question today therefore, is whether humanity is determined to control intelligence greater than its own before they become impossible to govern.