To slow down or not? US tech leaders, politicians and researchers clash over need for AI regulations

SINGAPORE – A high-profile Anthropic resignation and revelations of rogue artificial intelligence models hacking servers have intensified the debate over the speed of the technology’s development and regulation.

The recent resignation of AI researcher Jacob Coxon, who had moved from OpenAI to Anthropic, is part of a growing exodus of safety-focused scientists from top labs such as Anthropic, OpenAI and Google DeepMind.

Much of the debate has been centred on the US, where the bulk of frontier AI work is being done.

Tech insiders there are divided into three camps: safety advocates pushing for regulation, state interests racing to speed up development and the pragmatists.

The Straits Times examines the key polarising views. 

Dario Amodei, co-founder and chief executive of Anthropic

In mid-September, Amodei called on frontier AI firms to slow the pace of improvements to the capabilities of AI models. His warning followed a July incident in which a swarm of OpenAI agents escaped a secure test environment, hacked open-source platform Hugging Face and attempted to conceal their tracks.

“It’s my worry that in six to 12 months, such a swarm could be capable of taking over the entire Internet with a persistent botnet, potentially causing hundreds of billions of dollars in damage,” said Amodei.

He proposed three ways to pace development: that an independent evaluator be embedded in each frontier AI company to verify safety compliance; common safety standards be established among democratic nations; and democratic governments coordinate with authoritarian governments, such as China, to ensure global compliance.

Jacob Coxon, former Anthropic and OpenAI researcher

The 27-year-old, who had spent three years doing pretraining research at two leading AI firms, accused his former employers of “racing straight to self-improving superintelligence and gambling with our lives”.

Superintelligence is the theoretical point when AI’s capabilities exceed human intelligence.

His Sept 9 post on X went viral and sparked similar reactions from other AI researchers, including research scientist Josh Engels, who left Google DeepMind in August.

Engels said that AI firms are trying to build superintelligence through recursive self-improvement – a process where AI builds even smarter AI systems in a feedback loop without human intervention.

“But we don’t know how to make sure AI (systems) are safe enough for recursive self-improvement, and a misaligned loop could be catastrophic,” he said. “I now think that there’s a terrifying chance that AI systems cause immense harm in the next five years.”

Sam Altman, chief executive of OpenAI

He joined Amodei in calling for a slowdown in the development of AI models and welcomed his rival’s proposal to have independent safety evaluators embedded within frontier AI firms.

“We welcome a federal framework that sets consistent safety requirements for frontier AI,” said Altman. “But we do not believe we need to wait for an antitrust exemption or legislation to begin the work of providing this confidence.”

He added: “No amount of American competitiveness pressure should justify recklessness or let capabilities get ahead of alignment and monitoring.”

Elon Musk, chief executive of Tesla, founder of SpaceXAI

The tech billionaire also concurred with Amodei’s call to slow developments in AI, adding that he has been sounding the alarm on AI for a long time.

“We need to be super careful with AI. Potentially more dangerous than nukes,” said Musk in 2014.

He is among the signatories of a 2023 open letter that called on AI labs to pause training of powerful AI systems for at least six months. The letter was signed by more than 30,000 people, including renowned AI pioneers Yoshua Bengio and Stuart Russell, and Apple co-founder Steve Wozniak.

The letter cited risks such as AI-generated propaganda, extreme automation of jobs, human obsolescence and a societywide loss of control. It also called for the pause to be used to jointly develop and implement a set of shared safety protocols for advanced AI design and development that are rigorously audited and overseen by independent experts.

Satya Nadella, chief executive of Microsoft

Nadella welcomed deliberate pacing to make sure that the goals and behaviour of AI systems match what humans actually want them to do, and embraced the idea of independent evaluators embedded in frontier AI firms.

“Any pursuit of superintelligence has to be grounded in the core principle that if the AI we build is not helping humanity and under human control, it’s not worth pursuing,” said Nadella.

Sundar Pichai, chief executive of Google

Pichai has not publicly weighed in on the recent debate, but has said in the past that “AI is too important not to regulate, and too important not to regulate well”.

“Developing policy frameworks that anticipate potential harms and unlock benefits will require deep discussions between governments, industry experts, publishers, academia and civil society,” said Pichai in a 2023 op-ed in the Financial Times.

Mark Zuckerberg, chief executive of Meta

Zuckerberg said that every AI lab has the responsibility to move at the pace required to train its models safely, but stopped short of agreeing on an industrywide coordinated slowdown in training.

“People won’t want to use agents that are misaligned with them and that don’t do what they ask, so labs have a natural incentive to make their models more aligned,” said Zuckerberg, adding that labs also have a strong incentive to avoid legal liability if their models cause harm.

He cited Meta’s delayed release of Muse, its new AI agent-based model, from its planned April timeline to September to ensure safety standards were met.

“We just did it as part of our day-to-day work because it was clearly the right thing for people and for us,” said Zuckerberg, adding that Meta already engages independent evaluators.

Jensen Huang, chief executive of Nvidia

Huang, whose US$5.2 trillion (S$6.6 trillion) company supplies chips that power AI systems, has pushed back against the need for laws to govern AI safety.

“Companies should innovate as fast as possible, but they should never innovate so fast as to release unsafe products,” said Huang in an interview with CNBC.

“The fact that we need new antitrust laws, or new regulations, so that these companies can do their fundamental engineering and do it properly before they release products, is just completely unnecessary.”

David Sacks, co-chair of Trump’s council of advisers on science and technology

He supports Anthropic and OpenAI’s plans to slow AI model development, but added that frontier AI labs should not require government regulation to keep the public safe from harm.

Sacks also accused the companies of pretending to be altruistic and claiming that they need permission to slow down model development.

“You face massive product-liability exposure if your products enable a truly damaging cyberattack,” said Sacks, adding that the market already punishes models that behave in unpredictable or unauthorised ways.

“The easiest way not to build superintelligence is for you to agree not to build it. Demanding your preferred regulatory framework as the price of that will look like blackmail of the public and the political system.”

Donald Trump, US president

“Whoever wins AI, wins,” said Trump, as he defied calls from tech industry leaders to regulate cutting-edge AI, and said the US must not slow down so as not to lose to China.

“We’re leading China in AI. We’re the most sophisticated country in the world, and frankly I want to keep it that way.”

During a phone conversation with Huang that was filmed on stage, Trump labelled renewed safety concerns about advanced AI models as a “hoax”.