Why it’s hard for humans to have the final say over AI

The riskier the setting in which powerful AI systems are deployed, the more we seem to reach for an intuitive solution: that humans should always be the ones to make the final decisions.

In the context of war, the public and regulatory debate (and one source of the recent row between Anthropic and the US government) has focused on the seemingly binary distinction between fully autonomous weapons and those that are subject to “human control”. In the corporate world, too, the deployment of semi-autonomous agents has led companies to turn to experienced humans as the ultimate decision-makers. Amazon, for instance, has reportedly said that junior and mid-level software engineers require more-senior engineers to sign off on AI-assisted changes.

But does this solution necessarily lead to the best of both worlds, in which machines boost speed, accuracy and productivity, while humans supply expertise, context, judgment and accountability?

The good news is that we’ve been thinking about how best to combine machines and humans for a very long time. As far back as 1951, a psychologist called Paul Fitts devised a list of those things “men are best at” and those that “machines are best at”. The bad news is that we don’t seem to have learnt much from our many mistakes along the way.

The first issue is that AI operates at superhuman speed. On the battlefield, for example, even systems that leave final decisions to humans can churn through mountains of data and vastly increase the number of potential targets to hit. But when so-called “kill chains” are compressed from hours to minutes or even seconds, it calls into question how much real-time control humans can realistically provide.

In the lower-stakes world of the office, we are beginning to see a similar phenomenon, as AI agents vastly speed up the pace and volume of work that still has to be directed and reviewed by humans. One eight-month study into generative AI use at a US tech company found that a surge in productivity came with “cognitive fatigue, burnout, and weakened decision-making”.

The second issue is that many humans are inclined to trust machines even when they are warned not to. The phenomenon of “automation bias” has been documented repeatedly in all sorts of settings over the years, from drivers following their GPS systems into rivers to students following robots away from fire exits in a simulated emergency.

I have written before about an experiment at Volvo Cars in which almost 30 per cent of people allowed a semi-autonomous car to crash straight into an object on the road.

In February, two academics at Wharton business school coined the term “cognitive surrender” to describe a phenomenon in which a person simply “relinquishes cognitive control and adopts the AI’s judgment as their own”. (“Cognitive surrender” is also, incidentally, a good description of what happens to me at about 8pm every evening.)

The third problem is that accountability becomes blurred. Who is to blame when something goes wrong? The temptation will be to blame the human who made the final decision but, if they were operating in a system that was not designed to mitigate the previous two problems, that might not be fair, nor lead to appropriate structural remedies.

Instead, humans might find themselves in what academic Madeleine Clare Elish has called the “moral crumple zone”. “Just as the crumple zone in a car is designed to absorb the force of impact in a crash,” she wrote in a paper in 2019, “the human in a highly complex and automated system may become simply a component – accidentally or intentionally – that bears the brunt of the moral and legal responsibilities when the overall system malfunctions.”

These problems are not insurmountable. Indeed, we have already learnt a lot about how to mitigate them in certain professions. Airline pilots, for example, have been interacting with automation technology for decades. There have been some disasters along the way, but also a lot of lessons learnt, for example, in the importance of training pilots on how various modes of automation work (and how they can fail), as well as encouraging them to hand-fly on occasion to maintain their skills.

There is instinctive appeal to the idea that humans must have the final say over these powerful new technologies. But the history of human-machine interaction tells us that is not as easy to achieve as it sounds. What’s more, the illusion of human control can be more dangerous than its clear absence. FINANCIAL TIMES