Caller ID destroying official understanding of jobs market

LONDON – Britain’s Office for National Statistics (ONS) used to knock on doors and ask people if they were employed, as a traditional way of compiling the country’s labour market data.

But when Covid-19 forced social distancing in 2020, door-knocking was out. At one point during the pandemic, the survey was reliant on how many phone numbers it could find for a representative sample – and also on people actually answering their mobile phones when an unknown number called. Only a quarter of Britons say they answer such calls, with most wary of scams. Fewer than half of British households have a landline, and those numbers can be difficult to find.

Response rates plunged. By 2023, the data had become so patchy the ONS was forced to suspend its unemployment reading as it desperately sought to repair a key release that helps inform government policy, influences interest rates and drives billions of dollars of investment decisions.

An ensuing crisis of confidence in Britain’s data ended with the resignation of national statistician Ian Diamond in May, following fierce criticism from politicians and senior central bankers.

Getting households to pick up the phone is just one of many obstacles facing government agencies tasked with collecting numbers on everything from joblessness to agricultural output. Just as cellphone users swipe away unwanted callers, people screen visitors with video doorbells and an increasing number live in apartment blocks impenetrable to data collectors, compounding a collapse in survey responses. The pandemic exacerbated most of these trends.

That has left statisticians from Australia to Britain to the US scrambling to update collection methods stuck in the 20th century without introducing new 21st century biases. The solution that agencies and pollsters, from the ONS to the Bureau of Labour Statistics in the US, have zeroed in on: online surveys. 

These polls are less costly than telephone or face-to-face surveys and more convenient for respondents, with statistics agencies using artificial intelligence (AI) to target non-responders. Yet as agencies are finding, the big switch in economic data-collecting is throwing up new problems for the indicators global investors and policymakers rely on.

“There’s no magic mode,” said Mr Steve MacFeely, chief statistician at the Organisation for Economic Cooperation and Development. “They’re well aware of the risk and are doing it because they have no other choice. It’s the grim reality of official statistics today.”

Loss in measurement quality

Problems with the new wave of online surveys include fresh biases in the data, confusion over questions that a computer cannot address, gaining trust amid a barrage of spam and fraud in people’s inboxes, and the fact that households may be just as likely to ignore an e-mail with an online survey as a phone call. 

“What you gain in coverage/response, you possibly lose in measurement quality or content features,” said Mr Eric Harrison, deputy director of the European Social Survey.

Problems with Britain’s labour market figures serve as a cautionary tale. Early results of Britain’s online questionnaire showed the new online survey was producing significantly different readings from the existing survey that relies on telephone calls and knocking on doors, people familiar with the situation said.

One potential issue is that respondents are not able to ask an interviewer to clarify a question they do not understand. The ONS is looking at an AI chatbot that could help. 

“Moving the survey online has been challenging due to the survey length and complexity meaning many households didn’t finish it,” an ONS spokesperson said. “We’ve tested a shorter version that’s giving improved and more complete responses and this new version will go live in the field next month.” 

Another key risk from online surveys is that they create new unknown biases that statisticians have to counteract. In Britain, the existing labour force survey struggled to get enough young workers to answer. For an online questionnaire, it could be the opposite problem. 

“If you were to do a face-to-face survey, and you were to do a telephone or an online labour force survey, you’d get different results from all three,” said Mr MacFeely.

In the US, Ms Joanne Hsu, director of consumer surveys at the University of Michigan, believes that being behind the computer screen may also skew how people respond to economic surveys.

The Michigan survey’s closely watched figures on economic sentiment and inflation expectations in the US fully transitioned to a web-only mode in 2024 after seven years of testing the new responses alongside its existing phone-based method. While it helped to double the survey’s reach to nearly 1,500 responses, Ms Hsu noticed that respondents are more likely to give extreme answers on the web – especially for inflation expectations, which recently surged to the highest in three decades.

“People might think twice about giving any sort of extreme value to an interviewer,” she said.

Former US Bureau of Labour Statistics commissioner William Beach said US data could soon be in the same state as Britain’s moribund labour statistics without action to modernise its surveys.

Labour force surveys “are dying, they’re decaying, they’re in very serious trouble”, he added on Bloomberg’s Odd Lots podcast recently.

“Unless we modernise that survey, we will see a time when we will be like the British, unable to publish portions of it that just don’t have sufficient sample for statistical release.” BLOOMBERG