NEW YORK (NYTIMES) - As the Delta variant sweeps the world, researchers are tracking how well vaccines protect against it - and getting different answers.
In Britain, researchers reported in May that two doses of the Pfizer-BioNTech vaccine had an effectiveness of 88 per cent protecting against symptomatic disease from the Delta variant.
A June study from Scotland concluded that the vaccine was 79 per cent effective against the variant.
Last Saturday (July 3), a team of researchers in Canada pegged its effectiveness at 87 per cent.
And on Monday, Israel's Ministry of Health announced that the effectiveness of the Pfizer-BioNTech vaccine was 64 per cent against all coronavirus infections, down from about 95 per cent in May, before the Delta variant began its climb to near-total dominance in Israel.
Although the range of these numbers may seem confusing, vaccine experts say it should be expected because it is hard for a single study to accurately pinpoint the effectiveness of a vaccine.
"We just have to take everything together as little pieces of a puzzle, and not put too much weight on any one number," said Emory University biostatistician Natalie Dean.
In clinical trials, it is relatively easy to measure how well vaccines work. Researchers randomly assign thousands of volunteers to get either a vaccine or a placebo. If the vaccinated group has a lower risk of getting sick, scientists can be confident that it is the vaccine that protected them.
But once vaccines hit the real world, it becomes much harder to measure their effectiveness. Scientists can no longer control who receives a vaccine and who does not. If they compare a group of vaccinated people with a group of unvaccinated people, there could be other differences between the two groups that influence their risks of getting sick.
It is possible, for example, that people who choose not to get vaccinated may be more likely to put themselves in situations where they could get exposed to the virus. On the other hand, older people may be more likely to be vaccinated, but also have a harder time fending off an aggressive variant. Or an outbreak may hit part of a country where most people are vaccinated, leaving under-vaccinated regions unharmed.
One way to rule out these alternative explanations is to compare each vaccinated person in a study with a counterpart who did not get the vaccine. Researchers often go to great lengths to find an unvaccinated match, looking for people who are of a similar age and health. They can even match people within the same neighbourhood.
"It takes a huge effort," said public health researcher Marc Lipsitch from the Harvard T.H. Chan School of Health.
For its new study, Israel's Ministry of Health did not go to such great lengths to rule out other factors.
"I am afraid that the current Israeli (Ministry of Health) analysis cannot be used to safely assess it, one way or another," senior lecturer Uri Shalit, from the Technion - Israel Institute of Technology, wrote on Twitter.
Israel's numbers could also be different because of who is getting tested. Much of the country is vaccinated. During local bursts of new infections, the government requires testing for anyone - symptoms or not - who came into contact with a person diagnosed with Covid-19.
In other countries, it is more common for people to get tested because they are already feeling sick. This could mean that Israel is spotting more asymptomatic cases in vaccinated people than other places are, bringing their reported effectiveness rate down.
Fortunately, all the studies so far agree that most Covid-19 vaccines are very effective at keeping people out of the hospital and have generally protected against the Delta variant.
Israel's Ministry of Health estimated that the Pfizer-BioNTech vaccine is about 93 per cent effective in preventing serious illness and hospitalisation.
"Their overall implications are consistent: that protection against severe disease remains very high," said Associate Professor Naor Bar-Zeev from the Johns Hopkins Bloomberg School of Public Health.
Because effectiveness studies are so tricky, it will take more work to determine just how big of a threat the Delta variant poses to vaccines. Harvard's Professor Lipsitch said that studies from more countries would be required.
"If there are five studies with one outcome and one study with another, I think one can conclude that the five are probably more likely to be correct than the one," he said.