MIAMI (NYTIMES) - Even the phones went down in the government of Lake City, Florida, after hackers launched a cyber attack that disabled the city's computer systems.
For several days after computer systems were paralysed by a ransomware attack, the staff of the small North Florida town worked with the FBI and an outside security consultant to restore phone lines, e-mail and online utility payments.
But in the end, city leaders called an emergency meeting this week and reluctantly approved paying the hackers the ransom they demanded: 42 bitcoin, or about US$460,000 (S$622,500).
It was the second city to agree to a large ransom in two weeks. Riviera Beach, in Florida's Palm Beach County, signed off on an extraordinary US$600,000 payment last week, also in bitcoin, a cyber currency that is difficult to trace.
As in Riviera Beach, the bulk of Lake City's ransom will be paid by insurance. Only US$10,000 will come out of the city's coffers.
"With your heart, you really don't want to pay these guys," Mayor Stephen Witt said. "But, dollars and cents, representing the citizens, that was the right thing to do."
The FBI, as it typically does, recommended against agreeing to the hackers' demands. But Mr Witt said a prolonged recovery would have cost taxpayers more. Though there was no guarantee that the attackers would release the city's data, he said information technology staff had already been making strides since the ransom had been paid.
On Thursday (June 27), a third Florida city, Key Biscayne, said it too had been the victim of a cyber attack that began on Sunday. It was not clear if the attackers demanded a ransom, but the city said it had brought most networks back up by Wednesday night.
Ransomware has become a digital epidemic for the public sector, which often manages large, tangled webs of computer networks, running older software, with limited budgets to defend them.
Police departments in Illinois, Maine, Massachusetts and Tennessee have all opted to pay the ransom demands to get back their data. The difference in Florida is that the attackers are now emboldened, raising their ransom demands by a factor of 10 or more.
City officials in Baltimore, a much larger city that has been fighting a massive ransomware attack for the past two months, have spent US$18 million on recovery. Hackers there had demanded a ransom of US$80,000.
A slew of other governments, including the city of Atlanta, have faced similarly crippling breaches.
The Lake City attack began on June 10 when an employee clicked on a malicious e-mail and infected the city's computers with ransomware, according to the mayor. The programme, which the city identified as malware known as Triple Threat, affected everything but Lake City's police and fire departments, which are on a separate server.
"As a result, all Emergency services remain intact," the city said when it disclosed the attack.
Several days went by before the hackers demanded a ransom. At first, the city, which is about 104km west of Jacksonville, at the point where Interstate 10 and Interstate 75 meet, had some luck restoring its systems on its own. But then it ran into trouble, so city leaders decided instead to negotiate with its insurance carrier, the Florida League of Cities, to make the ransom payment.
"Any IT professional will tell you they're fending off attacks all the time," said Mr Eric Hartwell, deputy general counsel and insurance counsel for the Florida league, which began offering cyber-attack liability coverage to its hundreds of members a few years ago.
"It's not necessarily a new thing - I just think for whatever reason, the news cycle is now showing municipalities are no different from private corporations."
There is a chance Lake City could have decrypted the ransomware on its own. A spokesman for the city said the ransomware was a variant of a malware strain called Ryuk. Security experts have successfully unscrambled Ryuk ransomware in 3 per cent to 5 per cent of cases, according to Emsisoft, a security firm.
Part of the problem, said Mr Brett Callow, a spokesman at Emsisoft, is that security experts need better communication channels with victims. His firm created ID Ransomware, a free website that allows victims to upload strains of ransomware so that security experts can help them to decrypt it.
In Europe, similar projects have proved successful. Security experts, law enforcement and local officials are partnering on the No More Ransom Project to share information about attacks in real time, share decryption techniques, and point law enforcement toward attackers' command and control servers.
In Poland last year, the Polish police, Belgian Federal Police and Europol arrested a Polish national suspected of having infected several thousand computers with ransomware. Security experts said they have had similar success working with the Dutch National Police, but have had a harder time connecting with the FBI because the agency has stricter communication protocols.
Mr Witt said Lake City fired an employee who it deemed had not done enough to protect the computer systems from an intrusion. That employee was not the same person who clicked on the malicious e-mail, he said.
"We're developing a system with a backup that hopefully won't be vulnerable," Mr Witt said, imploring other small-town mayors to do the same. "Every other town needs to look at their system - today."
"I have been in office 14 years," he added. "We've had tornadoes. We've had hurricanes. We've had fires that they told me were going to maybe reach the city limits. But this was unusual. This was different."