How hackers got away with SingHealth's crown jewels

An inquiry in the past two weeks into the SingHealth cyber attack has uncovered new details about the data breach and also identified weaknesses and lapses in the public healthcare group and its IT vendor. Based on testimonies of witnesses, Senior Tech Correspondent Irene Tham and Hariz Baharudin piece together an account of how Singapore's biggest cyber attack unfolded.

ST PHOTO: SYAMIL SAPARI

Some time in August last year, somewhere in the Singapore General Hospital, a computer workstation became infected with malware, likely after a user fell prey to a phishing attack.

It is a common trap that ensnares many Internet users. And it did not help that the computer was running an outdated version of Microsoft Outlook, making it defenceless against new viruses. But this time, the incident possibly led to Singapore's worst data breach.

Through the phishing attack, the cyber hackers gained a foothold in public healthcare group SingHealth's vast store that houses the medical and personal data of five million patients.

Right after the entry, the attackers called back overseas to say: "We're in."

Instead of rampaging through the store for patient records - specifically for that of Prime Minister Lee Hsien Loong - the attackers laid low for four months before moving around the network slowly to gather more user accounts to execute their next moves.

The conventional safeguards of Integrated Health Information Systems (IHiS) - an agency that runs the IT systems of public healthcare institutions - were no match for the hackers' advanced techniques.

For instance, malware created for the attack escaped detection by even the world's top anti-virus software makers.

CROWN JEWELS

Details of the attack - as well as an account of what went right and wrong - were revealed when a high-level Committee of Inquiry (COI) into the cyber attack held a series of hearings in the past two weeks, which ended last Friday.

What went right: Some IHiS staff took action to investigate and even to end the attack despite the lack of instruction from their superiors.

What went wrong included how a server exploited by the hackers had not received the necessary security software updates for over a year, and how IHiS lacked a framework spelling out timely responses to cyber-security risks.

On July 20, Singaporeans first learnt of the country's worst data breach, which took place undetected from June 27 to July 4. It saw the attackers stealing the personal data of 1.5 million SingHealth patients, and the medical prescriptions of 160,000 people, including PM Lee.

ST ILLUSTRATION: MANNY FRANCISCO

To steal the data, the attackers had to first obtain user account passwords to access SingHealth's electronic medical records (EMR) system. They targeted inactive administrator accounts - of which one had an easily cracked password: P@ssw0rd.

Between May and June this year, hackers used these accounts to remotely log in to a server that had an open connection to the EMR.

The open link, which had been set up temporarily for database migration to a new cloud-based system, was scheduled to be disconnected last month, according to evidence shared with the COI.

COI chairman Richard Magnus said last Friday: "It would appear to the COI, even at this stage, that the attacker had one and only one malicious intent - that of exfiltrating data from the crown jewels of the network, which is the EMR."

Though the attackers had a direct route to the EMR, they were unable to access it. They made multiple failed attempts to log in, using either non-existent user accounts or those that were not granted access.

FIRST SLEUTH

Their attempts went unnoticed for about three weeks until June 11 when Ms Katherine Tan, a database administrator at IHiS, spotted the unusual network activity.

In a way the first sleuth to arrive at the crime scene, Ms Tan informed her colleagues via e-mail - including a more senior staff member, Mr Lum Yuan Woh, IHiS' assistant director (infra services - systems management) - about the access attempts.

Ms Tan found it odd that administrator accounts with no access rights to the EMR database were being used to enter it. One of these accounts belonged to a colleague, whom she verified had not tried to enter the EMR system.

Over the next two days, she compiled more error logs of attempts to reach the EMR database and became more convinced that someone was repeatedly trying to break into it.

She sent more e-mails alerting colleagues and Mr Lum, thinking that IHiS was dealing with "what could be classified a security incident".

Ms Tan did not think it was necessary to report the incidents to more people, as she thought Mr Lum would know what to do.

But Mr Lum, too, did not report the incident to higher-ups; it did not occur to him that the breached administrator accounts could do any harm.

NO ALARM BELLS

One of the e-mails landed in the inbox of IHiS system engineer Benjamin Lee. He took the initiative to study the suspicious activities forensically and alert two key cyber-security executives at IHiS - Mr Ernest Tan Choon Kiat, senior manager (infra services - security management), and Mr Wee Jia Huo, cluster information security officer.

Mr Lee also set up a chat group using an internal secure chat system on June 13 with some colleagues, including both Mr Tan and Mr Wee, to discuss the unauthorised attempts to access the EMR system. Though these chat groups were rarely formed, both Mr Tan and Mr Wee did not realise the severity of the incidents. Neither did they follow up on the e-mails they were copied on.

Tasked by the Attorney-General to lead evidence in the COI, Solicitor-General Kwek Mean Luck said in his Sept 21 opening statement that IHiS staff "did not fully appreciate that multiple cyber-security incidents, culminating in a breach of the database, were occurring".

The fact that several different username-password combinations had been used in attempts to connect to the database did not ring any "alarm bells" for Mr Tan, he said when giving his account of the incidents to the COI.

Mr Wee did not create a framework spelling out timely responses to cyber-security risks, though he was the one in charge of assessing and reporting risks.

He said he relied on Mr Tan to initiate any alerts on cyber threats and recommend if they should be reported. But Mr Tan said it was not his job to report to higher-ups even if a cyber-security incident had occurred. It was Mr Wee's job.

MANAGEMENT INACTION

The COI also heard about management inaction and misjudgment on the part of IHiS in 2014.

A staff member was found to have reported an alleged flaw in the EMR system, which was supplied by vendor Allscripts Healthcare Solutions. But no action was taken to investigate the supposed loophole.

Mr Zhao Hainan, who was then an IHiS systems analyst, had written an e-mail on Sept 17, 2014, to flag the alleged "loophole" to Allscripts' rival, Epic Systems.

In the e-mail, which Allscripts obtained from Epic and sent to IHiS, Mr Zhao alleged that the supposed coding flaw could allow hackers to "gain admin control of the whole database easily". Even medical students, nurses and pharmacists could have such access, he wrote.

COI members quizzed former IHiS chief executive officer Chong Yoke Sin and other IHiS staff during the Sept 28 hearing on why they did not take action on the supposed "loophole" found.

Dr Chong said she had considered Mr Zhao's action to be "primarily a disciplinary issue, and not an IT security issue". Her impression was that his motive was to seek personal gain from Epic.

Asked why he did not check on the alleged flaw, Mr Clarence Kua, an IHiS employee assigned to SingHealth as deputy director (chief information officer's office), said his focus was to confirm that Mr Zhao had sent the e-mail to Epic.

His stance prompted Mr Magnus to say: "You can focus on two things at the same time."

ATTACKERS SUCCEED

On June 26, the attackers successfully obtained access to the EMR system and began stealing the data the next day.

The stolen records involved 1.5 million patients who had visited SingHealth's specialist outpatient clinics and polyclinics from May 1, 2015, to July 4 this year. Their non-medical personal data that was illegally accessed and copied included names, NRIC numbers, addresses, gender, race and dates of birth.

The Government called it a deliberate, targeted and well-planned cyber attack that was "not the work of casual hackers or criminal gangs".

By July 4, Mr Wee and Mr Tan had not reported the incident to management despite knowing of attempts to access 100,000 EMR records, as they viewed it only as a "potential breach" and not a "confirmed" one.

The data breach was halted on July 4 when Ms Tan terminated the unauthorised EMR database queries - though she had not been told to do so.

Though the data thieves had run away with the "crown jewels", no one in IHiS had a clue, for almost a week. Some dismissed the unusual database queries as a surprise audit.

There was general consensus that the terminated queries were a "security incident", but the Cyber Security Agency (CSA) was not informed.

A staff member told his superior - Mr Henry Arianto, IHiS deputy director of product management and delivery in the clinical care department - that the hacker did not steal any data. And this erroneous message spread. Mr Arianto reported the incorrect finding to senior IHiS staff on July 9.

Any sighs of relief, however, were short-lived.

On July 10, Mr Arianto decided to "double-check" by simulating one of the attempts of the hackers. What he found left him "shocked", said Mr Arianto. His employee was wrong. Data had indeed been stolen.

CRISIS MODE

The crisis mode in IHiS kicked into high gear following this discovery. To determine the extent of the breach, IHiS senior management immediately set up a "war room" in the Connection One building in Bukit Merah.

On the same day, CSA was informed of the attack, as were the Health Ministry and SingHealth.

Database queries from June 27 to July 4 were recreated to determine the extent of the breach.

On July 11, it was discovered that PM Lee's data had been stolen using his NRIC number, along with that of two others, who were non-VIPs. A police report was made the next day.

But the attackers did not give up. Using other footholds in SingHealth's network, they tried to execute commands from yet another server on July 19 - amid investigations of their earlier breach.

IHiS responded by taking remediation measures to deal with these attempts that day.

The COI, which has to submit a report on its findings and recommendations by the year end, privately held its first hearing on Aug 28.

A second tranche of hearings, both public and private, started on Sept 21 and ended last Friday. More hearings will continue at the end of this month.

Mr Kwek said last Friday that the next tranche would highlight the need for organisations to have adequate and updated cyber defences.

"The nature of the attack, in particular the skill and sophistication used in the SingHealth attack, highlights the challenges cyber defenders face," he said. "There is a need for cyber defenders and defences to evolve and keep pace with the changing threat landscape."

Join ST's Telegram channel and get the latest breaking news delivered to you.

A version of this article appeared in the print edition of The Straits Times on October 08, 2018, with the headline How hackers got away with SingHealth's crown jewels. Subscribe