Software bug in server behind SingPass outages

It resulted in some expired records not being automatically removed, causing slowdown

A software bug in the server of a vendor was behind two SingPass outages last month that disrupted hundreds of essential e-government services, in what was the longest disruption to SingPass since it was set up in 2003.

Senior Minister of State for Communications and Information Janil Puthucheary yesterday told Parliament that investigations found that the software bug in the system of the vendor - Dutch cyber-security firm Gemalto - manifested itself only after an enhancement to the SingPass and CorpPass system in January this year.

CorpPass, used by businesses, was also affected by the outages on Feb 8 and 9.

The enhancement complied with all technical specifications and was properly tested, Dr Janil said.

"However, the interaction between the enhancement and the software bug caused some records to persist in the system instead of being automatically removed 30 days after they expired, which was the root cause of the slowdown," he added.

"While the bug itself was elusive, the symptoms - slowdown in system performance - could have been detected earlier.

"Our early detection and warning capabilities can be improved," he said. "We intend to do so by enhancing the software checks and diagnostics so that in such cases, the engineers can act before the system condition worsens to a state that would affect users."

Dr Janil also noted that while the system had the hardware backup to deal with hardware and infrastructure failure, such redundancy did not address unknown internal software bugs of this nature.

He said the Government will review the system's design "to improve all-around resiliency".

The Government is also reviewing its contracts with commercial providers to ensure they adequately cover service outages, he said.

The two outages lasted about 10 hours in total, disrupting essential e-government services such as the filing of employees' Central Provident Fund (CPF) contributions and work permit applications.

Some Malaysian workers had to return home as their work permits could not be processed. Companies risked fines because they could not file their employees' CPF contributions on time.

Experts have raised concerns that the SingPass authentication systems are not robust enough and could dent the public's trust in the national digital identity system, a key Smart Nation project.

SingPass has more than 3.3 million registered users and supports 57 million e-government transactions, including the filing of income taxes and the payment of parking fines.

CorpPass, rolled out in September 2016, is meant for corporate transactions, including the filing of corporate taxes.

A version of this article appeared in the print edition of The Straits Times on March 20, 2018, with the headline 'Software bug in server behind SingPass outages'. Print Edition | Subscribe