SGX trading halt: Software failed to detect faulty disk

The SGX says the faulty disk behind last Thursday's trading halt has been replaced, and that investigations are ongoing as to why the software did not detect the hardware problems.
The SGX says the faulty disk behind last Thursday's trading halt has been replaced, and that investigations are ongoing as to why the software did not detect the hardware problems.ST PHOTO: SEAH KWANG PENG

Bourse's attempts to reopen market hit by problems in orders and trade confirmation process

The longest trading disruption in the Singapore Exchange's (SGX) history was triggered by disk failure and a software application that did not detect the problem.

Compounding the situation, attempts to reopen the market last Thursday were stymied by problems in the orders and trade confirmation process, the SGX disclosed yesterday.

Trading was halted for over five hours that day after the discovery of a technical fault that resulted in some clearing messages not being generated and some trade confirmation messages being duplicated.

The hardware was supplied by computer giant and IT firm Hewlett- Packard (HP), while the software application was from United States stock exchange Nasdaq.

SGX chief executive Loh Boon Chye told reporters yesterday that the root cause was "a faulty disk that runs the applications that generates trade confirmation messages".

SGX president Muthukrishnan Ramaswami added: "The hardware supporting the clearing registration process had disk errors, and the application software that was supposed to detect the failure, and automatically switch the process to the secondary data centre, didn't kick in and function as it should.

"This resulted in missing trade confirmations during the period it was running on the faulty hardware, and duplicate messages created during the move to the secondary centre."

Mr Ramaswami noted that the software glitches meant that messages sent to update trades contained duplicates and missing confirmations, noting: "As a result, their order books were not synchronised with trades done."

But the recovery took longer than expected because several files on the missing and duplicated clearing confirmation messages had to be generated for each of its securities trading and clearing members.

The SGX added that the faulty disk was replaced last Thursday and health checks have been done.

Mr Ramaswami said investigations are ongoing as to why the application did not detect the hardware problems.

"Nasdaq is also investigating. This is not a known issue that exists anywhere else. This is the first time this has happened," he added.

The SGX declined to comment when asked whether it would seek compensation from HP and Nasdaq.

Nasdaq told The Straits Times last Friday that it was aware of a hardware issue that halted trading, saying: "We have worked closely with SGX to ensure a permanent resolution of this matter."

Mr Ramaswami also disclosed yesterday that the SGX had hoped to reopen the market last Thursday for an hour, to allow participants to close off their positions, but "the reconciliation process had not been done by 4pm. So we made the choice to not (reopen) because members weren't clear what orders had been fulfilled or not".

If a trading disruption were to occur again, the SGX said it would hope to reopen the market for at least 30 minutes, with a five- minute closing option, if members demanded it.

Last Thursday's outage was the bourse's third major market disruption in two years.

Mr Loh said the SGX is working with Nasdaq to improve the software to ensure it detects specific hardware problems.

The exchange will also work with members to improve the order and trade reconciliation process.

A version of this article appeared in the print edition of The Straits Times on July 20, 2016, with the headline 'SGX trading halt: Software failed to detect faulty disk'. Print Edition | Subscribe