Data centre ‘building doctor’: He keeps power, cooling systems running

In this series, people from all walks of life offer an inside – often unfiltered – take on their livelihoods and what keeps them going in their jobs. In this instalment, hear from data centre engineer Keith Fong, as he shares what it takes to ensure a constant supply of power and cooling to support AI workloads around the clock, as told to The Straits Times.

I am 29 and a trained mechanical engineer working in a data centre’s site operations team.

No, I’m not a software service engineer or someone providing IT support. I am one of four team leads in our 20-strong department at the STTGDC data centre in Tai Seng, taking care of the building’s day-to-day operations around the clock. We are responsible for the monitoring and maintenance of the data centre’s physical infrastructure and critical systems.

STTGDC has six data centres in Singapore and over 100 globally, including in Malaysia, India, South Korea, Japan, Britain and Germany.

A data centre houses computer servers, storage systems and networking equipment that store and process data to enable many of the digital and cloud services people use every day, such as banking transactions and online shopping.

My team and I monitor these systems, coordinate maintenance, plan for contingencies and respond to alarms or abnormalities when they occur. We have to ensure uninterrupted cooling and power around the clock. There must be no downtime at all.

We also provide “remote hands” service to our customers. This means I can check equipment, investigate faults and troubleshoot issues on behalf of our customers, if needed.

On a typical workday, I can easily put in more than 10,000 steps as I inspect critical equipment across six floors with a total area equivalent to that of two football pitches.

Data centres are considered a critical service, like hospitals and traffic lights. A data centre outage would disrupt digital services that businesses and consumers rely on, with knock-on effects across the economy.

We have very strict security procedures, including a fence intrusion detection system, mantrap access control to data halls and on-site security guards.

There are several types of data centres. STTGDC provides colocation, which means it rents out space, power and cooling services to companies. The IT equipment inside the data centre belongs to our customers.

Data centres previously handled less power-intensive workloads, such as e-mails and websites. But the growing use of AI requires more powerful servers with greater processing capacity, driving up demand for colocation, connectivity and support services.

During any particularly critical period for our customers, we want the data centre to be as stable as possible and avoid introducing unnecessary system changes or maintenance.

My team and I are like doctors of the building. During routine inspections, we are alert to anything that looks, sounds, smells or feels different from the norm – an area may be abnormally hot or there may be a burning smell, for instance.

We also rely on tools like sensors. For example, our data centre is equipped with VESDA, or very early smoke detection apparatus system, which continuously samples the air and can detect very small concentrations of smoke particles, long before conventional smoke detectors or the naked eye picks it up.

I have a degree in mechanical engineering, which taught me more about refrigerant cycles and computational fluid dynamics. This knowledge is commonly used in data centres.

I joined STTGDC as an intern about five years ago, before joining full-time to work 12-hour shift cycles. Now, as a team lead, I work office hours but can always be recalled if there are emergencies, acting as a second point of escalation.

No day on the job looks the same. The air-conditioning might be working today, but not tomorrow. So how do we anticipate such events? And what are our backup procedures to make sure that cooling is not affected?

We have resilience measures. When a power outage occurs, the lighting may go out for a few split seconds, but our equipment – ranging from backup generators to uninterruptible power supplies – kicks in to protect our facility from going down. The servers keep running.

We are also trained emergency responders. If a fire breaks out at the data centre, we would first assess the situation and carry out appropriate initial responses, including attempting to contain or extinguish the fire. At the same time, we would notify SCDF so that professional emergency responders can attend, if required.

We can administer first aid, cardiopulmonary resuscitation and an automated external defibrillator if necessary.

There is definitely an adrenaline rush any time a critical alarm goes off. We would have to be on-site to assess the situation, identify the cause and work together to resolve it safely while ensuring no disruption to services. Ultimately, our role is to maintain a stable environment so that the IT systems can continue operating reliably.

Different customers and IT equipment can have different environmental requirements. This is typically 18 deg C to 27 deg C and 20% to 80% for relative humidity. STTGDC also takes reference from Singapore’s Tropical Data Centre standard, as part of the broader industry approach towards improving data centre energy efficiency here.

Cooling a data centre is not about making the room as cold as possible. It is about maintaining a stable and controlled environment.

We do not want condensation forming around electrical equipment, but the environment cannot be too dry either, to prevent risk of electrostatic discharge.

Data centres are not new and have been around since the start of the internet boom in the 1990s, storing and hosting lower power workloads like e-mails and websites.

Cloud computing was a major driver of the data centre boom long before the AI surge, as businesses increasingly moved away from running their own servers and outsourced their computing and storage needs to cloud providers.

More recently, our customers increasingly want to install powerful AI servers. The power demand of such AI server racks is about 20kW, compared with 2kW to 3kW for conventional server racks.

People often say data centres are highly water- and power-intensive. In Singapore, there are regulatory limits for the data centre industry. The water we use is non-potable.

Similarly, when a building has been allocated a certain amount of electricity, the set limit will not increase just because there are AI workloads. Our Tai Seng data centre has an IT load of 13.7MW – meaning this is the total power it can draw for the computing, storage and networking hardware.

Across STTGDC’s global data centres, about 83% of our electricity is renewable. We recycle the water used for our cooling systems before releasing it as sewage. We use AI to analyse data and monitor trends across the day or the hour to improve the data centre’s energy and water efficiency.

We recently replaced 10 rooftop air-cooled chillers with ones that are 39% more energy-efficient. This was one of my biggest challenges on the job, as we had to maintain cooling while the works were carried out. Planning itself took about three years, while execution was done in phases over a year to avoid impacting customers’ equipment.

When nothing goes wrong, I know that I’ve done my part to help the digital economy run smoothly like clockwork. People can send out their e-mails and messages and make payments without any issues. That brings a quiet satisfaction to us as data center engineers.

Got an interesting job to share? Write in to us at ssharon@sph.com.sg. Catch up on the rest of our On The Job series here.