(source: SingTel)
In a rather ironic way, the fire that broke out at a SingTel telecom exchange yesterday could not have come at a better time.
This is not meant to rub it in for the thousands of users who have been cut off from broadband, phone and pay-TV services, or the folks who could not pay for stuff electronically or draw money from ATM machines.
Indeed, so widespread are the problems this time that there has not been a more pressing need for Singapore’s government regulator to take a closer, tougher look at such critical infocomm infrastructure.
The Infocomm Development Authority (IDA) was just planning to regularly check up on mobile networks run by the Big Three telecom operators in the country.
It had said this last week, as it handed out an unheard-of S$1.5 million fine to M1 for a major disruption to its mobile network in January this year. That outage lasted an unprecedented three days.
Then, a sprinkler had accidentally sprayed water on network equipment during a scheduled upgrade. Ironically, this time with SingTel, the fire could not be contained automatically and the firemen had to be called in.
Though they put the fire out in a quick 20 minutes, the damage to the fragile fibre optic cables – thin strands of glass fibre wrapped in protective rubber coating – had already been done.
SingTel then quickly got more than 100 engineers to work, but the network has now taken more than a day to restore fully. At 7pm today, the company said corporate customers should have everything back working, while others would have to wait until 7am tomorrow morning.
The damage, according to the biggest telco here, had been more extensive than expected. What it would have quietly acknowledged too is that it was lucky only one chamber of the facility carrying the cables was damaged.
Such central offices – there are nine throughout Singapore – also hold critical network equipment for other telcos, such as StarHub, M1 and Viewqwest, which run their own network switches that connect to the fibre optic cables.
If that equipment had been damaged, a lot more users would have been affected. Things would also take a lot longer to get back to normal. It’s like having to re-configure your home router from scratch after the old one had burnt out.
Still, the impact of the latest disruption has been widespread. And coming after several unprecedented outages in the past two years, there are genuine questions now. Just how is resilient Singapore’s critical infocomm infrastructure?
IDA’s call for mobile networks to be made more ready for such outages – and subjected to regular audits – should now be expanded to critical infrastructure like the Bukit Panjang exchange that has been damaged.
If that is already being checked on now, the lessons learnt from this latest outage should make it into a checklist for steps to take for the fastest restoration of services.
Understandably, there are many questions from the public.
What caused the fire? Why didn’t some inert gas, common in many data centres, get released to put out the fire? No answers for now.
That’s still being investigated and it’s fine for now that SingTel and national broadband network operator OpenNet, whose cables were also damaged in the facility, concentrated on getting things back online first before giving a full answer.
Could telcos not make things more resilient by having “redundant” or spare equipment running alongside a live network? That works in, say, having a second hard disk to back up data but might not be as practical in a nationwide network.
Laying a second bunch of cables on alternative routes, not to mention running a second set of network gear in standby, would more than double the cost – easily in the millions of dollars – to everyone connected islandwide. Would users want to pay for that additional cost?
In all well-run networks, there is already some sort of redundancy built in. One of the first things telcos do when a cable is cut accidentally, for example, is to re-route some of the most important data, say, to ATM machines, via alternative routes. Other users may take a while more to get online, as the damaged cables are patched back.
Ultimately, the response to this setback has to be a considered, effective one. Are these disruptions of late all separate incidents, or is there a common thread, a root cause that points to a systemic issue? Is this down to a lack of preparedness? Or, complacency in maintenance?
The solution may have to be more than fines, even if a telco is at fault. The IDA and telcos here will have to ramp up regular, tougher audits to ensure that Singapore’s fast, affordable telecom services are just as reliable. Recently, that reputation has been tarnished.
