XT back, but will business forgive?

Around 100,000 customers on the XT Network awoke yesterday morning to a day without cellphone coverage. A hardware failure at 4.30am triggered a massive outage in the network which affected all areas south of Taupo.Telecom kept the public informed on the outage from 7am on Twitter and Gen-i head Chris Quin sent emails to the company’s corporate clients throughout the day. TR kept pace with the following stories:Major XT Network outageTelecom apologises for ongoing XT outageXT services "functioning normally" - Quin

According to the Gen-i emails, the outage was caused when “we experienced an unplanned reset due to hardware failure in our Christchurch XT Mobile switch which resulted in the majority of XT cell sites between Taupo and the bottom of the South Island not operating.”

Telecom is considering how it will compensate customers for the outage. Meanwhile rumours are flying around the industry, with some suggesting that during installation of a software upgrade to fix a fault that occurred three week ago the network crashed.But many are shaking their heads in wonder – and horror – that such a massive outage could affect over half the country and that the network could have been down for so long. “This is much bigger than when IBM’s databank outage shut down Air New Zealand,” said one business user spoken to by TR.At around 2.30pm Quin was informing customers they were confident that services were back up, but official word from Telecom didn’t come until after 5pm.TUANZ Chair Chris O’Connell was fielding calls from business users affected by the XT outage during the day.“People rely on this network and they feel like their throat’s been cut,” he says. “It’s frightening how a single point of failure can bring down a network.”AA CIO Doug Wilson, whose organisation recently switched to XT says the company lost voice contact with the AA service fleet, but not data because that capability is still with Vodafone.“We are a bit lucky; we haven’t converted our mobile data across to XT from VFNZ (due to older hardware). We have only lost mobile phone connectivity which is only annoying,” he says. “In other words, we are still functional, but can’t speak to our people. Our road service customers are not affected.”However it’s likely that AA members who broke down in many rural areas would have experienced long delays as there was no way to contact the contractors who service those areas for the AA.Wilson is disappointed it took so long to fix. “Outages shouldn’t occur but when they do, they should be remedied much more quickly.”Fonterra CIO Chris Barendregt says the dairy giant – another recent convert to XT - says it wasn’t affected because it hasn’t switched cell phone coverage over in areas south of Taupo. But he says he’s been kept well informed by Telecom and Gen-i and appeared relatively at ease with what had occurred.In a positive for Telecom, neither Wilson nor Barendregt – both of whom are major customers of XT – said the outage has made them consider ditching Telecom's service.But will they be so forgiving if there is a next time?