"Lower-end modems" root cause of Spark outage...
Spark New Zealand has moved to explain its recent distributed denial of service (DDoS) attack, blaming the recent outage on poorly-configured modems.
In an address to the media, the telco says overseas hackers appear to have been attacking web addresses in Eastern Europe, and were bouncing the traffic off Spark customer connections which caused the DDoS attack.
“The DDoS attack was dynamic, predominantly taking the shape of an ‘amplified DNS attack’ which means an extremely high number of connection requests – in the order of thousands per second - were being sent to a number of overseas web addresses with the intention of overwhelming and crashing them,” says Richard Llewellyn, Head of Corporate Communications, Spark New Zealand.
“Each of these requests, as it passes through our network, queries our DNS server before it passes on – so our servers were bearing the full brunt of the attack.”
While the Spark network did not crash, the telco did experience extremely high traffic loads hitting its DNS servers which meant many customers had either slow or at times no connectivity (as their requests were timing out).
“There were multiple attacks, which were dynamic in nature,” Llewellyn adds.
They began on Friday night, subsided, and then began again early Saturday, continuing over the day. By early Sunday morning traffic levels were back to normal and have remained so since.
“We did see the nature of the attack evolve over the period, possibly due to the cyber criminals monitoring our response and modifying their attack to circumvent our mitigation measures – in a classic ‘whack a mole’ scenario,” Llewellyn adds.
How did they get access through the Spark Network?
Since the attacks began, Llewellyn says the company deployed people working 24/7 to identify the root causes, alongside working to get service back to normal.
“During the attack, we observed that a small number of customer connections were involved in generating the vast majority of the traffic,” he adds.
“This was consistent with customers having malware on their devices and the timing coincided with other DNS activity related to malware in other parts of the world.
“However, while we’re not ruling out malware as a factor, we have also identified that cyber criminals have been accessing vulnerable customer modems on our network.”
Llewellyn says these modems have been identified as having “open DNS resolver” functionality, which means they can be used to carry out internet requests for anyone on the internet.
As a result this makes it easier for cyber criminals to ‘bounce’ an internet request off them (making it appear that the New Zealand modem was making the request, whereas it actually originates from an overseas source).
Llewellyn also stresses that most of these modems were not supplied by Spark and tend to be older or lower-end modems.
“What remains clear is that good end user security remains an important way to combat these attacks,” he adds.
“With the proliferation of devices in households, that means both the security within your device and the security of your modem.”
What did Spark do?
Spark has now disconnected those modems from its network and are contacting all the affected customers.
“We have also taken steps at a network level to mitigate this modem vulnerability,” Llewellyn adds.
“We are now in the process of scanning our entire broadband customer base to identify any other customers who may be using modems with similar vulnerabilities and will be contacting those identified customers in due course to advise them on what they should do.”
With respect to malware, Llewellyn says Spark continues to “strongly encourage” its customers to keep their internet device security up to date, conduct regular scans and regularly update the operating software and firmware on their home network.
The telco also continues to advise customers not to click on suspicious links or download files when they are not sure of the contents.
“We have also taken steps at the network level to make it more difficult for cyber criminals to exploit the DNS open resolver modem vulnerability and we’re using the latest technology to strengthen our network monitoring and management capabilities,” he adds.
“For security reasons we can’t detail these steps, however this is an ongoing battle to stay one step ahead of cyber criminals who are continually using more and more sophisticated tactics.
“We can’t say what other networks experienced. However, cyber criminals often look for clusters of IP addresses to use in any particular DDoS attack.
“That makes it more likely that these IP addresses belong to the customers of a single ISP – even more likely with a large ISP like Spark.”