How Bad is MainOne’s BGP Error and Why They Must Prevent a Recurrence

Yesterday, Nigerian internet service provider, MainOne admitted that it made a configuration error during a network upgrade. That error caused a disruption of key Google services by routing traffic to China and Russia.

Nigeria’s @Mainoneservice has claimed responsibility for a 74-minute glitch during a network upgrade on Monday, which accidentally resulted in some Google global traffic being routed through China and Russia. [Reuters]

— Techloy 💙💛 (@techloy) November 14, 2018

The 74-minute glitch was caused when MainOne mis-configured a border gateway protocol (BGP) filter used to route traffic across the internet. This resulted in some Google traffic being sent through MainOne’s partner, China Telecom.

During this period, some Google services were not accessible to some of its customers who preferred to use the leaked routes. However, MainOne claims that the routes were immediately shut down as soon as the errors were discovered.

In the end it wasn’t a major disaster because no clearly tragic incident has followed. But it could easily have been very bad.

What may have happened?

In the past, BGP filter glitches such as this one have caused multiple outages. This shows that traffic from global technology companies like Google could be vulnerable to disruptions that may be caused by problems in other firms, especially internet partners around the globe that help direct internet traffic.

According to Gadgets 360,  MainOne spokesman Tayo Ashiru explained that engineers mistakenly forwarded addresses for Google services that were supposed to be local to China Telecom. The Chinese company in turn, sent the bad data to Russia’s TransTelecom, a major internet presence.

Ashiru said MainOne did not yet understand why China Telecom did that as the Russian state-run company normally doesn’t allow Google traffic on its network.

Costs and Dangers…

This type of traffic misdirection has the capacity to knock essential offline services as well as facilitate espionage and even financial theft.  Recall that Google was similarly afflicted in 2015 when an Indian provider jumbled up its services. A similar occurrence, last year, led to an internet blackout in Japan.

In perhaps the best-known case, Pakistan Telecom used a similar opportunity in 2008 to hijack YouTube’s global traffic for a few hours while trying to enforce a domestic ban. The simple action by the state-owned company sent all YouTube traffic into a virtual ditch in Pakistan.

Big news coming out of #crypto—"the theft, from users of https://t.co/w9k7aotwS5, resulted in a loss of around 215 #Ether, or roughly $152,000-worth of the #cryptocurrency"–@iblametom explores the blind spot in #internet #security https://t.co/pZRGNWqWNk via @Forbes

— Olga Feldmeier (@OlgaFeldmeier) April 28, 2018

There is also a history of downside effect on financial companies. In April 2017, a similar incident affected MasterCard and Visa among other key sites.  And in April 2018, another hijacking enabled cryptocurrency theft. According to data from Chainalysis which tracks digital money,  that incident resulted in a loss of around 215 Ether, or roughly $152,000-worth of the cryptocurrency.

Forbes described the incident graphically:

“…Hackers were able to take control of a number of Domain Name System (DNS) servers to redirect users who were trying to reach myetherwallet.com, taking them instead to a phishing site where the keys to their accounts could be stolen.”

The painful part is for how long everyone was taken for a ride:

“This redirection was happening for two hours without anyone noticing. And it was possible because of what MyEtherWallet described as a decade-old attack, taking advantage of vulnerabilities in public facing DNS servers.”

Way forward…

It is a good thing MainOne has come forward to take responsibility and promise to prevent a recurrence. The company’s press release reads:

“In order to prevent a re-occurrence, we have reviewed our supervisory and configuration policies to ensure these kinds of errors are avoided in the future. MainOne appreciates our partners for their patience and cooperation through this brief period of internet disruption and we assure you of our continued commitment to the high degree of trust that makes the internet work efficiently.”

We really hope that holds true for all of us.