Facebook Instagram and WhatsApp -BGP repeat after me bad BGP configuration -updated
Repeat after me - this is not some crazy complex conspiracy that a group of hackers decided to tango down their BGP and/or DNS -no this is likely an unintentional human error
Which was quickly followed by WhatsApp, Facebook and Instagram
Note the time date stamp of each tweet
Basic understanding BGP & DNS
When you type in a website that website translates to an IP address but there are a ton of steps that occur in nano-seconds. A bad routing error can result in unresolved address. There are known inherent risk of scaling up BGP -especially in a data center environment. Generally speaking BGP is used to “load balance Internet traffic” while it does have far more complex capabilities —there are always inherent risk associated with pushing a product’s limits. Here’s the thing -this isn’t the first time a shitty misconfiguration made Facebook et al’s website inaccessible;
Public Report Released April 2021…
Concerning Facebook et al BGP, Data Center and DNS -while this largely flew under the radar it is a very important academic and IT white paper. And while it might get you a few clicks and/or retweets but nothing in the public domain suggest that hackers penetrated Facebook’s various networks. One could argue as it relates to insider threats —it’s entirely possible that this wasn’t a malicious “insider threat” more like an accidental misconfiguration.
Running BGP in Data Centers at Scale…
Anubhavnidhi Abhashkumar and Kausik Subramanian, University of Wisconsin–Madison; Alexey Andreyev, Hyojeong Kim, Nanda Kishore Salem, Jingyi Yang, and Petr Lapukhov, Facebook; Aditya Akella, University of Wisconsin–Madison; Hongyi Zeng, Facebook 2
https://www.usenix.org/conference/nsdi21/presentation/abhashkumar or found here
,..we present Facebook’s BGP-based data cen- ter routing design and how it marries data center’s stringent requirements with BGP’s functionality. We present the de- sign’s significant artifacts, including the BGP Autonomous System Number (ASN) allocation, route summarization, and our sophisticated BGP policy set. We demonstrate how this design provides us with flexible control over routing and keeps the network reliable. We also describe our in-house BGP software implementation, and its testing and deployment pipelines
…past research has shown that BGP in the Internet suffers from convergence issues [33, 37], routing instabilities [32], and frequent misconfigurations [21, 36]. Since we control all routers in the data center, we have flexibility to tailor BGP to the data center which wouldn’t be possible to achieve in the Internet.
We show how we tackled common issues faced in the Internet by fine-tuning and optimizing BGP in the data center (§4). For instance, our routing design and predefined backup path policies ensure that under common link/switch failures, switches have alternate routing paths in the forwarding table and do not send out fabric-wide re-advertisements, thus avoiding BGP convergence issues.
Factually Important observation 👇🏻
“Between 15:50 UTC and 15:52 UTC Facebook and related properties disappeared from the Internet in a flurry of BGP updates," explained John Graham-Cumming, CTO of Cloudflare.
"The BGP routes pointing traffic to Facebook's IP address space have been withdrawn. The Internet no longer knows where to find Facebook's IPs. One symptom is that DNS requests are failing," added Johannes B. Ullrich, Ph.D., Dean of Research at the SANS Technology Institute…”But this is just the result of Facebook hosting its DNS servers inside its own network. Even with working DNS (for example if you still have cached results), the IPs are currently not reachable."
So the point? There is nothing in the public ethos to confirm Facebook et al were victims of a cyber intrusion/hack. Yes I know the timing is curious given the explosive allegations disclosed in yesterday’s 60 Minutes Segments
..although it is possible that an after action report might (at a later date) determine Facebook was hacked —but again right now there’s nothing in the public domain that supports that. What is currently in the pubic domain is Facebook had a bad BGP configuration, coupled with the documented history of previous outages due to a bad configuration. I know it’s complicated but let me give you an analogy to understand: you send a letter to a specific address but the post office returns that letter as an undeliverable address ← that is probably the most simplistic analogy to explain the current outage.. in that analogy BGP is the Post Office which returns the “undeliverable letter” back to the sender. When you try to go to Facebook -your request is returned because Facebook’s ANS servers can’t be (currently) located. I hope that makes sense and explains a really complex issue in simplistic terms…
On a related note - yesterday (October 3rd and on September 30th) chatter was a hacker had obtained nearly 1.5B as in BILLION Facebook Users PII -this is entirely separate - but it appears many have decided to conflate the DarkWeb Leak and today’s Facebook et al services being down - FFS the article (embedded below) explicitly states this as a caveat…
…Clarification: This is completely unrelated to the global Facebook outage experienced on 4 October 2021…
In late September 2021, a user of a known hacker forum posted an announcement claiming to possess the personal data of more than 1.5 billion Facebook users. The data is currently up for sale on the respective forum platform, with potential buyers having the opportunity to purchase all the data at once or in smaller quantities.
One prospective buyer claims to have been quoted $5,000 for the data of 1 million Facebook user accounts.
https://www.privacyaffairs.com/facebook-data-sold-on-hacker-forum/
Also Facebook Domain up for sale? Survey says um NOPE https://domainnamewire.com/2016/04/14/facebook-now-owns-domain-name-registrar/ because
Facebook is its own domain name registrar—and Registrarsafe.com is also offline, as it shares infrastructure with the rest of Facebook.
October 4, 2021 7:20PM update…
It was a bad BGP configuration that caused the hours long outrage - not some masterful hacker …at least that’s what Facebook has publicly stated..
…Doug Madory is director of internet analysis at Kentik, a San Francisco-based network monitoring company. Madory said at approximately 11:39 a.m. ET today (15:39 UTC), someone at Facebook caused an update to be made to the company’s Border Gateway Protocol (BGP) records. BGP is a mechanism by which Internet service providers of the world share information about which providers are responsible for routing Internet traffic to which specific groups of Internet
And honest to god this is a HILARIOUS TROLL by Prime Video - h/t to Loo -she instantly gets my sense of humor…
Whispers should we also talk about Twitter’s issues today concerning the overflow of inbound IP traffic too?
Back in 2019 it was falsely reported that Facebook had been hacked or targeted for a protracted DDOS attack - none of that was correct - it was later identified as a bad BGP configuration
The April 2021 BGP (more specifically eBGP) whitepaper is available via Facebook Research website (but clearly that site is currently inaccessible
https://research.fb.com/wp-content/uploads/2021/03/Running-BGP-in-Data-Centers-at-Scale_final.pdf —hence why I embedded the University of Wisconsin’s link
Thank you for slamming on the breaks here. 🥰 Taking deep breaths now.
The timing was ripe for people to lose their minds. The Pandora Papers and the 60 Minutes Whistleblower interview had just hit the news. Thanks for this! Employees using outlook and texts to communicate! Must’ve been quite a scene....plus not being able to get into their offices. Former FB guy Wajahat Ali wrote a very good opinion piece for The Daily Beast. Have a nice evening Filey!