Back in 2011, Tim was working at a company that was the equivalent of a Rackspace in Europe, half the infrastructure was there, the other half in the US. The biggest client was Facebook and suddenly the connection between the two continents went dead. There was monitoring in 2011, and all the charts flatlined. The company didn’t own their data center and their colocation center kept saying that everything was fine, but obviously, it wasn’t.
The connection kept going up and down like a roller-coaster every hour or two with no explanation, Facebook were on the phone saying they would drop the company, and as 75% of business, that wasn’t good to hear. Lo and behold it turned out that at the facility there was a primary power generator and a backup generator, the primary went down and the backup was struggling, then Air-conditioning shutdown and the heat built up which caused even more issues. The reason the primary shut down? Someone tripped over the main power cable and had physically unplugged it.
Lesson learned: If you use a colocated data center then check they are secured, competent, and check for all potential problems.