Ho John Lee’s Weblog

I’ve generally been satisfied with hosting services at Dreamhost, which provide a lot of capabilities at a modest cost. However, yesterday’s power outage in Los Angeles shut down Dreamhost and a number of other sites in data centers that were supposed to have hardened power and redundant network connections. An obvious question is: what happened to the backup power?

One of the main points of using a hosting or colocation service is having better connectivity, environmental controls, and power. In Dreamhost’s case, the latter would be the backup UPS and diesel generators which are supposed to start up when the power grid goes offline.

There is a series of posts on the Dreamhost blog on yesterday’s outage. Looks like the upstream network providers (Level 3, Global Crossing, and Mzima) failed while DH still had power from their backup system, then a few minutes later the backup power failed.

Shortly thereafter the entire building where our data center is located’s back-up generators (there are SUPPOSED to be four) stopped working, and all power was gone. We were able to get back into our data center then, and it was like the day after tomorrow or something. Really creepy just walking through rows of dark, quiet, dead server after dark, quiet, dead server.

Dreamhost is apparently housed at Media Temple’s Garland Building, which is equipped with networking, HVAC, and backup power for data centers. Here’s what Media Temple had to say to their customers:

On Monday, September 12, 2005, at approximately 12:35 p.m., the building experienced a total loss of electrical power from the DWP on their primary grid. At this time, the building generators started and began supplying adequate power to the tenants.
At approximately 12:55 p.m., the building power was partially restored by the DWP. At approximately 1:05 p.m., the building experienced another total power failure from the DWP.
During this period of 12:55 p.m. to 1:05 p.m., two of the five generators failed. The remaining three generators were unable to sustain the power requirements of the building causing the emergency electrical systems to transfer into a “load shedding mode” and the building’s UPS system to turn itself off, thus preventing permanent UPS and related equipment damage.

Media Temple’s own site has some info on the power outage:

Question: Doesn’t (mt) Media Temple have UPS Backup? Answer:
Yes. (mt) Media Temple and the Garland Building (our LA data center building location) has one of the most sophisticated, redundant power systems in all of Los Angeles. As a matter of fact there are few buildings that rival the amount of redundancy and investment which has been put into the Garland Building’s backup power systems. As explained however today; the building’s power failure was caused by “human error” similar to the larger, citywide issue.
When the citywide power outage occurred, the Garland Building’s main UPS and backup power generator systems activated successfully and provided power to the building during the early stages of the city’s power outage. After approximately 30 minutes, the Garland Building engineers began to “further assist” the power situation which resulted in some form of “human error” on their part and interrupted the backup power service altogether thus causing the entire building to loose power. All tenants and data center occupants were instructed to immediately evacuate the building as the building engineers continued to “work” on the issue. After approximately 2 hours, building engineers restored power to the building near simultaneous to the rest of the Los Angeles power deprived areas. Building engineers state they are acutely aware of what caused the Garland Building power failure and have remedy to prevent such failures in the future.

So, we have a large portion of the Los Angeles power grid accidentally shut down by a small human error, followed by a very sophisticated backup power system accidentally shut down by another small human error. It’s remarkably common for these sorts of problems to come up, as most facilities don’t actually run in their “backup” modes enough to test regularly. A few years back, we had a cage full of servers in a brand new colo facility, which successfully switched to generator power during a rolling power blackout, but then shut down a while later due to heat buildup since the building HVAC wasn’t on backup power yet.

Gives you greater appreciation for this guy in New Orleans, who kept the DirectNIC data center running through Hurricane Katrina and its aftermath.

This entry was posted on Tuesday, September 13th, 2005 at 9:11 pm and is filed under Web Site Development, Front Page. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.