Posted on Leave a comment

Amazon’s US-East data center agony – takes Netflix offline for 18 hours on Christmas Eve

image thumb221

Amazon logoAmazon’s US-East Region 1 data center is a popular choice for their ELB (Elastic Load Balancing) service because it’s cheap and typically is the first data center to get new services when Amazon rolls them out.  It also tends to be a headache for US-East customers who rely on its CDN delivery network.  On Christmas Eve at approximately 2:00 PM PST, the US East data center, located in suburban Washington, D.C., went offline and stayed down until around 8:45 AM PST knocking Netflix, Heroku (but not Amazon Prime) offline during a critical customer viewer time-frame.

Netflix noted on Twitter:

We’re sorry for the Christmas Eve outage. Terrible timing! Engineers are working on it now. Stay tuned to @netflixhelps for updates. — Netflix US (@netflix)

Bunch of ELBs down, lots of happy Netflix instances getting no traffic, still waiting for AWS to fix it. Some devices working, others not. — adrian cockcroft (@adrianco)

While Amazon noted on their service health dashboard:

5:49pm PST: “we continue to work on resolving issues with the Elastic Load Balancing Service in the US-EAST-1 region. Traffic for some ELBs are currently experiencing significant levels of traffic loss.”

This isn’t the first time the US-East data center has cratered.  An outage in October knocked Pinterest, Instagram, Foursquare, and Reddit offline.

[lws_easteregg p=22825]