Google App Engine Back Up After Major Service Disruption – Dropbox and Tumblr Also Suffer

Update: Google App Engine is back up. They just issued a statement:

Google App Engine has now been restored and users should see service returning to normal. Our team is still continuing to investigate and determine the root cause of the issue. We know that many of our customers rely on App Engine for their mission critical applications. We apologize for the inconvenience caused by this outage and we appreciate our customers’ patience.

A Google spokesperson said an “event” occurred this morning,  which caused the load balancing issue.  They are still looking into the root cause. They plan to post an incidence report.

In order to restore service, Google had to shut down traffic, then slowly bring the service back up to a stable level. Google said this is the first major “High Replication Datastore” outage they’ve had since moving to this infrastructure. Google App Engine had developed a reputation for going down during its beta, which ended earlier this year.
Here’s our report from earlier today:

 Google App Engine is down with a major service disruption affecting thousands of customers around the world that use the platform-as-a-service.

The Google  Engine team posted the following update:

At approximately 7:30am Pacific time this morning, Google began experiencing slow performance and dropped connections from one of the components of App Engine.  The symptoms that service users would experience include slow response and an inability to connect to services.  We currently show that a majority of App Engine users and services are affected.  Google engineering teams are investigating a number of options for restoring service as quickly as possible, and we will provide another update as information changes, or within 60 minutes.

Dropbox and Tumblr also reported outages, pointing to a potential much larger issue across the web.

On Hacker News, comments on the outage point to the Internet Traffic Report, which shows significant packet loss in the past 24 hours as well as traffic disruptions. The Next Web has a good chronicle of the issues facing the different services.

Nimsoft reports the outage is affecting users around the world, with particular issues in Europe. France is pretty much out entirely. In South America, Chile is entirely down. The United States and Canada are hovering around 70% performance.

Here’s the status page from Google: