We've used it occasionally on a ~400GB table and it's worked perfectly for us too. We run it on EC2 i2.8xlarge instances and it finishes in under 2 hours. But it doesn't use very much CPU or I/O, so we could probably run it just as well on a smaller instance like an i2.2xlarge.
One downside is that you need enough free space on your disk to hold the packed copy of the table, so you need to run it before your disk gets too full. The last couple of times we've waited too long, so we've had to move the data to a larger host before we could repack it.
Just spoke to AWS Premium Support around 4:35 AM. They don't have any more information than what's on the status page, basically. I was hoping to get an ETA for a resolution, but no such luck.
We're seeing the same problems with DynamoDB... it started around 2:20 AM PDT (1 hour ago). We're not having any problems with EC2—what kind of problems are you seeing? I hope it's just affecting the API and not running instances.
Side note: Medium posted an article last week about how most of their data is stored in DynamoDB, so I'm sure the Medium outage is related.
Right now I can't load Lambda, CloudWatch or SQS. People on Twitter reporting massive ASG issues that caused their deployments to scale up massively suddenly.
S3 is already supposed to be AZ-tolerant. In the US Standard region, they say your data is actually replicated across multiple geographic locations in Virginia and the Pacific Northwest. That's probably true—your data may be highly durable—but availability incidents can apparently span both geographic regions.
S3's uptime is probably an order of magnitude better than I can provide myself. From 2010-2011 I worked with a medium-size MogileFS cluster, with about 160 million files and 50 TB of data, triple-replicated. We had great uptime until we ran out of I/O capacity on the metadata DB. Then we had a week of 10% downtime, while we ordered new database servers from Dell.
As far as I remember, S3's US Standard region hasn't had a serious incident since Fall 2012. That's a pretty great uptime record in general, even though it's terribly frustrating on days like today.
My current company spends a few hundred dollars a month on S3. I certainly couldn't match S3's uptime at that budget. Maybe I could do it at 2-3X the budget, but with that budget it might be easier to mirror my files across multiple S3 regions.
> My current company spends a few hundred dollars a month on S3. I certainly couldn't match S3's uptime at that budget. Maybe I could do it at 2-3X the budget, but with that budget it might be easier to mirror my files across multiple S3 regions.
Exactly. Build multi-region support into your app(s), enable S3's replication so objects in your primary region are replicated to another region, and then properly handle loss of a region (go read-only or write to another region and restore consistency later).
You'll still be spending less than attempting to maintain a highly durable object store yourself across multiple datacenters or geographic areas.
