This is the best thing I've read all day.

While I'm not sure what this exact probe is built upon, some popular choices for mission critical applications I've seen in the past are:

* VxWorks for real time operating system (proprietary)

* Ada/C++ for programming language.

> Of course, I could be completely wrong, and missing a great alternative service.

AppNeta TraceView is an alternative to New Relic: http://www.appneta.com/products/traceview/

Disclosure: I work for AppNeta.

interestingly enough, here's the original commit that defaulted `validate=True`:

https://github.com/boto/boto/commit/95939debc3813468264159d5...

EDIT: Looks like the original committer is an Amazon employee.

The commit added another parameter so it is possible to skip the expensive get_all_keys call.

It is defaulted to true so that old code that called get_bucket() won't break. (Since old code would have used get_bucket with only one parameter.)

It was always true before that commit.

Therefore, in essence, an Amazon employee made it possible to save 90% of your S3 bills.

Before you jump on the conspiracy train, most Boto contributors are Amazon employees.

Agreed, I doubt Amazon had any evil intentions with this change.

I wonder more if this was a product of Amazon developers using S3 (i.e. dogfooding) and not noticing the cost side effect because I'm assuming they don't get billed?

The change is the opposite of what you're implying, it adds the option to turn it off. Before this commit it was on by default. My guess is, when the option was added, it defaulted to validate=True to maintain backward compatibility.

We actually do get billed, but we don't have to pay. I always check my bill to make sure that I am not using any resources that I don't need.

I also pay for my own personal EC2 instance and about 350 GB of S3 storage. Begin a genuine user and customer of AWS helps me to be a better employee.

Jeff,

You may want to check LIST request statistics over the next few weeks. Between this thread, an Issue for boto, etc. I'm curious if you see a noticeable decline in LIST requests with the attention this has brought. I'm just curious from a data standpoint.

Ahh cool! I completely agree that using your own products from a customer point of makes a better employee. Thanks for the input =)

Mitch just started working for AWS ~1.5 years ago or so I think? That code was written long before he worked for Amazon.

Yeah don't hate on em, Boto folks are pretty helpful.

So the committer added an option to avoid the expensive behavior and that's a sign of evil?

> interestingly enough, here's the original commit that defaulted `validate=True`:

It's a commit which specifically added validate to allow skipping validation. If you read the diff, the call originally unconditionally performed the validation call.

That's a change by the original author of boto. I don't think he was an Amazon employee that far back...

This is a bit easier to read if you ignore whitespace (add ?ws=1 to the end):

https://github.com/boto/boto/commit/95939debc3813468264159d5...

i updated the article to move the inline update to the `2. Enable mod_pagespeed` section as requested =)

Final table no longer makes sense. Might make more sense to extract your update to a new article and link to it from the top?

nice! just ordered a used copy =)

"During pager hand-off, last week's guy and this week's guy should talk about what happened and if there's anything they should know"

Agreed, we were thinking of doing week long rotations (Tuesday - Tuesday) with a "hand off conversation" happening on Tuesdays.

Tuesday does solve the 3 day weekend problem. What do you do if Monday is a holiday? Trade on Monday morning and meet up outside of work, or just hold it till Tuesday. Most of the time we just hold it.

The reason for this discussion is because up until a certain seniority level, you get "hazard pay" for carrying the pager. You get paid 1 hour for every so many you're on call. A weekend/holiday is 24 hours instead of 8 on the day your receive it or 16 on a weekday.

You should also cover rules for holding the pager. Ours include no alcohol, and no more than 1 hour away from the site (certain emergencies may require on-site visits). You also need to respond within 20 minutes, otherwise it gets escalated, or in certain larger locations, sent to the backup on-call person.

Wow, thanks for the brain dump!

We're an established team/product, so we have an internal wiki, help/support desk, and use PagerDuty. We just want to shift away from only a few people (basically 2) handling DevOps emergencies and spread the experience over more members of our engineering team.

With the "all on call/no schedule" route, have you ever had a scenario where no one acknowledged an issue?

We have not discussed this point, but I'm assuming it's off the table due to cost. And yes, this is for 24x7 rotations where everyone is in the same timezone...

I'd love to get feedback on pricing etc, to see how far off I am, if you have a second to email me? If not, nbd.

http://blog.pagerduty.com/2011/03/on-call-best-practices-par...

This is a series of posts that have pretty sane defaults; I personally would not do a daily rotation, but rather rotations of 5 days, and alternating weekends (one guy does M-F, one guy does Sat/Sund) and you switch off.

Nice link! And I agree the daily rotation is a bad idea, however we're leaning towards doing a Tuesday-to-Tuesday week long shift.