Especially bad for people that were using MongoHQ to manage their DB backups to S3:
As a precaution, we took additional steps on behalf of our customers to invalidate the Amazon Web Services credentials we were storing for you (for the purposes of backups to S3). While this prevents the abuse of your AWS credentials by any malicious party, it may have resulted in additional unintended consequences for your AWS environment if you were utilizing the same AWS credentials for other purposes.
If you had S3 backups configured with MongoHQ, when possible, we have disabled the IAM access keys via AWS. In any case, please go to the AWS Management Console and regenerate any keys given to MongoHQ .
I was one of those people bitten by this last night. My client called and told me he was getting access denied when trying to upload files through his CMS. After some digging I found the S3 key had been revoked. This was concerning, as I hadn't touched the CMS code I wrote in like 3 years and I've had issues deploying old stuff to Heroku in the past. I really wish MongoHQ had contacted me first about revoking the keys.
We're very sorry about this, we went into a little bit of a panic when we realized that we held IAM credentials that gave full access to peoples' EC2 accounts and did what we thought was best. In hindsight, we should have gotten ahold of Amazon immediately and let them manage that process.
So this is why our app broke yesterday. We've spent the last 24 hours wondering how our AWS key was removed. Thankfully I was able to learn about this via HN. Still waiting on a note from MongoHQ. And when we went to file an issue with AWS there was no Premium Support option.
We're very sorry about this, we went into a little bit of a panic when we realized that we held IAM credentials that gave full access to peoples' EC2 accounts and did what we thought was best. In hindsight, we should have gotten ahold of Amazon immediately and let them manage that process.
You should have a support issue open in your AWS portal now, if you need any help getting new keys for other apps. If you can't find it hit us up at support@mongohq.com and we'll escalate.
You are 100% right on both points. We'll be updating our security page as we get more details, I expect we'll have some rough timeline information tomorrow.
Thanks. I appreciate being overly cautious when it comes to security rather than under cautious. And this has made us realize that we should have a unique profile for each service.
I am commenting without knowing the specifics of your application, so apologies if it doesn't apply.
You should look into using separate AWS keys for your DB backups and whatever it is your app uses those credentials for. This not only prevents any future availability issues because of key revocation, it also allows you to set fine grained permissions on your access keys limited just to what they're being used for.
While annoying for users who we're using the credentials they provided for other things (which they really shouldn't have been doing anyway, and hopefully MongoHQ makes it clear that they shouldn't) this really seems like about the only thing they could safely do under the circumstances.
As a precaution, we took additional steps on behalf of our customers to invalidate the Amazon Web Services credentials we were storing for you (for the purposes of backups to S3). While this prevents the abuse of your AWS credentials by any malicious party, it may have resulted in additional unintended consequences for your AWS environment if you were utilizing the same AWS credentials for other purposes.
If you had S3 backups configured with MongoHQ, when possible, we have disabled the IAM access keys via AWS. In any case, please go to the AWS Management Console and regenerate any keys given to MongoHQ .