This is apparently a common issue as there are at least a dozen other posts in recent months on this subreddit with similar stories. I'm going to share my specific (active) case. I'm curious to hear the community's thoughts on how this should be handled and who should be responsible for what.
My AWS account was hacked
I had an inactive AWS account for over a year. Somehow my password (unique to AWS) was compromised, allowing a malicious attacker to charge my linked credit card and host some servers. Upon discovery of my credit card being used, I canceled the credit card, set up 2FA, changed my password on the account, and opened a ticket to AWS stating the situation.
I did not touch the AWS-specific software the attacker configured on my account because I do not understand it and don't want to interfere with any security investigation they might need to make. I opened the account as a trial and never used it (I ended up going with DigitalOcean). Amazon requires you to link a card to the account to open one.
I have never used my AWS account and all activity (billable and non-billable) is related to the attack.
After opening a ticket, I received a response saying it would go to the Specialized Security Team. Great.
I received a reply 5 days later saying they are still working on it. Then, 7 days after that, I receive a message telling me to apply 2FA and to reset my password. I already stated in the initial ticket I had done so. The message also included more than a dozen documentation links to generic information about how to delete resources from the account but it did not request me to delete them myself. It seemed implied, but I do not want to interfere with an investigation, and I also do not understand AWS.
I replied 1 day later stating the above, that I already configured 2FA, that all activity is malicious because the account was inactive, and that I have yet to delete anything to avoid interference or making the situation worse. I asked what I should do as the next steps, and that I would still like to keep my account after the issue is resolved.
1 day later, I receive a reply saying that we should continue with the clean-up process. The same dozen documentation links are copy/pasted, stating to explicitly check for malicious activity in the logs, and then information on how to delete resources. I wasn't asked to delete them myself.
3 days later, I receive another reply saying they identified malicious activity in my account but have been unable to reach me and closed the ticket.
I reopened the ticket and asked, "Is the security team unable to remove the services created by the attacker? If this is the case I'd like to cancel the compromised account."
5 days later I receive a response stating the same copy/pasted answer, but with the first sentence now saying to check the logs for activity and delete them. I posted the reply at the bottom of this post for full context.
The next day, the ticket is closed a second time. At this point, I read the reply and it is clear that they want me to delete the resources myself. I tried deleting the EC2 instances and other components but I'm unsure if I've done a thorough job. In the meantime, the account has accrued another $500 in activity since initially opening the ticket. It appears charges are still accruing.
I reopened the ticket today in a chat and they said they are waiting to hear back from another team again.
Discussion
Look, I'm not an AWS expert. The account appears heavily compromised due to a password leak. I work 50 hours a week and am taking a class part-time, this is already a significant time sink. I don't understand AWS, I work in embedded systems. Initially, it was not clear if I should be doing it myself and I don't want to legally step over a security investigation. I am aware that Amazon has had plenty of lawsuits over similar issues to this, and I realize that there is some mutual responsibility here.
In my opinion, it was my responsibility to keep the account credentials secure, though I would argue that is negligent to allow anything less than 2FA, especially when it requires a credit card and has such high costs associated with abuse. I was unaware of 2FA being an option when I made the trial account.
Now that the damage has been done, I am legitimately confused as to how to undo what the malicious attacker had done to the account. Thankfully, it is not intertwined with any of my own activity or software, so it is clear what is malicious. I do not believe I should be responsible for having to clean up such a complicated system, and I shouldn't have to pay the amount on the balance after the activity was identified by both parties.
Additionally, if they do expect me to remove the software myself, I should be provided with accurate succinct information on how to do so, which clearly is not the case. It is incredibly convoluted and frustrating to me. I am also confused as to why AWS is not interested in investigating the security issue, as there may be a way to trace and find a bitcoin address or something, whatever these servers are doing.
This is a friendly reminder to enable 2FA on all of your accounts. What are your thoughts, /r/aws? Do you agree with their approach to these issues? Should AWS change its policies? Do you have any advice for me to finally get this resolved? If things continue, I'll get a reply in 5 days with the same copy/pasted message, and I need to take an AWS certification just to fix the account.
Thanks.
Appendix - AWS Reply
We are following-up regarding re-securing your account.
As previously communicated, we have detected an abnormal pattern in your AWS account that matches unauthorized activity.
In accordance with the AWS Customer Agreement and/or other agreements with us governing your use of our service, and to protect your account from excessive charges, we may terminate any suspected unauthorized resources on your account and limit your ability to use the AWS service. However, please note that this does not make your account secure.
The following steps are required to secure your AWS account. If you don't take the steps within five days (in accordance with the AWS Customer Agreement), we may suspend your account. Let us know after you complete the steps, and please make sure to leave the support case open.
STEPS TO SECURE YOUR ACCOUNT:
- Step 1: Check your CloudTrail log in each region for unsanctioned activity (such as creation of unauthorized IAM users, AWS access keys, policies, roles, or temporary security credentials) and delete them. See the following for more information:
https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events.html
https://aws.amazon.com/cloudtrail/pricing/
To delete IAM user keys go to your AWS Management Console here: https://console.aws.amazon.com/iam/home#users
To delete Root User Keys go here: https://console.aws.amazon.com/iam/home#security_credential
If your application uses any exposed or compromised access key, you need to replace the key. To replace the key, first create a second key (at that point both keys will be active) and then modify your application to use the new key. Then disable (but do not delete) the exposed key by clicking on the “Make inactive” option in the console. If there are any problems with your application, you can reactivate the exposed key. When your application is fully functional using the new key, please delete the exposed key. See the following for additional information:
https://aws.amazon.com/premiumsupport/knowledge-center/delete-access-key/
To delete unauthorized IAM Users, go here: https://console.aws.amazon.com/iam/home#users
To delete unauthorized roles go here: https://console.aws.amazon.com/iam/home#/roles
To revoke temporary credentials, see the following:
Temporary credentials can also be revoked by deleting the IAM User.
NOTE: Deleting IAM users may impact production workloads and should be done with care. You cannot revoke temporary credentials obtained via the root user, see the following for more information:
Please take steps to prevent any new credentials from being publicly exposed. See Best Practices of Managing your Access Keys here:
http://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html
-
Step 2: Do the following to check for unauthorized AWS usage:
-
Check your CloudTrail logs
-
Check your “Bills” page: https://console.aws.amazon.com/billing/home#/bill
-
Check Cost Explorer: https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/ce-what-is.html
Some common usage types to check for are EC2 instances, EC2 Spot bids, Lambda functions, AMIs, EBS volumes, EBS snapshots, Lightsail instances, and Sagemaker notebook instances. Here’s some helpful documentation on deleting resources associated with those services:
EC2 instances: https://aws.amazon.com/premiumsupport/knowledge-center/delete-terminate-ec2/
EC2 Spot bids: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-requests.html#terminating-a-spot-instance
Lambda functions: https://docs.aws.amazon.com/lambda/latest/dg/API_DeleteFunction.html
AMIs: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/deregister-ami.html
EBS volumes: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-deleting-volume.html
EBS snapshots: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-deleting-snapshot.html#ebs-delete-snapshot
Lightsail instances: https://lightsail.aws.amazon.com/ls/docs/en_us/articles/delete-an-amazon-lightsail-instance
Sagemaker notebook instances: https://docs.aws.amazon.com/sagemaker/latest/dg/ex1-cleanup.html
For information on how to delete a resource associated with other AWS services, please see our documentation for the specific service on the following page:
https://docs.aws.amazon.com/index.html#lang/en_us
Keep in mind that unauthorized usage can occur in any region and that your console may show you only one region at a time. To switch between regions, you can use the dropdown in the top-right corner of the console screen.
Also, please make sure that no termination protection or any other back-up restoration system such as ELB or AutoScaling groups is enabled on the resources you want to delete. More information can be found here:
https://aws.amazon.com/premiumsupport/knowledge-center/ec2-instance-relaunched-after-termination/
https://aws.amazon.com/premiumsupport/knowledge-center/source-ec2-instances/
We recommend that you enable amazon GuardDuty:
Amazon GuardDuty is an AWS threat detection service that helps you continuously monitor and protect your AWS accounts and workloads. Enabling Amazon GuardDuty on your accounts gives you further visibility into malicious or unauthorized activity, alerting you to take action in order to reduce the risk of harm. To learn more, visit:
https://aws.amazon.com/guardduty
Let us know after you complete the steps to secure your account. Please make sure to leave the support case open.
If you require guidance with this process, please notify us by replying to this case in your Support Center. For additional information on re-securing your account see:
https://aws.amazon.com/premiumsupport/knowledge-center/potential-account-compromise
If you prefer to speak directly with an agent by phone, please find this case in your Support Center (https://console.aws.amazon.com/support/home ), then click the "Phone" button and we will be in touch as soon as possible.
We value your feedback. Please share your experience by rating this correspondence using the AWS Support Center link at the end of this correspondence. Each correspondence can also be rated by selecting the stars in top right corner of each correspondence within the AWS Support Center.
AWS Support: https://aws.amazon.com/premiumsupport/knowledge-center/
AWS Documentation: https://docs.aws.amazon.com/
AWS Cost Management: https://aws.amazon.com/aws-cost-management/
AWS Training: http://aws.amazon.com/training/
AWS Managed Services: https://aws.amazon.com/managed-services/
No comments:
Post a Comment