evannuil / aws-snapshot-tool

A Python/Boto script that automates snapshot making of EBS volumes and rotating.

aws-snapshot-tool

aws-snapshot-tool is a python script to make it easy to roll snapshot of your EBS volumes.

Simply add a tag to each volume you want snapshots of, configure and install a cronjob for aws-snapshot-tool and you are off. It will even handle rolling snapshots on a day, week and year so that you can setup the retention policy to suit.

Features:

• Python based: Leverages boto and is easy to configure and install as a crontab
• Simple tag system: Just add a tag to each of your EBS volumes you want snapshots of
• Configure retention policy: Configure how many days, weeks, and month snapshots you want to retain
• SNS Notifications: aws-snapshot-tool works with Amazon SNS our of the box, so you can be notified of snapshots

Usage

1. Install and configure Python and Boto (See: https://github.com/boto/boto)

2. Create a SNS topic in AWS and copy the ARN into the config file

3. Subscribe with a email address to the SNS topic

4. Create a snapshot user in IAM and put the key and secret in the config file

5. Create a security policy for this user (see the iam.policy.sample)

6. Copy config.sample to config.py

7. Decide how many versions of the snapshots you want for day/week/month and change this in config.py

8. Change the Region and Endpoint for AWS in the config.py file

9. Optionally specify a proxy if you need to, otherwise set it to '' in the config.py

10. Give every Volume for which you want snapshots a Tag with a Key and a Value and put these in the config file. Default: "MakeSnapshot" and the value "True"

11. Install the script in the cron:

    # chmod +x makesnapshots.py
    # crontab -e
    30 1 * * 1-5 /opt/aws-snapshot-tool/makesnapshots.py day
    30 2 * * 6 /opt/aws-snapshot-tool/makesnapshots.py week
    30 3 1 * * /opt/aws-snapshot-tool/makesnapshots.py month
    

Additional Notes

The user that executes the script needs the following policies: see iam.policy.sample

aws-snapshot-tool's Issues

IAM Policy needs version

Add on line 2:

"Version": "2012-10-17",

Copying from config.sample to config.py

I am unable to copy the sample into the config.py file.

Whenever I attempt to edit the file, I receive this error message:
"You don’t own the file “config.py” and don’t have permission to write to it" and "You can duplicate
this document and edit the duplicate. Only the duplicate will include your changes."

Any suggestions on how to resolve this? Much appreciated in advance.

EBS Snapshot Archive Support to Reduce AWS Costs

Per https://aws.amazon.com/blogs/aws/new-amazon-ebs-snapshots-archive/, AWS introduced an "archive" status for snapshots in November of last year (also see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/snapshot-archive.html for more details).

https://aws.amazon.com/ebs/pricing/ then shows the following:

• Standard Snapshot
  • $0.05/GB-month
  • $0 per GB of data retrieved
• Archived Snapshot
  • $0.0125/GB-month
  • $0.03 per GB of data retrieved

Having this tool offer an option to make it so snapshots are given the "archive" status could make things cost 1/4th as much as it otherwise would for situations where these snapshots are made as a precautionary measure while actually restoring from a snapshot is rare (or otherwise is infrequent enough where the archive restore cost is still cheaper than the higher ongoing cost it would otherwise incur.)

Example Scenarios

For example, one might have something like 16.76TB worth of rolling EBS snapshots for server backups via this tool. However, the need to restore from one of these snapshots is really rare. That's $838/mo when left as normal snapshots while it could otherwise be $209.50/mo with them being archived. To do a full restore of absolutely everything is then $502.80 on-demand per the restoring from archive cost which is still less than the standard snapshot cost would be for the month ($712.30 for the mo cost + on-demand restore cost). However, these are rolling backups so that's actually spread across 48 snapshots where only one of them would need to be restored so the restore cost is likely somewhere around $10.48 (then $219.98 for mo cost + restore cost) which is then muuuch less than the standard monthly pricing (potentially saving around $618.02 to $125.70 per month [or even $628.50/mo if no snapshots needed to be restored that month.])

For an additional example, even when there's less data in snapshots like 3.132TB... that's $156.60/mo for standard snapshots that could otherwise be $38.84/mo with archived snapshots that would then cost $93.96 to restore everything (while just restoring 1 of 48 snapshots would be $1.96.) That's a potential savings of $117.76 (no snapshots needed to be restored during that period) to $23.80 (if every snapshot was restored once during that period) per month.

Recap

It just rarely, if ever, crosses that threshold in these use cases where the cost of constantly restoring from backups would make it so standard snapshots would be more cost-effective than archived snapshots.

I'm kinda surprised this hasn't been brought up sooner since it's just not as cost-effective as it could be (potentially leaving hundreds of dollars per month on the table) considering the new (as of late last year, anyway) for many rolling backup scenarios where restoring is often infrequent (or even none at all for months at a time.)

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/working-with-snapshot-archiving.html does appear to outline how one could go about having snapshots be converted to archive once they're created which I'm hopeful this can utilize.

Otherwise, I'm definitely curious what alternatives other people use/recommend for creating rolling snapshots like this does that then supports archived snapshots.

Script complaining about boto.sns.get_region() method call

I'm having problems executing the script, facing this issue:

$ python makesnapshots.py day
Traceback (most recent call last):
  File "makesnapshots.py", line 70, in <module>
    sns_region = boto.sns.get_region(ec2_region_name)
AttributeError: 'module' object has no attribute 'get_region'

I'm guessing maybe the boto.sns API has changed? As a result the script might need to be updated?

I can disable the SNS functionality in the python script for now, but worried that it won't notify us if there is successful/unsuccessful snapshots.

Let me know if there are certain commands for me to run to expose more detail about this environment.

Thanks
Cam

Add support for specifying config file from command line?

Ideally I would like to be able to backup multiple regions using the same unix account, so if I could specify region and credentials on the command line, I think it would work. WDYT? Thanks, L

AttributeError: 'NoneType' object has no attribute 'publish' when sending SNS message

I got the following trackback error when the script tries to send an SNS message 👍

Traceback (most recent call last):
File "./makesnapshots.py", line 181, in
sns.publish(mytopic_arn,message,"Finished AWS snapshotting")
AttributeError: 'NoneType' object has no attribute 'publish'

This was resolved by putting the following line right before the script sends messages :

sns = boto.connect_sns()

Should use instance role permissions instead of access keys

it'd be great if we didn't need to provide access key's in the config, and instead utilise amazons ec2 instance profile - http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html

Hourly snapshot support?

Just read about gitlab incident which reminded me our backups have daily-granularity. Would you please consider adding hourly?

Only 1 instance takes a snapshot when 4 instances are tagged and running

Key= MakeSnapshot
Value= True
All are in the same aws VPC

Crontab entry for monthly

Not a bug really. Just wanted to let you know that the cron entry for the monthly cron will actually run on days 1-7 of the month in parallel with the daily run due to this strange behavior in cron:

From man 5 crontab:

Note: The day of a command’s execution can be specified by two fields — day of > month, and day of week. If both fields are restricted (ie, aren’t *), the command will be run when either field matches the current time. For
example,
"30 4 1,15 * 5" would cause a command to be run at 4:30 am on the 1st and 15th of each month, plus every Friday.

I was able to overcome this with the following:

30 3 * * 1-5 /opt/aws-snapshot-tool/makesnapshots.py day
30 3 * * 6 /opt/aws-snapshot-tool/makesnapshots.py week
30 3 1-7 * * [ $(date +%u) -eq 0 ] && /opt/aws-snapshot-tool/makesnapshots.py month

Excellent script though, thanks!

How to ignore the deletion of snapshot which is in use by AMI

Hi,

We have couple of snapshots of EBS volumes which are in use by some AMIs. How we can make this snapshot tool to ignore these snapshots for deletion. Through SNS we get the below error message

Error in processing volumes: Error in processing volume with id:xxxxxxx

An option like this if can be configured in the config.py so that these snapshot ids can be ignored for deletion.

'ignore_snapshot_id: xxxxxx'

Thanks & Regards

Ankush Grover

Script freezes

Can anyone help me? I'm trying to run automated snapshots of an EC2 instance volume.

I'm following the guide at [http://www.coresoftwaregroup.com/blog/automated-amazon-ebs-volume-snapshots-with-boto] and I'm pretty sure I got all the config info correct. When I run it on the command line using "python makesnapshots.py day", it hangs. When I hit Ctrl-C to interrupt it, this is what I see (pasted below). The logfile for this script just says it started but doesn't get past that. What could it be?

ubuntu@ip-10-28-249-183:~/aws-snapshot-tool$ sudo python makesnapshots.py day
^CTraceback (most recent call last):
File "makesnapshots.py", line 100, in
vols = conn.get_all_volumes(filters={config['tag_name']: config['tag_value']})
File "/usr/lib/python2.7/dist-packages/boto/ec2/connection.py", line 1309, in get_all_volumes
[('item', Volume)], verb='POST')
File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 882, in get_list
response = self.make_request(action, params, path, verb)
File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 868, in make_request
return self._mexe(http_request)
File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 785, in _mexe
time.sleep(next_sleep)
KeyboardInterrupt

Appreciate anyone who can help !!

boto3 rewrite

Legacy boto broke with the introduction of public snapshots. I patched it, one thing led to another,
it ended up with boto3 rewrite of the script, Lambda compatibility etc: https://github.com/strofimovsky/aws-rolling-snaps

I'd be happy to commit it somewhere (I tried to keep compatibility to the maximum) or you can just link to the new repo. Whatever you think is more appropriate.

Thanks for the great useful tool!

Config File Error

Line 12 should read: 'tag_name': 'MakeSnapshot',

Suggestion

Hello,
thanks for this good tool, I would like to suggest you to update the config.sample file because the Region Endpoint has changed as you can read here:

https://forums.aws.amazon.com/message.jspa?messageID=427009

[...]
Switching from
RegionInfo(name='eu-west-1', endpoint='eu-west-1.ec2.amazonaws.com')

to:
RegionInfo(name='eu-west-1', endpoint='ec2.eu-west-1.amazonaws.com')
[...]

Best regards