evannuil / aws-snapshot-tool Goto Github PK
View Code? Open in Web Editor NEWA Python/Boto script that automates snapshot making of EBS volumes and rotating.
A Python/Boto script that automates snapshot making of EBS volumes and rotating.
Hello,
thanks for this good tool, I would like to suggest you to update the config.sample file because the Region Endpoint has changed as you can read here:
https://forums.aws.amazon.com/message.jspa?messageID=427009
[...]
Switching from
RegionInfo(name='eu-west-1', endpoint='eu-west-1.ec2.amazonaws.com')
to:
RegionInfo(name='eu-west-1', endpoint='ec2.eu-west-1.amazonaws.com')
[...]
Best regards
I'm having problems executing the script, facing this issue:
$ python makesnapshots.py day
Traceback (most recent call last):
File "makesnapshots.py", line 70, in <module>
sns_region = boto.sns.get_region(ec2_region_name)
AttributeError: 'module' object has no attribute 'get_region'
I'm guessing maybe the boto.sns API has changed? As a result the script might need to be updated?
I can disable the SNS functionality in the python script for now, but worried that it won't notify us if there is successful/unsuccessful snapshots.
Let me know if there are certain commands for me to run to expose more detail about this environment.
Thanks
Cam
Per https://aws.amazon.com/blogs/aws/new-amazon-ebs-snapshots-archive/, AWS introduced an "archive" status for snapshots in November of last year (also see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/snapshot-archive.html for more details).
https://aws.amazon.com/ebs/pricing/ then shows the following:
Having this tool offer an option to make it so snapshots are given the "archive" status could make things cost 1/4th as much as it otherwise would for situations where these snapshots are made as a precautionary measure while actually restoring from a snapshot is rare (or otherwise is infrequent enough where the archive restore cost is still cheaper than the higher ongoing cost it would otherwise incur.)
For example, one might have something like 16.76TB worth of rolling EBS snapshots for server backups via this tool. However, the need to restore from one of these snapshots is really rare. That's $838/mo when left as normal snapshots while it could otherwise be $209.50/mo with them being archived. To do a full restore of absolutely everything is then $502.80 on-demand per the restoring from archive cost which is still less than the standard snapshot cost would be for the month ($712.30 for the mo cost + on-demand restore cost). However, these are rolling backups so that's actually spread across 48 snapshots where only one of them would need to be restored so the restore cost is likely somewhere around $10.48 (then $219.98 for mo cost + restore cost) which is then muuuch less than the standard monthly pricing (potentially saving around $618.02 to $125.70 per month [or even $628.50/mo if no snapshots needed to be restored that month.])
For an additional example, even when there's less data in snapshots like 3.132TB... that's $156.60/mo for standard snapshots that could otherwise be $38.84/mo with archived snapshots that would then cost $93.96 to restore everything (while just restoring 1 of 48 snapshots would be $1.96.) That's a potential savings of $117.76 (no snapshots needed to be restored during that period) to $23.80 (if every snapshot was restored once during that period) per month.
It just rarely, if ever, crosses that threshold in these use cases where the cost of constantly restoring from backups would make it so standard snapshots would be more cost-effective than archived snapshots.
I'm kinda surprised this hasn't been brought up sooner since it's just not as cost-effective as it could be (potentially leaving hundreds of dollars per month on the table) considering the new (as of late last year, anyway) for many rolling backup scenarios where restoring is often infrequent (or even none at all for months at a time.)
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/working-with-snapshot-archiving.html does appear to outline how one could go about having snapshots be converted to archive once they're created which I'm hopeful this can utilize.
Otherwise, I'm definitely curious what alternatives other people use/recommend for creating rolling snapshots like this does that then supports archived snapshots.
I am unable to copy the sample into the config.py file.
Whenever I attempt to edit the file, I receive this error message:
"You don’t own the file “config.py” and don’t have permission to write to it" and "You can duplicate
this document and edit the duplicate. Only the duplicate will include your changes."
Any suggestions on how to resolve this? Much appreciated in advance.
Ideally I would like to be able to backup multiple regions using the same unix account, so if I could specify region and credentials on the command line, I think it would work. WDYT? Thanks, L
Hi,
We have couple of snapshots of EBS volumes which are in use by some AMIs. How we can make this snapshot tool to ignore these snapshots for deletion. Through SNS we get the below error message
Error in processing volumes: Error in processing volume with id:xxxxxxx
An option like this if can be configured in the config.py so that these snapshot ids can be ignored for deletion.
'ignore_snapshot_id: xxxxxx'
Thanks & Regards
Ankush Grover
Key= MakeSnapshot
Value= True
All are in the same aws VPC
Can anyone help me? I'm trying to run automated snapshots of an EC2 instance volume.
I'm following the guide at [http://www.coresoftwaregroup.com/blog/automated-amazon-ebs-volume-snapshots-with-boto] and I'm pretty sure I got all the config info correct. When I run it on the command line using "python makesnapshots.py day", it hangs. When I hit Ctrl-C to interrupt it, this is what I see (pasted below). The logfile for this script just says it started but doesn't get past that. What could it be?
ubuntu@ip-10-28-249-183:~/aws-snapshot-tool$ sudo python makesnapshots.py day
^CTraceback (most recent call last):
File "makesnapshots.py", line 100, in
vols = conn.get_all_volumes(filters={config['tag_name']: config['tag_value']})
File "/usr/lib/python2.7/dist-packages/boto/ec2/connection.py", line 1309, in get_all_volumes
[('item', Volume)], verb='POST')
File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 882, in get_list
response = self.make_request(action, params, path, verb)
File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 868, in make_request
return self._mexe(http_request)
File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 785, in _mexe
time.sleep(next_sleep)
KeyboardInterrupt
Appreciate anyone who can help !!
I got the following trackback error when the script tries to send an SNS message 👍
Traceback (most recent call last):
File "./makesnapshots.py", line 181, in
sns.publish(mytopic_arn,message,"Finished AWS snapshotting")
AttributeError: 'NoneType' object has no attribute 'publish'
This was resolved by putting the following line right before the script sends messages :
sns = boto.connect_sns()
Just read about gitlab incident which reminded me our backups have daily-granularity. Would you please consider adding hourly?
Not a bug really. Just wanted to let you know that the cron entry for the monthly cron will actually run on days 1-7 of the month in parallel with the daily run due to this strange behavior in cron:
From man 5 crontab
:
Note: The day of a command’s execution can be specified by two fields — day of > month, and day of week. If both fields are restricted (ie, aren’t *), the command will be run when either field matches the current time. For
example,
"30 4 1,15 * 5" would cause a command to be run at 4:30 am on the 1st and 15th of each month, plus every Friday.
I was able to overcome this with the following:
30 3 * * 1-5 /opt/aws-snapshot-tool/makesnapshots.py day
30 3 * * 6 /opt/aws-snapshot-tool/makesnapshots.py week
30 3 1-7 * * [ $(date +%u) -eq 0 ] && /opt/aws-snapshot-tool/makesnapshots.py month
Excellent script though, thanks!
Line 12 should read: 'tag_name': 'MakeSnapshot',
it'd be great if we didn't need to provide access key's in the config, and instead utilise amazons ec2 instance profile - http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
Legacy boto broke with the introduction of public snapshots. I patched it, one thing led to another,
it ended up with boto3 rewrite of the script, Lambda compatibility etc: https://github.com/strofimovsky/aws-rolling-snaps
I'd be happy to commit it somewhere (I tried to keep compatibility to the maximum) or you can just link to the new repo. Whatever you think is more appropriate.
Thanks for the great useful tool!
Add on line 2:
"Version": "2012-10-17",
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.