This is more of a question than an issue. How will your script handle file differences? Say I have a bunch of files, I sync them to glacier. Then a couple of the files change, and I run a sync again. Will it complain about files failing to match hash, or will it not see that they have changed?

Hi,

This Perl implementation module is working great - (This is not a issue list, but an feature requests :) )

a). I am able to upload the files inside the directory.
b). Create download request for the files (gave max-file=1 )
c). After 4hrs was able to download the file.

but just curious,

a). Is it possible to download individual archives? by name or some other way?
b). Delete individual files ( by file name or initiating the request for download )
c). List the archives per vault
d). Create/Delete vault
e). Handle error message properly, i.e if i run script without any options/short of options it just dies.

Just a request, but not issues list.

Thanks.

I would love to dump some of our companies old data into glacier.
But I can not do that unencrypted.

This tool seems to solve a lot of my problems. But plaintext is a nogo for me. Any chance encryption could be added?

Trying to upload a 1.5 TB tar archive with the following options:

--concurrency=10 --partsize=256

Kicks off child pids, transfer begins and runs for ~15 minutes, then fails:

MT-AWS-Glacier, part of MT-AWS suite, Copyright (c) 2012 Victor Efimov http://mt-aws.com/ Version 0.72beta
PID 10914 Started worker
PID 10915 Started worker
PID 10916 Started worker
PID 10917 Started worker
PID 10918 Started worker
PID 10919 Started worker
PID 10920 Started worker
PID 10921 Started worker
PID 10922 Started worker
PID 10923 Started worker
PID 10920 Created an upload_id mms0SKw4fYwRk7OrlOR1XBNYmm2uaGd0U25jaghYatulvQzL1y7t_xPatB25dZTOGN-5hDej7iIrz04iQVwSaoDmONTG
PID 10920 Uploaded part for /media/fppp/outgoing/xxx.tar at offset [2415919104]
PID 10916 Uploaded part for /media/fppp/outgoing/xxxb.tar at offset [0]
PID 10923 Uploaded part for /media/fppp/outgoing/xxx.tar at offset [268435456]
PID 10919 Uploaded part for /media/fppp/outgoing/xxx.tar at offset [536870912]
PID 10917 Uploaded part for /media/fppp/outgoing/xxx.tar at offset [805306368]
PID 10917 HTTP 500 This might be normal. Will retry
PID 10915 Uploaded part for /media/fppp/outgoing/xxx.tar at offset [1879048192]
PID 10918 Uploaded part for /media/fppp/outgoing/xxx.tar at offset [1610612736]
PID 10922 Uploaded part for /media/fppp/outgoing/xxx.tar at offset [1073741824]
PID 10921 Uploaded part for /media/fppp/outgoing/xxx.tar at offset [1342177280]
PID 10914 Uploaded part for /media/fppp/outgoing/xxx.tar at offset [2147483648]
PID 10920 Uploaded part for /media/fppp/outgoing/xxx.tar at offset [2684354560]
PID 10919 Uploaded part for /media/fppp/outgoing/xxx.tar at offset [3489660928]
PID 10917 Uploaded part for /media/fppp/outgoing/xxx.tar at offset [3758096384]
PID 10914 Uploaded part for /media/fppp/outgoing/xxxb.tar at offset [5100273664]
PID 10916 Uploaded part for /media/fppp/outgoing/xxx.tar at offset [2952790016]
PID 10923 Uploaded part for /media/fppp/outgoing/xxx.tar at offset [3221225472]
PID 10920 Uploaded part for /media/fppp/outgoing/xxx.tar at offset [5368709120]
PID 10922 Uploaded part for /media/fppp/outgoing/xxx.tar at offset [4563402752]
PID 10919 Uploaded part for /media/fppp/outgoing/xxx.tar at offset [5637144576]
PID 10921 Uploaded part for /media/fppp/outgoing/xxx.tar at offset [4831838208]
PID 10915 HTTP 408 This might be normal. Will retry
PID 10917 HTTP 408 This might be normal. Will retry
Error:
PUT http://glacier.us-east-1.amazonaws.com/-/vaults/video-test/multipart-uploads/mms0SKw4fYwRk7OrlOR1XBNYmm2uaGd0U25jaghYatulvQzL1y7t_xPatB25dZTOGN-5hDej7iIrz04iQVwSaoDmONTG
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJD3CQKG5ENGHYQUA/20121106/us-east-1/glacier/aws4_request, SignedHeaders=content-length;content-range;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-glacier-version;x-amz-sha256-tree-hash, Signature=2b1b637f2131173dbba62c68d8b14830b8bfe29ad2c3ce54e1dd1cf4595fa217
Host: glacier.us-east-1.amazonaws.com
User-Agent: mt-aws-glacier/0.72beta (http://mt-aws.com/) libwww-perl/6.03
Content-Length: 268435456
Content-Range: bytes 4026531840-4294967295/*
Content-Type: application/octet-stream
X-Amz-Content-Sha256: f1c1d14f7b439d9365a4881bc29698912102c6647e8ea8598aeea0188c5a42d5
X-Amz-Date: 20121106T170833Z
X-Amz-Glacier-Version: 2012-06-01
X-Amz-Sha256-Tree-Hash: 4441c69054ad0c84c6673efe877e6928e199b08b98fb99a74354061f429c0043

\xAF\xA8\xA2\xFE\xFF\xAE\xA7\xA1\30\xFF\xAD\xA6\xA1\xFF\xAE\xA6\xA1\xFF\xAF\xA7\xA2\xFF\xAF\xA8\xA2\xFF\xAE\xA7\xA1\xFF\xAD\xA7\xA1\xFF\xAE\xA7\xA1\xFF\xAD\xA8\xA1\xFF\xAE\xA9\xA2\xFF\xAD\xA7\xA2\xFF\xAD\xA6\xA1\xFF\xAE\xA7\xA3\xFF\xAF\xA8\xA4\xFF\xAE\xA7\xA3\xFF\xAD\xA6\xA2\xFF\xAE\xA8\xA4\xFF\xAE\xA7\xA3\xFF\xAE\xA8\xA4\xFF\xAE\xA9\xA5\xFF\xAD\xA8\xA4\xFF\xAC\xA7\xA4\xFF\xAD\xA7\xA4\xFF\xAE\xA8\xA5\xFF\xAE\xA8\xA4\xFB\xFF\xAD\xA8\xA4\xFE\xFF\xAD\xA8\xA3\xFD\xFF\xAD\xA8\xA4\r\xFF\xAD\xA8\xA3\xFF\xAD\xA7\xA3\xFF\xAD\xA8\xA3\xFF\xAE\xA8\xA4\xFF\xAD\xA8\xA4\xFF\xAC\xA8\xA4\xFF\xAC\xA8\xA3\xFF\xAD\xA8\xA3\xFF\xAE\xA9\xA4\xFF\xAF\xA8\xA4\xFF\xAE\xA8\xA4\xFF\xAF\xA8\xA4\xFF\xAE\xA8\xA4\xFE\xFF\xAE\xA7\xA4\xFC\xFF\xAD\xA7\xA3\2\xFF\xAC\xA7\xA2\xFF\xAD\xA7\xA3\xFE\xFF\xAC\xA8\xA3\xFE\xFF\xAD\xA8\xA4\t\xFF\xAE\xA9\xA5\xFF\xAE\xA8\xA5\xFF\xAE\xA7\xA4\xFF\xAE\xA8\xA4\xFF\xAF\xA8\xA5\xFF\xAF\xA8\xA4\xFF\xAE\xA7\xA4\xFF\xAE\xA8\xA4\xFF\xAE\xA7\xA3\xFE\xFF\xAD\xA7\xA3\3\xFF\xAE\xA8\xA4\xFF\xAF\xA8\xA4\xFF\xAE\xA7\xA4\xFE\xFF\xAE\xA7\xA3\1\xFF\xAE\xA7\xA4\xFD\xFF\xAE\xA7\xA3\6\xFF\xAE\xA8\xA4\xFF\xAF\xA8\xA4\xFF\xAF\xA9\xA5\xFF\xAF\xA8\xA5\xFF\xAD\xA8\xA4\xFF\xAD\xA7\xA3\xFC\xFF\xAD\xA8\xA4\1\xFF\xAE\xA8\xA4\xFD\xFF\xAD\xA8\xA3\3\xFF\xAE\xA9\xA4\xFF\xAF\xA9\xA5\xFF\xAD\xA7\xA3\xFE\xFF\xAD\xA8\xA4\10\xFF\xAE\xA8\xA4\xFF\xAD\xA8\xA3\xFF\xAD\xA7\xA3\xFF\xAD\xA8\xA3\xFF\xAE\xA8\xA3\xFF\xAE\xA8\xA4\xFF\xAE\xA9\xA4\xFF\xAD\xA8\xA4\xFE\xFF\xAC\xA7\xA3\5\xFF\xAD\xA7\xA4\xFF\xAD\xA8\xA4\xFF\xAD\xA8\xA5\xFF\xAD\xA7\xA4\xFF\xAD\xA8\xA4\xFE\xFF\xAE\xA8\xA5\xFD\xFF\xAD\xA8\xA4\5\xFF\xAD\xA8\xA3\xFF\xAD\xA8\xA4\xFF\xAE\xA9\xA5\xFF\xAF\xA9\xA5\xFF\xAE\xA9\xA5\xFC\xFF\xAE\xA8\xA4\2\xFF\xAD\xA8\xA4\xFF\xAC\xA7\xA3\xFE\xFF\xAD\xA8\xA4\t\xFF\xAE\xA8\xA4\xFF\xAD\xA8\xA3\xFF\xAD\xA7\xA3\xFF\xAE\xA8\xA4\xFF\xAC\xA7\xA3\xFF\xAD\xA8\xA4\xFF\xAE\xA9\xA4\xFF\xAD\xA8\xA4\xFF\xAD\xA8\xA3\xFD\xFF\xAD\xA8\xA4\xFE\xFF\xAD\xA7\xA5\2\xFF\xAD\xA8\xA6\xFF\xAE\xA8\xA6\xFD\xFF\xAD\xA8\xA6\2\xFF\xAE\xA9\xA7\xFF\xAC\xA7\xA5\xFE\xFF\xAC...
(+ 268434944 more bytes not shown)
HTTP/1.1 403 Forbidden
Date: Tue, 06 Nov 2012 17:25:54 GMT
Content-Length: 165
Content-Type: application/json
Client-Date: Tue, 06 Nov 2012 17:25:54 GMT
Client-Peer: 72.21.195.182:80
Client-Response-Num: 1
X-Amzn-RequestId: liig5Jr4BfYHLFEXowrdjY0mZn176MnqvrOIdhgmRbkZEIQ

{"message":"Signature expired: 20121106T170833Z is now earlier than 20121106T172054Z (20121106T172554Z - 5 min.)","code":"InvalidSignatureException","type":"Client"}

PARENT Exit

On master:HEAD:

A few thousand files in my backup folder to sync. mtglacier ran for about an hour and failed with:

ERROR (parent): Unable to open task file "/path/to/file" for reading, errno=Too many open files

Fails under both single and multiprocess.

The operating system I'm running it on is the latest Raspbian on Raspberry Pi.

EDIT: Add more environment info.

This is an edge case, but probably a common one due to Glaciers pricing policy.
I am using mt-aws-glacier to backup my Aperture archives, which is full of big images and tons of incredibly small metainformation files. Obviously I can use existing archiving tools to get everything to fit into a set of files with any filesize, but this is limiting since I am backing up from the working directory every few weeks - in other words, I would have to regenerate new archive files every time I want to sync to glacier.

Would it be possible to include a pre-archiving functionality in mt-aws-glacier so that all files under XX bytes are zipped together to YY number of XX byte sized archives. With md5 or something similar, which I assume is already used, I would have to upload the archives which have been changed every time I sync, yet the amount of files synced would be considerably smaller.

I may have a go at editing this tomorrow. It's 10.40pm here and I'm not explaining very well...

Hello,

Problem
We need to know what our glacier backup doing every day.
We use PRTG for monitoring our IT infrastructure. Unfortunately it does not have sensor for glacier. But we can create custom sensor and read test files in linux machine with mt-aws-glacier.
Question
Is it possible to get "Inventory Last Updated" from glacier service side via mt-aws-glacier and write it to text file in local machine?
If it is possible it make us very happy!
Thank you!
Alex

I got this last night after having to control-c out of the upload. I rebooted "just in case" and there are no other updates or changes. I am on an unmodified pull of the release branch. What information do you need from me?

perl -v:
This is perl 5, version 12, subversion 4 (v5.12.4) built for darwin-thread-multi-2level

macintosh-2:mt-aws-glacier kelly$ port list installed | grep p5.12
p5.12-encode-locale @1.20.0 perl/p5-encode-locale
p5.12-file-listing @6.20.0 perl/p5-file-listing
p5.12-html-form @6.0.0 perl/p5-html-form
p5.12-html-parser @3.690.0 perl/p5-html-parser
p5.12-html-tagset @3.200.0 perl/p5-html-tagset
p5.12-http-cookies @6.10.0 perl/p5-http-cookies
p5.12-http-daemon @6.10.0 perl/p5-http-daemon
p5.12-http-date @6.20.0 perl/p5-http-date
p5.12-http-message @6.60.0 perl/p5-http-message
p5.12-http-negotiate @6.10.0 perl/p5-http-negotiate
p5.12-io-socket-ssl @1.440.0 perl/p5-io-socket-ssl
p5.12-json-xs @2.230.0 perl/p5-json-xs
p5.12-libwww-perl @6.40.0 perl/p5-libwww-perl
p5.12-lwp-mediatypes @6.10.0 perl/p5-lwp-mediatypes
p5.12-lwp-protocol-https @6.20.0 perl/p5-lwp-protocol-https
p5.12-mime-base64 @3.130.0 perl/p5-mime-base64
p5.12-mozilla-ca @20110904 perl/p5-mozilla-ca
p5.12-net-http @6.10.0 perl/p5-net-http
p5.12-net-libidn @0.120.0 perl/p5-net-libidn
p5.12-net-ssleay @1.480.0 perl/p5-net-ssleay
p5.12-uri @1.590.0 perl/p5-uri
p5.12-www-robotrules @6.10.0 perl/p5-www-robotrules

macintosh-2:mt-aws-glacier me$ git status
On branch master
Untracked files:
(use "git add ..." to include in what will be committed)

config.cfg
nothing added to commit but untracked files present (use "git add" to track)

macintosh-2:mt-aws-glacier me$ ./mtglacier.pl sync --config config.cfg --journal ../Documents/glacier-logs/Joe-Laptop-Backup.log --from-dir ~/Desktop/jd/ --to-vault Joe-Laptop-Backup --concurrency=2
MT-AWS-Glacier, part of MT-AWS suite, Copyright (c) 2012 Victor Efimov http://mt-aws.com/ Version 0.82beta

PID 292 Started worker
PID 293 Started worker
Skip .DS_Store
...snip...
Found 24000 local files
PID 293 Created an upload_id 1BQOjEjLyqnQHdL0ePjyZiSWz5-ZPzgziccrkH1_jbjDXl71-8ksCY_Nd0fvGmfIFeFXCY7_HJIcEX7NnNafZ9sdUQZu
DIE outside EVAL block [0]
Call stack:
ParentWorker::process_task(./mtglacier.pl:131)
ParentWorker::wait_worker(ParentWorker.pm:63)
JobListProxy::finish_task(ParentWorker.pm:84)
JobProxy::finish_task(JobListProxy.pm:80)
FileCreateJob::finish_task(JobProxy.pm:53)
FileUploadJob::new(FileCreateJob.pm:66)
main::ANON(FileUploadJob.pm:38)
Fatal Error: 0 Died at FileUploadJob.pm line 38, line 1.

I've installed mt-aws-glacier from CPAN and I get error when running a query (and other commands) . Previously I'd installed via git and it ran perfectly well, but I haven't tried this update yet.

$ mtglacier retrieve-inventory --config=config.txt --vault=test
String found where operator expected at /usr/local/share/perl5/App/MtAws/ConfigDefinition.pm line 37, near "message 'filter_error'"
(Do you need to predeclare message?)
String found where operator expected at /usr/local/share/perl5/App/MtAws/ConfigDefinition.pm line 38, near "scope 'filters'"
(Do you need to predeclare scope?)
Array found where operator expected at /usr/local/share/perl5/App/MtAws/ConfigDefinition.pm line 42, at end of line
(Do you need to predeclare lists?)
String found where operator expected at /usr/local/share/perl5/App/MtAws/ConfigDefinition.pm line 88, near "message 'mutual'"
(Do you need to predeclare message?)
String found where operator expected at /usr/local/share/perl5/App/MtAws/ConfigDefinition.pm line 89, near "message 'mandatory_with'"
(Do you need to predeclare message?)
String found where operator expected at /usr/local/share/perl5/App/MtAws/ConfigDefinition.pm line 100, near "validate 'dir'"
(Do you need to predeclare validate?)
syntax error at /usr/local/share/perl5/App/MtAws/ConfigDefinition.pm line 37, near "message 'filter_error'"
syntax error at /usr/local/share/perl5/App/MtAws/ConfigDefinition.pm line 42, near "lists @l"
Global symbol "$filter_error" requires explicit package name at /usr/local/share/perl5/App/MtAws/ConfigDefinition.pm line 45.
syntax error at /usr/local/share/perl5/App/MtAws/ConfigDefinition.pm line 45, near "} if"
syntax error at /usr/local/share/perl5/App/MtAws/ConfigDefinition.pm line 48, near "} elsif"
Global symbol "$F" requires explicit package name at /usr/local/share/perl5/App/MtAws/ConfigDefinition.pm line 49.
Global symbol "@l" requires explicit package name at /usr/local/share/perl5/App/MtAws/ConfigDefinition.pm line 54.
Global symbol "$F" requires explicit package name at /usr/local/share/perl5/App/MtAws/ConfigDefinition.pm line 54.
Global symbol "@l" requires explicit package name at /usr/local/share/perl5/App/MtAws/ConfigDefinition.pm line 56.
syntax error at /usr/local/share/perl5/App/MtAws/ConfigDefinition.pm line 100, near "validate 'dir'"
/usr/local/share/perl5/App/MtAws/ConfigDefinition.pm has too many errors.
Compilation failed in require at /usr/local/share/perl5/App/MtAws.pm line 60.
BEGIN failed--compilation aborted at /usr/local/share/perl5/App/MtAws.pm line 60.
Compilation failed in require at /usr/local/bin/mtglacier line 5.
BEGIN failed--compilation aborted at /usr/local/bin/mtglacier line 5.

Please add a Max-Retrieval-Rate option to the restore command so we can protect ourselves from accidentally incurring massive charges for retrieval. For example, if I store 100GB at Glacier in 10GB files and I run the restore command with max-number-of-files=10 (which is a low, reasonable number), I will request the retrieval of my entire 100GB vault. While it only costs me $1.00 a month to store my 100GB, running the above command will cost me $179.70 for the retrieval even if I don't end up downloading the files to my computer. The bandwidth to download the 100GB will only cost me $12 at $0.12/GB, while the retrieval will probably make me cry. This will obviously negate the cost savings over using S3 for example.

Amazon charges us at the maximum retrieval rate we incur during any month (during a 4 hour window) as if we retrieved at that rate for the ENTIRE month (720 hours). Until Amazon adds a retrieval cap on their end (which is probably unlikely), every client of Glacier capable of retrieval should implement this.

Since the journal keeps track of what files we have retrieved and when, it shouldn't be too hard for the script to do a simple calculation and see how much data has been "restored"/"retrieved" in the previous 4 hours and to compare that to our Max Retrieval Rate option. If executing the restore command would violate the Max Retrieval Rate, it should fail completely. We could then continue to try the command until doing so will not violate the rate (if we were scripting it, for example).

For example, let's say I set my Max Retrieval Rate at 3GB/Hr. Not factoring in the free retrieval allowance, this means if I hit this rate, I will be charged 3GB x 0.01 x 720 hours = $21.6 for the month, no matter how much I download for the remainder of the month as long as it's lower than this rate. I'd be much more comfortable knowing my maximum retrieval cost will be $21.6 instead of my entire archive x 0.01 x 720 (less free allowance), which in the case of 100GB, it's $179.70 as I mentioned.

It would also be good to incorporate the free allowance calculation. Per the Amazon FAQ: "You can retrieve up to 5% of your average monthly storage (pro-rated daily) for free each month. For example, if on a given day you have 75 TB of data stored in Amazon Glacier, you can retrieve up to 128 GB of data for free that day (75 terabytes x 5% / 30 days = 128 GB, assuming it is a 30 day month). In this example, 128 GB is your daily free retrieval allowance. Each month, you are only charged a Retrieval Fee if you exceed your daily retrieval allowance.". The journal should know how much data we have in the vault before we "restore", and since this looks like its pro-rated daily, it shouldn't be hard to factor this into a metering system. Perhaps the option Max-Chargable-Retreival-Rate could factor this in (ie, the maximum retrieval rate I'm willing to pay for above and beyond my free allowance rate). Also note the download bandwidth cost is in addition to the "retrieval" costs, which is currently $0.12GB up to 10TB.

If you want to check the math yourself,
http://liangzan.net/aws-glacier-calculator/
http://timewasteblog.com/2012/08/30/amazon-glacier-date-retrieval-charge-explained-with-example/

Here is an example of a client that does this:
http://fastglacier.com/amazon-glacier-smart-retrieval.aspx

I'm new to the tool, but I wanted to figure out if this was a problem with my gentoo.

When I run the tool I get these two warnings:

server mt-aws-glacier # ./mtglacier
Prototype after '@' for App::MtAws::ConfigEngine::validation : @_ at lib/App/MtAws/ConfigEngine.pm line 320.
Prototype after '%' for App::MtAws::ConfigEngine::command : $%;$ at lib/App/MtAws/ConfigEngine.pm line 329.
MT-AWS-Glacier, Copyright 2012-2013 Victor Efimov http://mt-aws.com/ Version 0.88beta

I'm going to try the tool anyway, but if you have advice for how to fix this, I'd appreciate it.

Thanks,

Brian

I'm trying to recreate my journal.log file using the retrieve-inventory command, whey I run:
./mtglacier download-inventory --config=glacier.cfg --vault=Personal --new-journal=new-journal.log

it's giving me the output

DIE outside EVAL block [0]
Call stack:
App::MtAws::main(./mtglacier:5)
App::MtAws::ANON(/Users/michelp/Dropbox/aws/glacier/mt-glacier/lib/App/MtAws.pm:309)
Fatal Error: 0 unexpected end of string while parsing JSON string, at character offset 139389 (before "(end of string)") at /Users/michelp/Dropbox/aws/glacier/mt-glacier/lib/App/MtAws.pm line 309, line 2.

Perl version:

This is perl 5, version 12, subversion 4 (v5.12.4) built for darwin-thread-multi-2level
(with 2 registered patches, see perl -V for more detail)

Thank you

I was just looking at the docs for retrieval (Initiate Job) and download (Get Job Output). And just as I was thinking I'd have to write some custom code for this, you added support for multi-segment download.

The other thing I was thinking I'd have to write some custom code for was the "Ranged Archive Retrieval" feature (documented in http://docs.aws.amazon.com/amazonglacier/latest/dev/api-initiate-job-post.html, see the RetrievalByteRange parameter). Since I don't see an existing issue for this, nor do I see it in your readme for coming-soon or planned-next-version features, I thought I'd try out this issue system and bring it to your attention.

Keep up the good work!

Apologies if this is not the best place for this but I couldn't see any other way to contact you.

I have a scenario where I have uploaded a number of archived backups to numerous vaults in amazon.

E.g. I have a vault for November 2013.

Is there a way/command within mt-aws-glacier to delete only selected files within a vault. E.g. I want to delete all items in the November 2013 vault that were created before the 16th November 2013?

I could not see any delete commands within the documentation other than for deleting the entire vault?

Is this possible at the moment, and if not do you know any other way in which I could achieve this?

many thanks!

Phil

When I upload files using the latest version of mt-aws-glacier [0.87 beta], the file names uploaded are pretty much changed to something else like "File Name: mt2 eyJmaWxlbmFtZSI6InVwZGF0ZSA0L3VwZGF0ZSAyIGZvcm0gSUVHL0Z1bGwvSW5HcmlkX0RhbnNUZXNZZXV4...."

Is that normal?
Can't I get them back in the original filename??

Thanks

I'm trying to download a vault inventory of about 5000 files, which fails with the following error message:

manu@nyx:/Users/manu/…/mt-aws-glacier$./mtglacier download-inventory --config=glacier.cfg --vault=personal --new-journal=personal.log
MT-AWS-Glacier, Copyright 2012-2013 Victor Efimov http://mt-aws.com/ Version 0.942beta

PID 8810 Started worker
PID 8810 Fetched job list for inventory retrieval
PID 8810 Downloaded inventory in JSON format
DIE outside EVAL block [0]
Call stack: 
App::MtAws::main(./mtglacier:5)
App::MtAws::ForkEngine::with_forks(/Users/manu/Downloads/mt-aws-glacier/lib/App/MtAws.pm:371)
App::MtAws::__ANON__(lib/App/MtAws/ForkEngine.pm:54)
App::MtAws::__ANON__(/Users/manu/Downloads/mt-aws-glacier/lib/App/MtAws.pm:348)
Fatal Error: 0 unexpected end of string while parsing JSON string, at character offset 218806 (before "(end of string)") at /Users/manu/Downloads/mt-aws-glacier/lib/App/MtAws.pm line 348, <GEN0> line 2.

Some sort of progress counter, est time left.

More verbosity on startup, when it sits and looks like nothing is happening.

Hi.

Great piece of software, it work good so far, except when trying to delete from vault with "--delete-removed". It gives an error below. How could I debug further?
I says invalid vault, although sync works perfectly with the same command, just without "--delete-removed".

Error:
===REQUEST:
DELETE http://glacier.eu-west-1.amazonaws.com/-/vaults/helena/archives/***REMOVED***
Authorization: AWS4-HMAC-SHA256 Credential=_REMOVED/20130915/eu-west-1/glacier/aws4_request, SignedHeaders=host;x-amz-date;x-amz-glacier-version, Signature=_REMOVED
Host: glacier.eu-west-1.amazonaws.com
User-Agent: mt-aws-glacier/1.051 (http://mt-aws.com/) libwww-perl/6.04
X-Amz-Date: 20130915T082648Z
X-Amz-Glacier-Version: 2012-06-01

===RESPONSE:
HTTP/1.1 400 Bad Request
Date: Sun, 15 Sep 2013 08:27:26 GMT
Content-Length: 142
Content-Type: application/json
Client-Date: Sun, 15 Sep 2013 08:27:26 GMT
Client-Peer: 178.236.7.103:80
Client-Response-Num: 1
X-Amzn-RequestId: x-MLCr04NMP2khb9DcFzgfD5chucO40hpIQ1Dh7cnWk0-zA

{"code":"InvalidParameterValueException","message":"Invalid vault name: arn:aws:glacier:eu-west-1:149083434021:vaults/helena","type":"Client"}

ERROR (child 2838): Unexpected reply from remote server
Error:
===REQUEST:
DELETE http://glacier.eu-west-1.amazonaws.com/-/vaults/helena/archives/***REMOVED***
Authorization: AWS4-HMAC-SHA256 Credential=_REMOVED/20130915/eu-west-1/glacier/aws4_request, SignedHeaders=host;x-amz-date;x-amz-glacier-version, Signature=_REMOVED
Host: glacier.eu-west-1.amazonaws.com
User-Agent: mt-aws-glacier/1.051 (http://mt-aws.com/) libwww-perl/6.04
X-Amz-Date: 20130915T082648Z
X-Amz-Glacier-Version: 2012-06-01

===RESPONSE:
HTTP/1.1 400 Bad Request
Date: Sun, 15 Sep 2013 08:27:26 GMT
Content-Length: 142
Content-Type: application/json
Client-Date: Sun, 15 Sep 2013 08:27:26 GMT
Client-Peer: 178.236.7.103:80
Client-Response-Num: 1
X-Amzn-RequestId: DtvNiP8eGj0b74dYY0qCVCLgOi6dYURh0ulH7NEvXuOPkFA

{"code":"InvalidParameterValueException","message":"Invalid vault name: arn:aws:glacier:eu-west-1:149083434021:vaults/helena","type":"Client"}

Error:
===REQUEST:
DELETE http://glacier.eu-west-1.amazonaws.com/-/vaults/helena/archives/***REMOVED***
Authorization: AWS4-HMAC-SHA256 Credential=_REMOVED/20130915/eu-west-1/glacier/aws4_request, SignedHeaders=host;x-amz-date;x-amz-glacier-version, Signature=_REMOVED
Host: glacier.eu-west-1.amazonaws.com
User-Agent: mt-aws-glacier/1.051 (http://mt-aws.com/) libwww-perl/6.04
X-Amz-Date: 20130915T082648Z
X-Amz-Glacier-Version: 2012-06-01

===RESPONSE:
HTTP/1.1 400 Bad Request
Date: Sun, 15 Sep 2013 08:27:26 GMT
Content-Length: 142
Content-Type: application/json
Client-Date: Sun, 15 Sep 2013 08:27:27 GMT
Client-Peer: 178.236.7.103:80
Client-Response-Num: 1
X-Amzn-RequestId: 5vQE_1nc-kOjEsCE3GvIx-YUMTs1Iwaf-zgFdOo6ctImsA0

{"code":"InvalidParameterValueException","message":"Invalid vault name: arn:aws:glacier:eu-west-1:149083434021:vaults/helena","type":"Client"}

ERROR (child 2835): Unexpected reply from remote server
Error:
ERROR (child 2836): Unexpected reply from remote server
===REQUEST:
DELETE http://glacier.eu-west-1.amazonaws.com/-/vaults/helena/archives/l***REMOVED***
Authorization: AWS4-HMAC-SHA256 Credential=_REMOVED/20130915/eu-west-1/glacier/aws4_request, SignedHeaders=host;x-amz-date;x-amz-glacier-version, Signature=_REMOVED
Host: glacier.eu-west-1.amazonaws.com
User-Agent: mt-aws-glacier/1.051 (http://mt-aws.com/) libwww-perl/6.04
X-Amz-Date: 20130915T082648Z
X-Amz-Glacier-Version: 2012-06-01

===RESPONSE:
HTTP/1.1 400 Bad Request
Date: Sun, 15 Sep 2013 08:27:26 GMT
Content-Length: 142
Content-Type: application/json
Client-Date: Sun, 15 Sep 2013 08:27:27 GMT
Client-Peer: 178.236.7.103:80
Client-Response-Num: 1
X-Amzn-RequestId: 9b81WYeLeMcaFEA0FFNROsfHpbo0U92P27NF6INXpLc5Hyk

{"code":"InvalidParameterValueException","message":"Invalid vault name: arn:aws:glacier:eu-west-1:149083434021:vaults/helena","type":"Client"}

ERROR (child 2837): Unexpected reply from remote server
EXIT eof/error when communicate with child process

EXIT on SIGCHLD

Hello,
it's working quite well, however, on certain directories, I get a "does not map to unicode" error as below, any idea how to fix ?

Thanks

Call stack:
App::MtAws::main(/root/mt-aws-glacier/./mtglacier:5)
App::MtAws::Journal::read_new_files(/root/mt-aws-glacier/lib/App/MtAws.pm:117)
App::MtAws::Journal::_read_files(/root/mt-aws-glacier/lib/App/MtAws/Journal.pm:203)
File::Find::find(/root/mt-aws-glacier/lib/App/MtAws/Journal.pm:244)
File::Find::_find_opt(/usr/share/perl/5.10/File/Find.pm:1297)
File::Find::_find_dir(/usr/share/perl/5.10/File/Find.pm:722)
App::MtAws::Journal::ANON(/usr/share/perl/5.10/File/Find.pm:902)
Encode::decode(/root/mt-aws-glacier/lib/App/MtAws/Journal.pm:243)
App::MtAws::ANON(/usr/lib/perl/5.10/Encode.pm:174)
Fatal Error: 0 utf8 "\xA0" does not map to Unicode at /usr/lib/perl/5.10/Encode.pm line 174.

Hi,

Fantastic work!! I hope Google are paying you lots!! :-)

Am using 1.059, Perl 5.10 on a QNAP NAS. It's an ARM so no very powerful, and have reduced concurrency to 2 and partsize to 8MB (on a 3Mb/s upload connection).

The sync fails with SIGCHLD (returned from a shell script) before completion every 100s/1000s files - when I restart it just carries on as normal. Sometimes there are 400/408 errors, sometimes not. How can I get more detailed info regarding what is causing this issue??

Have recommended your software on QNAP fourms!

Thanks, H.

Keeps crashing after a while of use with PARENT EXIT. how can i help u debug ?

With low bandwidth and a large backup is there any mechanism to detect already running if starting from a cron job ?

Also if not contradictory is there any option to periodically mirror local deletions to the vault ?

Not certain if this is an mt-aws-glacier issue or an Amazon Glacier issue.

I issued the purge-vault command, and got this error:

HTTP/1.1 400 Bad Request
Date: Sat, 20 Apr 2013 15:55:17 GMT
Content-Length: 142
Content-Type: application/json
Client-Date: Sat, 20 Apr 2013 15:55:17 GMT
Client-Peer: 204.246.163.151:80
Client-Response-Num: 1
X-Amzn-RequestId: VzMhvWr63bsqtDaRYGLbQ38v5fXK8rcwP5lIhtHPUCW0SkQ

{"code":"InvalidParameterValueException","message":"Invalid vault name: arn:aws:glacier:us-west-1:954508486505:vaults/Videos","type":"Client"}

PARENT Exit
DELETE http://glacier.us-west-1.amazonaws.com/-/vaults/Videos/archives/wLl1mzek-1O4jlgVGS_B7WMY2Ailw1-YQZvVioOmyipQlbgWJzihNtvQccrBe8cmVauJcGNyIzPxY_FZduDKtXyRmgwFMOTlBgECjNpuc5METK1M5ymK1q3dm9lP97C7Gh5jSQY2NQ
Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXXXXXXXXXXXX/20130420/us-west-1/glacier/aws4_request, SignedHeaders=host;x-amz-date;x-amz-glacier-version, Signature=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Host: glacier.us-west-1.amazonaws.com
User-Agent: mt-aws-glacier/0.942beta (http://mt-aws.com/) libwww-perl/5.836
X-Amz-Date: 20130420T155515Z
X-Amz-Glacier-Version: 2012-06-01

It looks to me, though, that the name "arn:aws:glacier:us-west-1:954508486505:vaults/Videos" is truly valid, as I look at my glacier account.

Hi There,

I'm getting this strange error with credential issues. I don't know why as my key and secret codes are fine already tested them on different glacier uploader.

I put my credentials in glacier.cfg file like manual said.

key=1111111/222222/333333/44444444444/5
secret=xxxxxxxxxxxxxxxxxxxxxxxx

Any suggestions would be very useful.

Below log file.

MT-AWS-Glacier, Copyright 2012-2013 Victor Efimov http://mt-aws.com/ Version 0.86beta

PID 16829 Started worker
PID 16830 Started worker
PID 16831 Started worker
Error:
POST https://glacier.eu-west-1.amazonaws.com/-/vaults/test1/multipart-uploads
Authorization: AWS4-HMAC-SHA256 Credential=trXVr9de6W/ixgoCtC/L14hhJZ/WC5hIOCbc26/7/20130204/eu-west-1/glacier/aws4_request, SignedHeaders=host;x-amz-archive-description;x-amz-date;x-amz-glacier-version;x-am
z-part-size, Signature=e2b76702376e0a3fec67160f792b1261a40eed1077bddadd923a816
Host: glacier.eu-west-1.amazonaws.com
User-Agent: mt-aws-glacier/0.86beta (http://mt-aws.com/) libwww-perl/6.03
Content-Length: 0
Content-Type: application/x-www-form-urlencoded
X-Amz-Archive-Description: mt2 eyJmaWxlbmFtZSI6ImRhdl9zd0LCJtdGltZSI6IjIjA0VDAzNTc1MVoifQ
X-Amz-Date: 20130204T120444Z
X-Amz-Glacier-Version: 2012-06-01
X-Amz-Part-Size: 16777216

(no content)
HTTP/1.1 403 Forbidden
Date: Mon, 04 Feb 2013 12:04:44 GMT
Content-Length: 248
Content-Type: application/json
Client-Date: Mon, 04 Feb 2013 12:04:45 GMT
Client-Peer: 178.236.6.231:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3
Client-SSL-Cert-Subject: /C=US/ST=Washington/L=Seattle/O=Amazon.com Inc./CN=glacier.eu-west-1.amazonaws.com
Client-SSL-Cipher: RC4-MD5
Client-SSL-Socket-Class: IO::Socket::SSL
X-Amzn-RequestId: MFSGTawLvs1Lg2dl5o5rriJBc8Ph4nv-BI_nr8DbwQXBuZE

{"message":"Credential must have exactly 5 slash-delimited elements, e.g. keyid/date/region/service/term, got '1111111/222222/333333/44444444444/5/20130204/eu-west-1/glacier/aws4_request'","code":"Incom
pleteSignatureException","type":"Client"}

Hello.

I've been trying without success to run glacier Backup with mt-aws-glacier. My source directory is just a few scripts gathered for testing.

./mtglacier.pl sync --config=/home/pi/glacier.cfg --from-dir /home/pi/test --to-vault=test --journal=/home/pi/journal.log --concurrency=3

All threads seems to get stuck on error 500 :

PID 2794 HTTP 500 This might be normal. Will retry (11 seconds spent for request)
PID 2793 HTTP 500 This might be normal. Will retry (11 seconds spent for request)
PID 2792 HTTP 500 This might be normal. Will retry (11 seconds spent for request)

The unit test fail too :

integration/utf8.t ................ ok
integration/metadata.t ............ Dubious, test returned 2 (wstat 512, 0x200)
Failed 188/188 subtests
integration/utf8_line_protocol.t .. ok
integration/journal_readwrite.t ... Dubious, test returned 2 (wstat 512, 0x200)
Failed 580/580 subtests
unit/journal_readjournal.t ........ Dubious, test returned 2 (wstat 512, 0x200)
Failed 31/31 subtests
unit/journal_writejournal.t ....... Dubious, test returned 2 (wstat 512, 0x200)
Failed 3/3 subtests
unit/journal_readfiles.t .......... Dubious, test returned 2 (wstat 512, 0x200)
Failed 26/26 subtests
unit/journal_sanity.t ............. Dubious, test returned 2 (wstat 512, 0x200)
Failed 82/82 subtests
unit/metadata.t ................... Dubious, test returned 2 (wstat 512, 0x200)
No subtests run
unit/config_engine_parse.t ........ Dubious, test returned 2 (wstat 512, 0x200)
Failed 33/33 subtests
unit/journal_openmodes.t .......... Dubious, test returned 2 (wstat 512, 0x200)
Failed 16/16 subtests
integration/config_engine_v078.t .. Dubious, test returned 2 (wstat 512, 0x200)
Failed 178/178 subtests
unit/u_treehash.t ................. ok
integration/journal.t ............. ok
integration/utf8_journal.t ........ ok
integration/t_treehash.t .......... ok

Test Summary Report

integration/metadata.t          (Wstat: 512 Tests: 0 Failed: 0)
  Non-zero exit status: 2
  Parse errors: Bad plan.  You planned 188 tests but ran 0.
integration/journal_readwrite.t (Wstat: 512 Tests: 0 Failed: 0)
  Non-zero exit status: 2
  Parse errors: Bad plan.  You planned 580 tests but ran 0.
unit/journal_readjournal.t      (Wstat: 512 Tests: 0 Failed: 0)
  Non-zero exit status: 2
  Parse errors: Bad plan.  You planned 31 tests but ran 0.
unit/journal_writejournal.t     (Wstat: 512 Tests: 0 Failed: 0)
  Non-zero exit status: 2
  Parse errors: Bad plan.  You planned 3 tests but ran 0.
unit/journal_readfiles.t        (Wstat: 512 Tests: 0 Failed: 0)
  Non-zero exit status: 2
  Parse errors: Bad plan.  You planned 26 tests but ran 0.
unit/journal_sanity.t           (Wstat: 512 Tests: 0 Failed: 0)
  Non-zero exit status: 2
  Parse errors: Bad plan.  You planned 82 tests but ran 0.
unit/metadata.t                 (Wstat: 512 Tests: 0 Failed: 0)
  Non-zero exit status: 2
  Parse errors: No plan found in TAP output
unit/config_engine_parse.t      (Wstat: 512 Tests: 0 Failed: 0)
  Non-zero exit status: 2
  Parse errors: Bad plan.  You planned 33 tests but ran 0.
unit/journal_openmodes.t        (Wstat: 512 Tests: 0 Failed: 0)
  Non-zero exit status: 2
  Parse errors: Bad plan.  You planned 16 tests but ran 0.
integration/config_engine_v078.t (Wstat: 512 Tests: 0 Failed: 0)
  Non-zero exit status: 2
  Parse errors: Bad plan.  You planned 178 tests but ran 0.
Files=16, Tests=1234, 58 wallclock secs ( 5.01 usr  0.14 sys + 51.63 cusr  0.88 csys = 57.66 CPU)
Result: FAIL

I'm running mt-aws from a raspberry pi (arm computer) under debian wheezy linux.

Any clue about this errors ?

Thank you

Hi,
I am using the release version on a QNAP NAS (ARM based). I uploaded 30GB of jpgs with concurrency 4 and partsize 8, and consistently got 400KB/sec over a 3Mb/sec upload link (i.e. using all bandwidth).

Now I am uploading large GB video files, and using concurrency 1 or 2 and partsize 32. I fail to get over 100KB/s and the CPU is constantly maxed out (which I would expect for hasing but not uploading). Am I doing something wrong?? (I don't have memory issues or other heavy processes running, and I tested that other processes can utilise all the bandwidth).

As a suitable point I will restart and go down to part 8 and con=4 as before to check if it improves things. But this limits the filesize I can upload.

It's STILL great software though (and am still researching my other issue). Thanks, H.

Hi,

We are very happy with your script. We have just 1 question.

Is it possible to delete file in vault in the process of synchronization when this file was deleted in source folder? Our inventory become very large.

Or we should delete all files in vault sometimes and do new sync?

Thank you!

Alex

I get the following error when uploading a 20gb file. It works when I upload a 10GB file.

DIE outside EVAL block [0]
Call stack:
main::process_forks(./mtglacier.pl:83)
ChildWorker::process(./mtglacier.pl:175)
GlacierRequest::finish_multipart_upload(ChildWorker.pm:54)
main::ANON(GlacierRequest.pm:284)
Fatal Error: 0 Can't call method "header" on an undefined value at GlacierRequest.pm line 284, line 961.
DIE outside EVAL block [0]
Call stack:
main::process_forks(./mtglacier.pl:83)
ParentWorker::process_task(./mtglacier.pl:213)
ParentWorker::wait_worker(ParentWorker.pm:29)
main::ANON(ParentWorker.pm:53)
Fatal Error: 0 Unexpeced EOF in Pipe at ParentWorker.pm line 53, line 960.

I am doing this

./mtglacier --help

and I get this output.

MT-AWS-Glacier, Copyright 2012-2013 Victor Efimov http://mt-aws.com/ Version 0.89beta

Use of uninitialized value in concatenation (.) or string at /tmp/mt-aws-glacier/lib/App/MtAws.pm line 111.
Cannot find encoding "" at /tmp/mt-aws-glacier/lib/App/MtAws.pm line 111.
Use of uninitialized value in concatenation (.) or string at /tmp/mt-aws-glacier/lib/App/MtAws.pm line 112.

What can I do?

Using Ubuntu 12.04.2 LTS
This is perl 5, version 14, subversion 2 (v5.14.2) built for x86_64-linux-gnu-thread-multi

Which additional information do you need?

Hi, this tool is great. Is there a reason why it doesn't follow symbolic links on the local filesystem? I'd like to have a "glacier" folder that contains links to large directories on various file systems, and back them all up as one job.

When uploading file content is "0" , error occured "UNEXPECTED ERROR"

UNEXPECTED ERROR (child ***):  at mt-aws-glacier/lib/App/MtAws/GlacierRequest.pm line 104
        App::MtAws::GlacierRequest::upload_part('App::MtAws::GlacierRequest=HASH(0x302ca98)', 'JG-1tB9qhBfv7LjzAB_t5Ww8gR7SQVJxtEXf78AJ0FZgUBTs-Jqw9_Zu_n5XY...', undef, 0, '5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9') called at mt-aws-glacier/lib/App/MtAws/ChildWorker.pm line 80
        App::MtAws::ChildWorker::process('App::MtAws::ChildWorker=HASH(0x290c188)') called at mt-aws-glacier/lib/App/MtAws/ForkEngine.pm line 108
        eval {...} called at mt-aws-glacier/lib/App/MtAws/ForkEngine.pm line 108
        App::MtAws::ForkEngine::start_children('App::MtAws::ForkEngine=HASH(0x2ebf1c0)') called at mt-aws-glacier/lib/App/MtAws/ForkEngine.pm line 58
        App::MtAws::ForkEngine::with_forks(1, 'HASH(0x2f2da48)', 'CODE(0x2f34d38)') called at mt-aws-glacier/lib/App/MtAws/SyncCommand.pm line 210
        App::MtAws::SyncCommand::run('HASH(0x2f2da48)', 'App::MtAws::Journal=HASH(0x2ce2228)') called at mt-aws-glacier/lib/App/MtAws.pm line 164
        App::MtAws::process() called at mt-aws-glacier/lib/App/MtAws.pm line 113
        eval {...} called at mt-aws-glacier/lib/App/MtAws.pm line 113
        App::MtAws::main() called at ./mtglacier line 4

diff --git a/lib/App/MtAws/LineProtocol.pm b/lib/App/MtAws/LineProtocol.pm
index 8bbdab5..f6baa76 100644
--- a/lib/App/MtAws/LineProtocol.pm
+++ b/lib/App/MtAws/LineProtocol.pm
@@ -72,7 +72,7 @@ sub get_data
                        return;
        }
        my $data = decode_data($data_e);
-   return ($pid, $action, $taskid, $data, $attachment ? \$attachment : ());
+ return ($pid, $action, $taskid, $data, defined($attachment) ? \$attachment : ());
 }
 
 sub send_data

On a CentOS 5.8 machine:

[root@host mt-aws-glacier]# perl -cw mtglacier.pl
"make_path" is not exported by the File::Path module
Can't continue after import errors at ChildWorker.pm line 28
BEGIN failed--compilation aborted at ChildWorker.pm line 28.
Compilation failed in require at mtglacier.pl line 35.
BEGIN failed--compilation aborted at mtglacier.pl line 35.

It looks like CentOS boxes come with File::Path v1.08, and make_path is from v2.0. I suspect the answer I'll get it "upgrade File::Path", but I wanted to see if there was a better workaround.

Just trying out 0.8 beta. I gather that the journal data is now being stored as metadata on glacier.

I'm getting this error when running:
./mtglacier.pl sync --config=/root/glacier.cfg --from-dir /mnt/media/video/Clips/ --to-vault=Clips --journal=/root/glacier-journal.log --concurrency=3

"encode_base64url" is not exported by the MIME::Base64 module
"decode_base64url" is not exported by the MIME::Base64 module
Can't continue after import errors at MetaData.pm line 28
BEGIN failed--compilation aborted at MetaData.pm line 28.
Compilation failed in require at GlacierRequest.pm line 31.
BEGIN failed--compilation aborted at GlacierRequest.pm line 31.
Compilation failed in require at ChildWorker.pm line 24.
BEGIN failed--compilation aborted at ChildWorker.pm line 24.
Compilation failed in require at ./mtglacier.pl line 35.
BEGIN failed--compilation aborted at ./mtglacier.pl line 35.

Should I not be specifying the journal on the command line as it's now being stored on glacier? How should I get my existing journal onto glacier, or do I have to reupload everything again?

I am getting this error when using your script. Is this due to IAM setting?

PID 5768 HTTP 500 This might be normal. Will retry

Thanks,
Degging

Errors come from Amazon like: wrong archvie id, wrong vault, wrong region wrong signature should:

1. print user-friendly explanation what's happened (with a relation to mtglacier concept, not just amazon concept)
2. should not raise "unexpected error" messages
3. should not crash main process with unexpected error

Hi @vsespb
First of all, let me say thank you for your tool, it's very good!
Is it possible to add an option to archive data in tar chunks before upload them to glacier?
Create tar files will save glacier requests and will also improve the upload when a directory is full of small files.

Thanks!

Cheers

When are you planning to add support for vault management (create/delete)?

currently I'm using mt-aws-glacier to archive large backups and logs, and eventually I want to have vaults created by calendar month, to ease rotation and purge management. Otherwise, this is pretty epic, thanks :)

Hello,

Am trying to restore over 1.6million files on Glacier,
However, I can only get a bit over 30K files, and then it fails, generally with something like below,
What should I do ?

Thanks

Elie

PID 14058 HTTP 500 This might be normal. Will retry (0 seconds spent for request)
Error:
POST http://glacier.eu-west-1.amazonaws.com/-/vaults/AnghamiM4A/jobs
Authorization: AWS4-HMAC-SHA256 Credential=AKIAIGSL5DQ6CEJV4TIA/20130523/eu-west-1/glacier/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-glacier-version, Signature=60f89f8889a0fd102a78ccc63e1c52c4d74effa3597ea4c60f3f2be104ae5ae3
Host: glacier.eu-west-1.amazonaws.com
User-Agent: mt-aws-glacier/0.953beta (http://mt-aws.com/) libwww-perl/6.04
Content-Length: 191
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Date: 20130523T161940Z
X-Amz-Glacier-Version: 2012-06-01

{
"Type": "archive-retrieval",
"ArchiveId": "gu5BplaYIGo9MpqJyD3E-l1C__wh_cMIAGS_7DqbxNrnWetaM0Dz1HwoNeFuSxrORLYMMiYa1HjlElbERrBiJJHYhnQKHYB5N5wqJM59CSBA8z25Kv-w4gJn_POFesqUwlrElijyTA"
}\n
HTTP/1.1 400 Bad Request
Date: Thu, 23 May 2013 16:19:39 GMT
Content-Length: 72
Content-Type: application/json
Client-Date: Thu, 23 May 2013 16:19:40 GMT
Client-Peer: XWe6lQNDSgo4yATi0aMjWEsbWA
EXIT on SIGUSR1

Running ./mtglacier download-inventory --config=glacier.cfg --vault=text --new-journal=text-journal.log I get the error

"UNEXPECTED ERROR (parent): malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "ArchiveId,ArchiveDes...") at /home/neal/mt-aws-glacier/lib/App/MtAws/Command/DownloadInventory.pm line 61."

using latest version of mt-glacier and JSON::XS

Hi, thanks for the great script. But I got some difficulties with --filter and --exclude. When mtglacier sync run, it said the excluded directory is skipped, but still uploading those directory and their subdirectories/files.

Operating System: FreeBSD 8.2

Here is all command that I tried:
mtglacier sync --config ~/.glacier.cfg --dir /usr/local/etc/nginx --exclude 'cache/' --vault nginx-backup --journal nginx-backup.log --concurency 3

mtglacier sync --config ~/.glacier.cfg --dir /usr/local/etc/nginx --exclude 'cache/**' --vault nginx-backup --journal nginx-backup.log --concurency 3

mtglacier sync --config ~/.glacier.cfg --dir /usr/local/etc/nginx --exclude '/usr/local/etc/nginx/cache/' --vault nginx-backup --journal nginx-backup.log --concurency 3

mtglacier sync --config ~/.glacier.cfg --dir /usr/local/etc/nginx --exclude '/usr/local/etc/nginx/cache/**' --vault nginx-backup --journal nginx-backup.log --concurency 3

mtglacier sync --config ~/.glacier.cfg --dir /usr/local/etc/nginx --filter '-/usr/local/etc/nginx/cache/' --vault nginx-backup --journal nginx-backup.log --concurency 3

mtglacier sync --config ~/.glacier.cfg --dir /usr/local/etc/nginx --filter '-/usr/local/etc/nginx/cache/**' --vault nginx-backup --journal nginx-backup.log --concurency 3

mtglacier sync --config ~/.glacier.cfg --dir /usr/local/etc/nginx --filter '-cache/' --vault nginx-backup --journal nginx-backup.log --concurency 3

When deleting archives from Amazon Glacier, delete only archives older than a certain age (specified in options)

Any plans to support standard input/output as file source/target?

Hi,

I'm trying the newer version (0.75 beta) to download some files and got this signature mismatch error. The older version is working fine for the same command. Note that the files were uploading using the older version.

MT-AWS-Glacier, part of MT-AWS suite, Copyright (c) 2012 Victor Efimov http://mt-aws.com/ Version 0.74beta
PID 30845 Started worker
PID 30846 Started worker
PID 30847 Started worker
PID 30848 Started worker
Error:
POST http://glacier.us-east-1.amazonaws.com/-/vaults/TA/jobs

Signature=a361e4836c5408f87935cc805234ede97ef41ac56182888b5bff0e676752f2ac
Host: glacier.us-east-1.amazonaws.com
User-Agent: mt-aws-glacier/0.74beta (http://mt-aws.com/) libwww-perl/5.833
Content-Length: 191
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Date: 20121116T201753Z
X-Amz-Glacier-Version: 2012-06-01

{
"Type": "archive-retrieval",
"ArchiveId": "1mAFFVTVymMS50YLEhbfKh-czVlg8-rLeKY-H_7bLOL2nh9-Wo9Xv0zvQgt7I40XlBZmCaVo-mkX_Nv-WBHzrur5vzdbZPJwILjDAG9RlA3W3is3R_peimaW3JHuRrUjVCc1_btrug"
}\n
HTTP/1.1 403 Forbidden
Date: Fri, 16 Nov 2012 20:17:53 GMT
Content-Length: 800
Content-Type: application/json
Client-Date: Fri, 16 Nov 2012 20:17:53 GMT
Client-Peer: 72.21.195.182:80
Client-Response-Num: 1
X-Amzn-RequestId: OPJaeYQ8vIKudljdzghJueux53F0WpRKF1YlEnXG-qSIl2Q

{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'POST\n/-/vaults/TA/jobs\n\ncontent-type:application/x-www-form-urlencoded; charset=utf-8\nhost:glacier.us-east-1.amazonaws.com\nx-amz-date:20121116T201753Z\nx-amz-glacier-version:2012-06-01\n\ncontent-type;host;x-amz-date;x-amz-glacier-version\n361acdfa41b281d2...
(+ 288 more bytes not shown)

PARENT Exit

Enhancement: Store multiple versions of same file (same filename) in Glacier.

Allow sync command to upload new versions of files without deleting old. Allow restore/restore-completed commands to restore older versions. Introduce command to remove older versions.

Allow upload-file to upload files if files with same name exist in Journal (it will be treated as new version)

Versioning will be based of file modification time (and file upload time in edge cases).

When i run the script with glacier.cfg, i am getting below message -
command run is - time perl mtglacier.pl sync --config=./glacier.cfg --from-dir backups/ --to-vault=mytestvault --journal=journal.log --concurrency=3
and the response was ,
{"message":"Signature expired: 20120907T094027Z is now earlier than 20120907T094737Z (20120907T095237Z - 5 min.)","code":"InvalidSignatureException","type":"Client"}

Thanks.

Hi Victor,

Thanks for the tool it does the job quite well.

Though I am wondering what the relation of journal to vault is.

In the warnings [1] you mention that the for each vault you should use only one journal file which makes it a bit "harder" to retrieve single or certain "type" of files. Issue #3 [2] mentioned it already and the solution would be to filter out specific files from the journal.

Still I am wondering why it would not be recommended to have a vault and then different journales for each type of archive inside?

e.g.
Vault: home-1
Archive types: pictures-1, pictures-2, pictures-n, docs-1, docs-2, docs-n, code-1, code-2, code-n, etc.

Having a separated journal (pictures, docs, code) for each archive type within the vault would make retrieval/restore easier and would give you a quick overview of files.

Thanks,
Christian

[1] https://github.com/vsespb/mt-aws-glacier#warnings--must-read-
[2] #3

How hard would it be to tell mtglacier to exclude patterns from files to sync (or only include certain patterns).

Like if I want to do --include *.gz or --exclude access_log --exclude error_log.

I just started uploading a big directory with chaotic filenames and got some invalid filenames. It will throw an "Invalid filename"-error, but no hint on which file is affected. In the end I added some print-statements to hunt it down.

Добрый день, Виктор! Прошу извинения, если что-то аглицки напишу не так -- пиетет к англоязычному сообществу требует написания вопроса на этом языке...

Dear Victor,

Could you give me an advice concerning the filenames I want to sync: how can I use characters like & « » Æ and so on? Every time I get this message:
Invalid octets in filename, does not map to desired encoding "UTF-8". But when I type
locale | egrep "LANG=" | cut -d . -f 2
it shows UTF-8. I use Debian GNU/Linux 5.0.

Thanks in advance,
Nikita

Hi.

I'm not sure what is wrong so I'm just guessing the title.

When uploading a file, they suddenly throws below errors. Any idea whats causing this?

$ mtglacier upload-file --config blankedvault-n.conf 
MT-AWS-Glacier, Copyright 2012-2013 Victor Efimov http://mt-aws.com/ Version 1.058

PID 10245 Started worker
PID 10246 Started worker
PID 10247 Started worker
PID 10248 Started worker
PID 10249 Started worker
PID 10247 Created an upload_id zigAwfwgMet11sadF40GGqL2Eqy5kSmMEbWlNVua7xbVY3kcFVebMJpS1DGFsuQZVs7tRFwSrYqfkoJOkV-Fxj4jaDF5
PID 10245 Uploaded part for n.tar.gz at offset [0]
PID 10247 Uploaded part for n.tar.gz at offset [67108864]
PID 10248 Uploaded part for n.tar.gz at offset [50331648]
PID 10249 Uploaded part for n.tar.gz at offset [16777216]
PID 10246 Uploaded part for n.tar.gz at offset [33554432]
PID 10247 Uploaded part for n.tar.gz at offset [100663296]
PID 10245 Uploaded part for n.tar.gz at offset [83886080]
PID 10248 Uploaded part for n.tar.gz at offset [117440512]
PID 10249 Uploaded part for n.tar.gz at offset [134217728]
PID 10246 Uploaded part for n.tar.gz at offset [150994944]
PID 10247 Uploaded part for n.tar.gz at offset [167772160]
PID 10248 Uploaded part for n.tar.gz at offset [201326592]
PID 10245 Uploaded part for n.tar.gz at offset [184549376]
PID 10249 Uploaded part for n.tar.gz at offset [218103808]
PID 10246 Uploaded part for n.tar.gz at offset [234881024]
PID 10248 Uploaded part for n.tar.gz at offset [268435456]
PID 10247 Uploaded part for n.tar.gz at offset [251658240]
PID 10249 Uploaded part for n.tar.gz at offset [301989888]
PID 10245 Uploaded part for n.tar.gz at offset [285212672]
PID 10246 Uploaded part for n.tar.gz at offset [318767104]
PID 10247 Uploaded part for n.tar.gz at offset [352321536]
PID 10248 Uploaded part for n.tar.gz at offset [335544320]
PID 10249 Uploaded part for n.tar.gz at offset [369098752]
PID 10245 Uploaded part for n.tar.gz at offset [385875968]
PID 10246 Uploaded part for n.tar.gz at offset [402653184]
PID 10247 Uploaded part for n.tar.gz at offset [419430400]
PID 10248 Uploaded part for n.tar.gz at offset [436207616]
PID 10249 Uploaded part for n.tar.gz at offset [452984832]
Error:
===REQUEST:
PUT http://glacier.ap-northeast-1.amazonaws.com/-/vaults/blankedvault-n/multipart-uploads/zigAwfwgMet11sadF40GGqL2Eqy5kSmMEbWlNVua7xbVY3kcFVebMJpS1DGFsuQZVs7tRFwSrYqfkoJOkV-Fxj4jaDF5
Authorization: AWS4-HMAC-SHA256 Credential=***REMOVED***/20131211/ap-northeast-1/glacier/aws4_request, SignedHeaders=content-length;content-range;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-glacier-version;x-amz-sha256-tree-hash, Signature=***REMOVED***
Host: glacier.ap-northeast-1.amazonaws.com
User-Agent: mt-aws-glacier/1.058 (http://mt-aws.com/) libwww-perl/6.05
Content-Length: 16777216
Content-Range: bytes 469762048-486539263/*
Content-Type: application/octet-stream
X-Amz-Content-Sha256: 8fc2f6fbfa943ee107ab4f638e72ddf701a5199f879e07b8486d53991612ea17
X-Amz-Date: 20131211T080913Z
X-Amz-Glacier-Version: 2012-06-01
X-Amz-Sha256-Tree-Hash: 91237e64e92b1c2c5de996a3df72b9b627796c5da9bfe7b404535f05238af65d

===RESPONSE:
HTTP/1.1 400 Bad Request
Date: Wed, 11 Dec 2013 08:10:32 GMT
Content-Length: 237
Content-Type: application/json
Client-Date: Wed, 11 Dec 2013 08:10:32 GMT
Client-Peer: 103.246.150.209:80
Client-Response-Num: 1
X-Amzn-RequestId: mvh1llKFIl0bHhtc40tVhhTJw5UG2ogwq_z6h8R3H8Qw7Gg

{"code":"InvalidParameterValueException","message":"Checksum mismatch: expected 91237e64e92b1c2c5de996a3df72b9b627796c5da9bfe7b404535f05238af65d, computed a7b93b01e8b642f5d54cebe61d41a418691ad1346ddb8f78be77f478aea05b08","type":"Client"}

ERROR (child 10245): Unexpected reply from remote server

EXIT on SIGCHLD
$

Entry of my blankedvault-n.conf

key=-CUT OUT-
secret=-CUT OUT- 
region=ap-northeast-1
dir=/disk1/
filename=/disk1/n.tar.gz
journal=blankedvault-n.log
vault=blankedvault-n
concurrency=5

Size of file

$ du -sh /disk1/n.tar.gz 
9.8G    /disk1/n.tar.gz
$

Thanks.

I have a NAS backup appliance that serves out a CIFS share. When attempting to run a sync with --dir from a path on this share, nothing is backed up. When I move those same files locally, the sync runs fine.

Example:

Does not backup

./mtglacier sync --config=glacier.cfg --vault=test --dir=/mnt/backup/tmp --journal=journal.log

mv /mnt/backup/tmp /tmp/

Works

./mtglacier sync --config=glacier.cfg --vault=test --dir=/tmp/tmp --journal=journal.log

Would like to be able to sync from CIFS shares.