Comments (7)
Hello.
Thank you for your feedback.
Yes. I am planning to add option for multithreaded upload of one file from STDIN. So you will be able to
redirect tar output to mtglacier input. Something like that: "tar cz ... | mtglacier"
from mt-aws-glacier.
That could be really good!
But how do you let it interact with the journal parameter?
Could be possible to do the same during the restore phase?
Thank you again
from mt-aws-glacier.
I was planning to do something like that:
tar cz ... | mtglacier --journal=/path/to/journal --stdin --stdin-file-name=myarchive.tar --from-dir=/path/to/data
that will
- put file into Glacier
- give it a filename "myarchive.tar" in journal
- possibly and optionally write it also into /path/to/data/myarchive.tar
after restore
- restore it to myarchive.tar. no unpacking.
that's all.
This way user will save time of writing huge archive to disk and reading it again by mtglacier.
What you asking for - auto arching/unarchiving - will make workflow pretty complex to the end user.
So I think it's better/easier for user to code .sh script which will split data to archives (maybe few archives, maybe not only tar etc) and then upload to glacier. And then possibly automate restore from archives using shell scripts.
Also other thoughts: Glacier full restore is pretty expensive, so typical use case is use it as secondary backup, not as primary. In this case ease of restore is not much required.
But maybe you have idea how to support, say, tar + tag.gz archiving / unarchiving and not make the app extra complicated to implement and use, and keep Journal/Sync concept easy understandable?
from mt-aws-glacier.
I do like your upload approach, i would say it's exactly what i would like to have.
Yes point #3 can be optional, everytime i upload a tar file i don't need to keep it.
About the restore, my question was just to understand your intentions, but i see your point of view.
I know Glacier full restore is pretty expensive but it will be anyway my first backup system, for now.
I guess i will implement the restore function with a bash script file that could be executed after mtglacier.
Something like this :
- iterate over the tar files list (ordered by a naming convention defined during upload)
- extract every tar file into the /targetfolder
- delete it
from mt-aws-glacier.
Ok. cool.
Yes, use shell scripting for now. I am going to implement reading from STDIN, but this is not first priority for me.
I will keep this issue open and inform you here after implementing this. But estimate is probably 2-3 months.
from mt-aws-glacier.
Ok, let me know if i can help you!
Anyway, i will do this workaround in order to simulate the STDIN
- create tar
- move tar in tmp folder
- mtglacier --journal=/somewhere/journal --from-dir=/tmpfolder
- remove tar
Once your implementation is ready i will switch my bash script.
from mt-aws-glacier.
Upload single file or file from STDIN implemented (but only together with Journal functionality). Multithreaded. Without intermediate file buffering.
Download of single files not implemented yet.
see upload-file
command in README.
So, I am closing this. Reopen if you have more questions.
from mt-aws-glacier.
Related Issues (20)
- downloading from synced directory HOT 7
- Sync resume? HOT 2
- 400 Bad Request HOT 4
- SSL certificate has expired for mt-aws.com HOT 2
- Use of uninitialized value in numeric eq (==) at Utils.pm
- Choosing Amazon Glacier Retrieval method HOT 1
- Error While Uploading file to Galcier HOT 6
- sync to Glacier from a cifs share results in 90% of the files being skipped HOT 6
- connection problem when using https HOT 9
- InvalidSignatureException during upload of a large file HOT 1
- Job getting killed HOT 8
- Why do commands immediately exit with OKAY DONE exit status? HOT 2
- Question: MTIME missmatch HOT 1
- Unable to ignore errors (?) HOT 1
- Error trying to restore data HOT 1
- HTTP Connection problem in restore-completed job HOT 1
- ResourceNotFoundException: Vault not found for ARN HOT 4
- Error While uploading file glacier HOT 5
- Out of Memory error HOT 3
- Slow listing operation during sync operation
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mt-aws-glacier.