izaakschroeder / vinyl-s3 Goto Github PK

Hi Izaak @izaakschroeder,
thank you for this wonderful idea and implementation!

May I ask you for your advice, as I'm having trouble setting AWS options and am not sure where and how to set them correctly. I understand that vinyl-s3 builds upon aws-sdk, so I read their docs and had some success.

I tried setting AWS options with AWS.config.update before instantiating vinyl-s3, but unfortunately that had no effect at all (and I don't quite understand why). What did work was setting environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY - fair enough.

Where I'm still having difficulty is the region.

I tried setting AWS_DEFAULT_REGION but unfortunately that was not successful.
I then tried AWS.config.update({region:'eu-west-1'});, but I'm still getting

PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

I'm a bit stuck here and am hoping you might have an idea?

Thank you very much for your help!

Best,
Andreas

UPDATE: setting AWS_REGION worked - but I'm still wondering if and how it's possible to set options programmatically? Thank you!

Use the ETag result from listObjects, and compare to the MD5 hash of the local file.

I'm using gulp-filter and gulp-vinyl-md5 to achieve something similar, but it is probably worth it to just have this library magically handle that.

I understand this isn't really an "issue". Is there a way to get vinyl-s3 to show the upload progress?

Assuming you figure out that you need the "through2" plugin, the meta task does not work.

It throws an exception:
TypeError: Cannot read property 'length' of null

I could not find a way to set headers for an s3 upload.
In my case I'd like to specify Content-Type, Content-Encoding, Cache-Control as well as x-amz-acl.
Did I overlook anything?

If options.meta is true

Running "yarn add --dev vinyl-s3" in Node 6.x fails:

$ yarn add --dev vinyl-s3
yarn add v0.17.9
[1/4] 🔍  Resolving packages...
warning vinyl-s3 > s3-glob > [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
[2/4] 🚚  Fetching packages...
error [email protected]: The engine "node" is incompatible with this module. Expected version "^1.0.0".
error Found incompatible module
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.

This is because the "engines" field of the s3-glob dependency is outdated. There are a few ways this could be resolved:

• s3-glob publishes a new version permitting Node 4+ as "engines".
• This repo subsumes the functionality of s3-glob and removes its dependency.
• This repo uses a different package to handle S3 globs.

Obviously the first one would be the most ideal, but any of the three would solve the issue.

See https://www.npmjs.com/package/gulp-data .

var data = require('gulp-data');
var s3 = require('vinyl-s3');
var gzip = require('gulp-gzip');

gulp.src('*.js')
  .pipe(gzip())
  .pipe(data({contentEncoding: 'gzip'})
  .pipe(s3.dest('s3:/mybucket'))

I tried to using vinyl-s3 to fetch metadata on existing object in my bucket.
I see the {read: false} option discussed, but things still move pretty slow.

I ended up using the aws-sdk directly and doing

var s3 = new AWS.S3();
s3.listObjects({
  Bucket: 'myBucket'
});

This returned the same metadata more than 100x faster. Any reason for this?

Details on the available config options are pretty sparse.

In particular #16 had information I needed, but I had to search for it.

It would be helpful if that was documented upfront for those unfamiliar with the aws-sdk.

The current array of engines doesn't cover any LTS Node engines. this needs anything greater than 0.8, the engines should be >=0.8 as is what it appears to need. Without up-to-date engines (or simply removing it entirely as the versions in currently are no longer supported), package managers like yarn will yell and be unable to install without exceptions

in my tests the first character of the destination file is dropped. I think filepath.substr in https://github.com/izaakschroeder/vinyl-s3/blob/master/lib/write-stream.js#L36 should not add +1 to the file.base.length.