A Node.js client designed to make large file uploads to Amazon S3 via the MultiPartUpload API simple and easy. It's built on top of the excellent Knox library from the guys over at LearnBoost.
- Simple and easy to use
- Pipe either a file, or a stream directly to S3 (No need to know the content length first!)
- Automatically separates a file/stream into appropriate sized segments for upload
- Asynchronous uploading of segments
- Handy events to track your upload progress
Planned
- Better error handling (reuploading failed parts, etc)
Installation is done via NPM, by running npm install knox-mpu-alt
To upload a stream, simply pass the stream when constructing the MultiPartUpload. The upload will then listen to the stream, and create parts from incoming data stream. When a part reaches the minimum part size, it will attempt to upload it to S3.
// Create a Knox client first
var client = knox.createClient({ ... }),
upload = null;
upload = new MultiPartUpload(
{
client: client,
objectName: 'destination.txt', // Amazon S3 object name
stream: stream
},
// Callback handler
function(err, body) {
// If successful, will return body, containing Location, Bucket, Key, ETag and size of the object
/*
{
Location: 'http://Example-Bucket.s3.amazonaws.com/destination.txt',
Bucket: 'Example-Bucket',
Key: 'destination.txt',
ETag: '"3858f62230ac3c915f300c664312c11f-9"',
size: 7242880
}
*/
}
);To upload a file, pass the path to the file in the constructor. Knox-mpu will split the file into parts and upload them.
// Create a Knox client first
var client = knox.createClient({ ... }),
upload = null;
upload = new MultiPartUpload(
{
client: client,
objectName: 'destination.txt', // Amazon S3 object name
file: ... // path to the file
},
// Callback handler
function(err, body) {
// If successful, will return body, containing Location, Bucket, Key, ETag and size of the object
/*
{
Location: 'http://Example-Bucket.s3.amazonaws.com/destination.txt',
Bucket: 'Example-Bucket',
Key: 'destination.txt',
ETag: '"3858f62230ac3c915f300c664312c11f-9"',
size: 7242880
}
*/
}
);The following options can be passed to the MultiPartUpload constructor -
clientRequired The knox client to use for this upload requestobjectNameRequired The destination object name/path on S3 for this uploadstreamThe stream to upload (required if file is not being supplied)fileThe path to the file (required if stream is not being supplied)headersAny additional headers to include on the requestspartSizeThe minimum size of the parts to upload (default to 5MB).batchSizeThe maximum number of concurrent parts that can be uploading at any one time (default is 4)maxUploadSizeThe maximum size of the file to upload (default inifinity). Useful if there is a stream with unknown length.noDiskIf true, parts will be kept in-memory instead of written to temp files (default to false).maxRetriesNumber of times to retry failed part upload (default is 0 for no retry).
The MultiPartUpload will emit a number of events -
initiatedEmitted when the multi part upload has been initiated, and received an upload ID. Passes the upload id through as the first argument to the eventuploadingEmitted each time a part starts uploading. The part id is passed as the first argument.uploadedEmitted each time a part finishes uploading. Passes through an object containing the part id and Amazon ETag for the uploaded part.failedEmitted each time a part upload fails. Passes an object containing the part id and error messagecompletedEmitted when the upload has completed successfully. Contains the object information from Amazon S3 (location, bucket, key and ETag)
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent