maxgillett / s3_multipart

Direct multipart uploading to Amazon S3 in Rails

S3 Multipart

The S3 Multipart gem brings direct multipart uploading to S3 to Rails. Data is piped from the client straight to Amazon S3 and a server-side callback is run when the upload is complete.

Multipart uploading allows files to be split into many chunks and uploaded in parallel or succession (or both). This can result in dramatically increased upload speeds for the client and allows for the pausing and resuming of uploads. For a more complete overview of multipart uploading as it applies to S3, see the documentation here. Read more about the philosophy behind the gem on the Bitcast blog.

What's New

0.0.10.6 - See pull request 23 for detailed changes. Changes will be documented in README soon.

0.0.10.5 - See pull request 16 and 18 for detailed changes.

0.0.10.4 - Fixed a race condition that led to incorrect upload progress feedback.

0.0.10.3 - Fixed a bug that prevented 5-10mb files from being uploaded correctly.

0.0.10.2 - Modifications made to the database table used by the gem are now handled by migrations. If you are upgrading versions, run rails g s3_multipart:install_new_migrations followed by rake db:migrate. Fresh installs do not require subsequent migrations. The current version must now also be passed in to the gem's configuration function to alert you of breaking changes. This is done by setting a revision yml variable. See the section regarding the aws.yml file in the readme section below (just before "Getting Started").

0.0.9 - File type and size validations are now specified in the upload controller. Untested support for browsers that lack the FileBlob API

Setup

First, assuming that you already have an S3 bucket set up, you will need to paste the following into your CORS configuration file, located under the permissions tab in your S3 console.

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>PUT</AllowedMethod>
        <AllowedMethod>GET</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <ExposeHeader>ETag</ExposeHeader>
        <AllowedHeader>Authorization</AllowedHeader>
        <AllowedHeader>Content-Type</AllowedHeader>
        <AllowedHeader>Content-Length</AllowedHeader>
        <AllowedHeader>x-amz-date</AllowedHeader>
        <AllowedHeader>origin</AllowedHeader>
        <AllowedHeader>Access-Control-Expose-Headers</AllowedHeader>
    </CORSRule>
</CORSConfiguration>

Next, install the gem, and add it to your gemfile.

gem install s3_multipart

Run the included generator to create the required migrations and configuration files. Make sure to migrate after performing this step.

rails g s3_multipart:install

If you are using sprockets, add the following to your application.js file. Make sure that the latest underscore and jQuery libraries have been required before this line. Lodash is not supported at this time.

//= require s3_multipart/lib

Also in your application.js file you will need to include the following:

$(function() {
  $(".submit-button").click(function() { // The button class passed into multipart_uploader_form (see "Getting Started")
    new window.S3MP({
      bucket: "YOUR_S3_BUCKET",
      fileInputElement: "#uploader",
      fileList: [], // An array of files to be uploaded (see "Getting Started")
      onStart: function(upload) {
        console.log("File %d has started uploading", upload.key)
      },
      onComplete: function(upload) {
        console.log("File %d successfully uploaded", upload.key)
      },
      onPause: function(key) {
        console.log("File %d has been paused", key)
      },
      onCancel: function(key) {
        console.log("File upload %d was canceled", key)
      },
      onError: function(err) {
        console.log("There was an error")
      },
      onProgress: function(num, size, done, percent, speed) {
        console.log("File %d is %f percent done (%f of %f total) and uploading at %s", num, percent, done, size, speed);
      }
    });
  });
});

This piece of code does some configuration and provides various callbacks that you can hook into. It will be discussed further at the end of the Getting Started guide below.

Finally, edit the aws.yml that was created in your config folder with the correct credentials for each environment. Set the revision number to the current version number. If breaking changes are made to the gem in a later version, then you will be notified when the two versions do not match in the log.

development:
  access_key_id: ""
  secret_access_key: ""
  bucket: ""
  revision: "#.#.#"

Getting Started

S3_Multipart comes with a generator to set up your upload controllers. Running

rails g s3_multipart:uploader video

creates a video upload controller (video_uploader.rb) which resides in "app/uploaders/multipart" and looks like this:

class VideoUploader < ApplicationController
  extend S3Multipart::Uploader::Core

  # Attaches the specified model to the uploader, creating a "has_one" 
  # relationship between the internal upload model and the given model.
  attach :video

  # Only accept certain file types. Expects an array of valid extensions.
  accept %w(wmv avi mp4 mkv mov mpeg)

  # Define the minimum and maximum allowed file sizes (in bytes)
  limit min: 5*1000*1000, max: 2*1000*1000*1000

  # Takes in a block that will be evaluated when the upload has been 
  # successfully initiated. The block will be passed an instance of 
  # the upload object as well as the session hash when the callback is made. 
  # 
  # The following attributes are available on the upload object:
  # - key:       A randomly generated unique key to replace the file
  #              name provided by the client
  # - upload_id: A hash generated by Amazon to identify the multipart upload
  # - name:      The name of the file (including extensions)
  # - location:  The location of the file on S3. Available only to the
  #              upload object passed into the on_complete callback
  #
  on_begin do |upload, session|
    # Code to be evaluated when upload completes  
  end

  # See above comment. Called when the upload has successfully completed
  on_complete do |upload, session|
    # Code to be evaluated when upload completes                                                 
  end

end

The generator requires a model to be passed in (in this case, the video model) and automatically creates a "has one" relationship between the upload and the model (the video). For example, in the block that the on_begin method takes, a video object could be created (video = Video.create(name: upload.name)) and linked with the upload (upload.video = video). When the block passed into the on_complete is run at a later point in time, the associated video is now accessible by calling upload.video. If instead, you want to construct the video object on completion and link the two then, that is ok.

The generator also creates the migration to add this functionality, so make sure to do a rake db:migrate after generating the controller.

To add the multipart uploader to a view, insert the following:

<%= multipart_uploader_form(input_name: 'uploader',
                            uploader: 'VideoUploader',
                            button_class: 'submit-button', 
                            button_text: 'Upload selected videos',
                            html: %Q{<button class="upload-button">Select videos</button>}) %>

The multipart_uploader_form function is a view helper, and generates the necessary input elements. It takes in a string of html to be interpolated between the generated file input element and submit button. It also expects an upload controller (as a string or constant) to be passed in with the 'uploader' option. This links the upload form with the callbacks specified in the given controller.

The code above outputs this:

<input accept="video" data-uploader="7b2a340f42976e5520975b5d5668dc4c19b38f2c" id="uploader" multiple="multiple" name="uploader" type="file">
<button class="upload-button" type="submit">Select videos</button>
<button class="submit-button"><span>Upload selected videos</span></button>

Let's return to the javascript that you inserted into the application.js during setup. The S3MP constructor takes in a configuration object with a handful of required callback functions. It also takes in list of files (through the fileList property) that is an array of File objects. This could be retrieved by calling $("#uploader").get(0).files if the input element had an "uploader" id, or it could be manually constructed. See the internal tests for an example of this manual construction.

The S3MP constructor also returns an object that you can interact with. Although not demonstrated here, you can call cancel, pause, or resume on this object and pass in the zero-indexed key of the file in the fileList array you want to control.

Tests

First, create a file setup_credentials.rb in the spec folder.

# spec/setup_credentials.rb
S3Multipart.configure do |config|
  config.bucket_name   = ''
  config.s3_access_key = ''
  config.s3_secret_key = ''
  config.revision = S3Multipart::Version
end

You can now run all of the RSpec and Capybara tests with rspec spec

Combustion is also used to simulate a rails application. Paste the following into a config.ru file in the base directory:

require 'rubygems'
require 'bundler'

Bundler.require :development

Combustion.initialize! :active_record, :action_controller,
                       :action_view, :sprockets

S3Multipart.configure do |config|
  config.bucket_name   = ''
  config.s3_access_key = ''
  config.s3_secret_key = ''
  config.revision = S3Multipart::Version
end

run Combustion::Application

and boot up the app by running rackup. A fully functional uploader is now available if you visit http://localhost:9292

Jasmine tests are also available for the client-facing javascript library. After installing Grunt and PhantomJS, and running npm install once, you can run the tests headlessly by running grunt jasmine.

To re-build the javascript library, run grunt concat and to minify, grunt min.

Contributing

S3_Multipart is very much a work in progress. If you squash a bug, make enhancements, or write more tests, please submit a pull request.

Browser Compatibility

The library is working on the latest version of IE, Firefox, Safari, and Chrome. Tests for over 100 browsers are currently being conducted.

To Do

• If the FileBlob API is not supported on page load, the uploader should just send one giant chunk (DONE)
• Handle network errors in the javascript client library
• File type validations (DONE)
• File size validations (DONE)
• More and better tests
• More browser testing
• Roll file signing and initiation into one request

s3_multipart's Issues

problem with startProgressTimer

startProgressTimer of the S3MP object isn't working correctly for me. These three lines seem to be the culprit:

          _.each(upload.inprogress,function(val) {
            done += val;
          });

console.dir(upload.inprogress) shows that index 0 is undefined

The files seem to be uploading fine. My progress meter doesn't work however. Any insights would be most welcome. Thanks.

I changed them to:

          _.each(upload.inprogress,function(val) {
            if(!isNaN(val)) done += val;
          });

and now it seems to be working. Is this just me?

Support for IE8/9?

I hate to ask this here - but I couldn't find any details anywhere: is there currently any support for IE 8 or 9?

multipart_uploader_form rails helper

I used the form helper found on the readme in my rhtml . But while executing this in browser it was throwing as " undefined method `join' for nil:NilClass " .

Can any one please help me how to solve this issue .

Javascrip Error : _.mixin({

Do you know how to resolve this one ? I got this when loading the page.

Help Please!

Thanks

Migration has wrong table name after generator is run

When I run this generator:

rails g s3_multipart:uploader invoice

The migration which is generated contains this:

class AddUploaderToInvoice < ActiveRecord::Migration
  def change
    change_table :invoice do |t|
      t.string :uploader
    end
  end
end

But this fails because the table/relation name in the database is "invoices". The generator should be updated to pluralize the table name in the migration (I think).

fix initiateMultipart response object change

See pull request: #32

Mixed-content errors using s3_multipart on https URL

See pull request: #33

How do you get the tests to run?

I'm following the instructions to run the gem test suite. I created setup_credentials.rb and put the suggested code in it and tried
rspec spec
which gives this error
spec/setup_credentials.rb:6:in `block in <top (required)>': uninitialized constant S3Multipart::Version (NameError)

So I then removed setup_credentials.rb and tried again and got
gems/activesupport-3.2.9/lib/active_support/dependencies.rb:251:in `require': cannot load such file -- setup_credentials.rb (LoadError)

Where do I go from here? Following the instructions gives an error, yet there is definitely code in specs so it must have worked at some point.

Documentation on pause/resume functionality

Would be better if there is any detailed document with demo with all functionality explained. I would like to see the pause & resume s3 uploads in action.

When the file size is from 5MB to 10MB, it causes an error

When the file size is from 5MB to 10MB, the value of 'content_lengths' becomes number.
If it is number, the options[:content_lengths] cannot split in TransferHelpers module.
So I think it should change something like this:

S3MP.prototype.signPartRequests = function(id, object_name, upload_id, parts, cb) {
  var content_lengths, url, body, xhr;

  if (parts.length == 1) {
    content_lengths = String(parts[0].size);
  } else {
    content_lengths = _.reduce(_.rest(parts), function(memo, part) {
      return memo + "-" + part.size;
    }, parts[0].size);
  }
  url = "/s3_multipart/uploads/"+id;
  body = JSON.stringify({ object_name     : object_name,
                          upload_id       : upload_id,
                          content_lengths : content_lengths
                        });

  xhr = this.createXhrRequest('PUT', url);
  this.deliverRequest(xhr, body, cb);
};

Pass in AWS credentials instead of hardcoding in aws.yml

First of all, this gem is amazing and thanks for making it. Second, bitcast.io is sweet, we never knew about it.

Issue:
Is it possible to set different AWS credentials for each user, instead of hard coding into "aws.yml"?

Details:
We let users upload files to an S3 bucket they specify, using their own AWS credentials that we encrypt & store in a model. From the s3_multipart readme.md, we have to hardcode AWS credentials in "aws.yml".

Url for signPartRequests and completeMultipart JS functions is relative

The URL path is relative.

S3MP.prototype.signPartRequests = function(id, object_name, upload_id, parts, cb) {
  var content_lengths, url, body, xhr;

  content_lengths = _.reduce(_.rest(parts), function(memo, part) {
    return memo + "-" + part.size;
  }, parts[0].size);

  url = "s3_multipart/uploads/"+id;
  body = JSON.stringify({ object_name     : object_name,
                          upload_id       : upload_id,
                          content_lengths : content_lengths
                        });

  xhr = this.createXhrRequest('PUT', url);
  this.deliverRequest(xhr, body, cb);
};

S3MP.prototype.completeMultipart = function(uploadObj, cb) {
  var url, body, xhr;

  url = 's3_multipart/uploads/'+uploadObj.id;
  body = JSON.stringify({ object_name    : uploadObj.object_name,
                          upload_id      : uploadObj.upload_id,
                          content_length : uploadObj.size,
                          parts          : uploadObj.Etags
                        });

  xhr = this.createXhrRequest('PUT', url);
  this.deliverRequest(xhr, body, cb);
};

The url variable should include a beginning forward slash so that it won't matter where you place the form helper.

[improvement] Pulling AWS credentials from ENV variables instead of hardcoding it in aws.yml file

I changed my config/initializers/s3_multipart.rb to look like this:

S3Multipart.configure do |config|
config.bucket_name = ENV['AWS_S3_BUCKET'],
config.s3_access_key = ENV['AWS_ACCESS_KEY_ID'],
config.s3_secret_key = ENV['AWS_SECRET_ACCESS_KEY'],
config.revision = "0.0.10.6"
end

multipart_uploader_form fails when types are not specified

I used the form helper found on the readme but I got this error:

undefined method `join' for nil:NilClass

After looking through the code I found it was caused by not specifying options[:types] in the method call.

module S3Multipart
  module ActionViewHelpers
    module FormHelper
      def multipart_uploader_form(options = {})
        p options
        uploader_digest = S3Multipart::Uploader.serialize(options[:uploader])
        html = file_field_tag options[:input_name], :accept => options[:types].join(','), :multiple => 'multiple', :data => {:uploader => uploader_digest}
        html << options[:html].html_safe
        html << button_tag(:class => options[:button_class]) do
          content_tag(:span, options[:button_text])
        end
      end
    end
  end
end

Maybe I'm missing something here. Thanks.

[question] How does it works with existing CarrierWave uploaders?

Can I simply add like this inside my current CarrierWave uploader?

class ImageUploader < CarrierWave::Uploader::Base
* extend S3Multipart::Uploader::Core*
end

Question (Files are not uploaded in S3 Bucket)

We followed the instructions as directed, but could not upload the file successfully on S3. OnComplete function runs successfully (We had put an alert in oncomplete function, which pops up fine after the upload), but files are not visible in the S3 Bucket...

What could be the potential source of error?

Helpers methods not available

Hi there,

I'm currently testing the s3_multipart implementation in our app but after following all the steps now I can't get any of our (Module) helpers to work, none of them is available, is there something special I need to do to include the helpers again?

Thanks

Rails 4.1.8
Ruby 2.1.1
s3_multipart 0.0.10.6

RequestTimeTooSkewed error on long-running requests

I was getting RequestTimeTooSkewed errors from S3 when requests ran longer than a certain time (I think it was ten minutes - ETA: probably fifteen minutes). This turns out to be because a datetime is encoded in the signature for each part, and if it differs too much from S3's server time, S3 no longer accepts the PUTs and instead returns the RequestTimeTooSkewed error. The times get out of sync because the signatures for all the upload's parts are created when the upload is initialized, but the parts get sent sometime later. The solution is to generate the signature just before the part is sent, rather than when the whole upload is initialized.

I've fixed this in my fork, but my fork is too different to make a pull request from, and I'm not very confident that what I've done is ideal. But I'll describe how I fixed it:

In upload.js, in the init function, remove the upload.signPartRequests block

In uploadpart.js, change the activate function to this:

UploadPart.prototype.activate = function() {
  var upload_part = this;
  this.upload.signPartRequest(this.upload.id, this.upload.object_name, this.upload.upload_id, this, function(response) {
    upload_part.xhr.open('PUT', '//'+upload_part.upload.bucket+'.s3.amazonaws.com/'+upload_part.upload.object_name+'?partNumber='+upload_part.num+'&uploadId='+upload_part.upload.upload_id, true);

    upload_part.xhr.setRequestHeader('x-amz-date', response.date);
    upload_part.xhr.setRequestHeader('Authorization', response.authorization);

    upload_part.xhr.send(upload_part.blob);
    upload_part.status = "active";
  });
};

In s3mp.js, in the beginUpload function, remove this if condition (though keep the contents of the block):

if (i[key] === num_parts) {
  ...keep this section...
}

Also in s3mp.js, add this function:

S3MP.prototype.signPartRequest = function(id, object_name, upload_id, part, cb) {
  var content_length, url, body, xhr;

  content_length = part.size;

  url = "/s3_multipart/uploads/"+id;
  body = JSON.stringify({ object_name     : object_name,
                          upload_id       : upload_id,
                          content_length : content_length,
                          part_number : part.num
                        });

  xhr = this.createXhrRequest('PUT', url);
  this.deliverRequest(xhr, body, cb);
};

The vendor/assets/javascripts/s3_multipart/lib.js file is the concatenation of all the Javascript files, so make all the changes there too (or regenerate that file).

support blob uploads

See pull request: #24

It all looks so wonderful. Then I ran rake db:migrate

rake db:migrate
rake aborted!
ArgumentError: Breaking changes were made to the S3_Multipart gem:

See the Readme for additional information.
/home/john/.rvm/gems/ruby-2.1.0-preview2@s3_multipart/gems/s3_multipart-0.0.10.6/lib/s3_multipart/config.rb:16:in check_for_breaking_changes' /home/john/.rvm/gems/ruby-2.1.0-preview2@s3_multipart/gems/s3_multipart-0.0.10.6/lib/s3_multipart/config.rb:10:inconfigure'
/home/john/.rvm/gems/ruby-2.1.0-preview2@s3_multipart/gems/s3_multipart-0.0.10.6/lib/s3_multipart.rb:12:in configure' /home/john/ruby-projects/research/s3_multipart/config/initializers/s3_multipart.rb:3:in<top (required)>'

This was in a clean installation, with nothing added apart from the gem. There's nothing in the README.rdoc to tell me what to do.

So there's no way for a new user to install and use this gem

running ruby 2.1.0-preview2 and Rails 4.1.0

Failing to upload files

When I try to upload via http://localhost:9292 using Combustion, console log says there was an error. I followed your instructions in README.md and each time I upload, I get the same error

When I ran rspec spec:

 1) Uploads controller should create an upload

 Failure/Error: parsed_body.should_not eq({"error"=>"There was an error initiating the upload"})

 expected: value != {"error"=>"There was an error initiating the upload"}
        got: {"error"=>"There was an error initiating the upload"}

(compared using ==)

Diff:{"error"=>"There was an error initiating the upload"}.==({"error"=>"There was an error initiating the upload"}) returned false even though the diff between {"error"=>"There was an error initiating the upload"} and {"error"=>"There was an error initiating the upload"} is empty. Check the implementation of {"error"=>"There was an error initiating the upload"}.==.
 # ./spec/integration/uploads_controller_spec.rb:8:in `block (2 levels) in <top (required)>'

Initializer breaks font-awesome-rails helpers

The loop that requires all the uploaders breaks view helpers defined in the font-awesome-rails gem, i.e. I'm starting to get undefined method 'fa_icon' errors. Seems like those requires somehow mess up lookup paths?

As a workaround, I had to explicitly include the module into ActionView in a separate initializer within my app:

ActionView::Base.send :include, FontAwesome::Rails::IconHelper

There might be other gems affected in the same way.

grunt outdated

grunt 0.4 now uses a gruntfile instead of grunt.js

Example view code

In the readme file's example view code, there is a comma missing after:
uploader: 'VideoUploader'

Also, doesn't the "types" attribute have to be specified here?

Uploads don't complete.

Would you happen to have any idea why none of my uploads complete? Less than a minute after a file starts uploading, it stops. I'm using a brand new rails app setup exactly as suggested in your README file.

[Log] File  has started uploading (application.js, line 27)
[Log] File  is 0.0019353080431962172 percent done (32768 of 1693167148 total) and uploading at 32768 (application.js, line 43)
[Log] File  is 0.07547701368465248 percent done (1277952 of 1693167148 total) and uploading at 1245184 (application.js, line 43)
[Log] File  is 0.08321824585743734 percent done (1409024 of 1693167148 total) and uploading at 131072 (application.js, line 43)
[Log] File  is 0.08321824585743734 percent done (1409024 of 1693167148 total) and uploading at 0 (application.js, line 43, x113)

I've tried both the master and development version of the gem as well.