refile / refile-s3

Amazon S3 Backend for Refile

Refile

Refile is a modern file upload library for Ruby applications. It is simple, yet powerful.

Links:

• API documentation
• Source Code
• Contributing
• Code of Conduct
• Example Application

Features:

• Configurable backends, file system, S3, etc...
• Convenient integration with ORMs
• On the fly manipulation of images and other files
• Streaming IO for fast and memory friendly uploads
• Works across form redisplays, i.e. when validations fail, even on S3
• Effortless direct uploads, even to S3
• Support for multiple file uploads
• Support for single file upload

Sponsored by:

Quick start, Rails

Add the gem:

gem "refile", require: "refile/rails"
gem "refile-mini_magick"

We're requiring both Refile's Rails integration and image processing via the MiniMagick gem, which requires ImageMagick (or GraphicsMagick) to be installed. To install it simply run:

brew install imagemagick # OS X
sudo apt-get install imagemagick # Ubuntu

Use the attachment method to use Refile in a model:

class User < ActiveRecord::Base
  attachment :profile_image
end

Generate a migration:

rails generate migration add_profile_image_to_users profile_image_id:string &&
profile_image_filename:string && profile_image_size:string &&
profile_image_content_type:string

rake db:migrate

Add an attachment field to your form:

<%= form_for @user do |form| %>
  <%= form.attachment_field :profile_image %>
<% end %>

Set up strong parameters:

def user_params
  params.require(:user).permit(:profile_image)
end

And start uploading! Finally show the file in your view:

<%= image_tag attachment_url(@user, :profile_image, :fill, 300, 300, format: "jpg") %>

How it works

Refile consists of several parts:

1. Backends: cache and persist files
2. Model attachments: map files to model columns
3. A Rack application: streams files and accepts uploads
4. Rails helpers: conveniently generate markup in your views
5. A JavaScript library: facilitates direct uploads

Let's look at each of these in more detail!

1. Backend

Files are uploaded to a backend. The backend assigns an ID to this file, which will be unique for this file within the backend.

Let's look at a simple example of using the backend:

backend = Refile::Backend::FileSystem.new("tmp")

file = backend.upload(StringIO.new("hello"))
file.id # => "b205bc..."
file.read # => "hello"

backend.get(file.id).read # => "hello"

As you may notice, backends are "flat". Files do not have directories, nor do they have names or permissions, they are only identified by their ID.

Refile has a global registry of backends, accessed through Refile.backends.

There are two "special" backends, which are only really special in that they are the default backends for attachments. They are cache and store.

By default files will be uploaded to ./tmp/uploads/store. If you would like to persist them between deploys of your application, you can override the upload folder by adding an initializer like this:

# config/initializers/refile.rb

Refile.backends['store'] = Refile::Backend::FileSystem.new('/etc/projectname-uploads/')

The cache is intended to be transient. Files are added here before they are meant to be permanently stored. Usually files are then moved to the store for permanent storage, but this isn't always the case.

Suppose for example that a user uploads a file in a form and receives a validation error. In that case the file has been temporarily stored in the cache. The user might decide to fix the error and resubmit, at which point the file will be promoted to the store. On the other hand, the user might simply give up and leave, now the file is left in the cache for later cleanup.

Refile has convenient accessors for setting the cache and store, so for example if you add the refile-s3 gem to your Gemfile:

gem "refile-s3"

Now you can upload files to S3 easily by using these accessors:

# config/initializers/refile.rb
require "refile/s3"

aws = {
  access_key_id: "xyz",
  secret_access_key: "abc",
  region: "sa-east-1",
  bucket: "my-bucket",
}
Refile.cache = Refile::S3.new(prefix: "cache", **aws)
Refile.store = Refile::S3.new(prefix: "store", **aws)

Try this in the quick start example above and your files are now uploaded to S3.

Backends also provide the option of restricting the size of files they accept. For example:

Refile.cache = Refile::S3.new(max_size: 10.megabytes, ...)

The Refile gem only ships with a FileSystem backend. Additional backends are provided by other gems.

• Amazon S3
• Fog provides support for a ton of different cloud storage providers, including Google Storage and Rackspace CloudFiles.
• Postgresql
• Gridfs
• In Memory

Uploadable

The upload method on backends can be called with a variety of objects. It requires that the object passed to it behaves similarly to Ruby IO objects, in particular it must implement the methods size, read(length = nil, buffer = nil), eof?, rewind, and close. All of File, Tempfile, ActionDispatch::UploadedFile and StringIO implement this interface, however String does not. If you want to upload a file from a String you must wrap it in a StringIO first.

2. Attachments

You've already seen the attachment method:

class User < ActiveRecord::Base
  attachment :profile_image
end

Calling attachment generates a getter and setter with the given name. When you assign a file to the setter, it is uploaded to the cache:

User.new

# with a ActionDispatch::UploadedFile
user.profile_image = params[:file]

# with a regular File object
File.open("/some/path", "rb") do |file|
  user.profile_image = file
end

# or a StringIO
user.profile_image = StringIO.new("hello world")

user.profile_image.id # => "fec421..."
user.profile_image.read # => "hello world"

When you call save on the record, the uploaded file is transferred from the cache to the store. Where possible, Refile does this move efficiently. For example if both cache and store are on the same S3 account, instead of downloading the file and uploading it again, Refile will simply issue a copy command to S3.

Other ORMs

Refile comes with ActiveRecord integration built-in, but is built to integrate with any ORM, so building your own should not be too difficult. Some integrations are already available via gems:

• refile-sequel
• refile-mongoid

Pure Ruby classes

You can also use attachments in pure Ruby classes like this:

class User
  extend Refile::Attachment

  attr_accessor :profile_image_id

  attachment :profile_image
end

Keeping uploaded files

By default Refile will delete a stored file when its model is destroyed. You can change this behaviour by passing in the destroy option.

class User < ActiveRecord::Base
  attachment :profile_image, destroy: false
end

Now Refile will not delete the profile_image file from the store if the user is destroyed.

3. Rack Application

Refile includes a Rack application (an endpoint, not a middleware), written in Sinatra. This application streams files from backends and can even accept file uploads and upload them to backends.

Important: Unlike other file upload solutions, Refile always streams your files through your application. It cannot generate URLs to your files. This means that you should always put a CDN or other HTTP cache in front of your application. Serving files through your app takes a lot of resources and you want it to happen rarely.

Setting this up is actually quite simple, you can use the same CDN you would use for your application's static assets. This blog post explains how to set this up (bonus: faster static assets!). Once you've set this up, simply configure Refile to use your CDN:

Refile.cdn_host = "https://your-dist-url.cloudfront.net"

Using the HTTPS protocol for Refile.cdn_host is recommended. There aren't any performance concerns, and it is always safe to request HTTPS assets.

Mounting

If you are using Rails and have required refile/rails.rb, then the Rack application is mounted for you at /attachments. You should be able to see this when you run rake routes.

You can configure Refile to use a different mount_point than /attachments:

Refile.mount_point = "/your-preferred-mount-point"

You could also run the application on its own, it doesn't need to be mounted to work.

If you are using a catch-all route (such as required by Comfy CMS), you will need to turn off Automounting and add the refile route before your catch all route.

(in initializers/refile.rb)

Refile.automount = false

in routes.rb

  mount Refile.app, at: Refile.mount_point, as: :refile_app

  # Make sure this routeset is defined last
  comfy_route :cms, :path => '/', :sitemap => true

Retrieving files

Files can be retrieved from the application by calling:

GET /attachments/:token/:backend_name/:id/:filename

The :filename serves no other purpose than generating a nice name when the user downloads the file, it does not in any way affect the downloaded file. For caching purposes you should always use the same filename for the same file. The Rails helpers default this to the name of the column.

The :token is a generated digest of the request path when the Refile.secret_key is configured; otherwise, the application will raise an error. The digest feature provides a security measure against unverified requests.

NOTICE: If you don't set the Refile.secret_key we will use rails secret_key_base to generate the token. We suggest you not to change the secret_key_base after you generated and hardcoded some attachment URLs in your application (e.g. blog post images), because the token will change and you'll not be able to retrieve in this case, the images.

Processing

Refile provides on the fly processing of files. You can trigger it by calling a URL like this:

GET /attachments/:token/:backend_name/:processor_name/*args/:id/:filename

Suppose we have uploaded a file:

Refile.cache.upload(StringIO.new("hello")).id # => "a4e8ce"

And we've defined a processor like this:

Refile.processor :reverse do |file|
  StringIO.new(file.read.reverse)
end

Then you could do the following.

curl http://127.0.0.1:3000/attachments/token/cache/reverse/a4e8ce/some_file.txt
elloh

Refile calls call on the processor and passes in the retrieved file, as well as all additional arguments sent through the URL.

4. Rails helpers

Refile provides the attachment_field form helper which generates a file field as well as a hidden field. This field keeps track of the file in case it is not yet permanently stored, for example if validations fail. It is also used for direct and presigned uploads. For this reason it is highly recommended to use attachment_field instead of file_field.

<%= form_for @user do |form| %>
  <%= form.attachment_field :profile_image %>
<% end %>

Will generate something like:

<form action="/users" enctype="multipart/form-data" method="post">
  <input name="user[profile_image]" type="hidden">
  <input name="user[profile_image]" type="file">
</form>

The attachment_url helper can then be used for generating URLs for the uploaded files:

<%= link_to "Image", attachment_url(@user, :profile_image) %>

Any additional arguments to it are included in the URL as processor arguments:

<%= link_to "Image", attachment_url(@user, :profile_image, :fill, 300, 300) %>

There's also a helper for generating image tags:

<%= attachment_image_tag(@user, :profile_image, :fill, 300, 300) %>

You can also provide a limit to the image:

<%= link_to "Image", attachment_url(@user, :profile_image, :limit, 400, 500) %>

If you don't care about the aspect ratio and want an exact dimension, you can use !:

<%= link_to "Image", attachment_url(@user, :profile_image, :limit, 400, "1000!") %>

Keep in mind that it's also important to remember you can not stretch the image, even you set a larger width or height the image will keep its default dimensions. For example: if you set 400x1000! for an image 600x800 it'll keep its height of 400x800.

If you just care about limit only one dimension, you can use nil in widht or height:

<%= link_to "Image", attachment_url(@user, :profile_image, :limit, 400, nil) %>
<%= link_to "Image", attachment_url(@user, :profile_image, :limit, nil, 400) %>

With this helper you can specify an image/asset which is used as a fallback in case no file has been uploaded:

<%= attachment_url(@user, :profile_image, :fill, 300, 300, fallback: "default.png") %>
<%= attachment_image_tag(@user, :profile_image, :fill, 300, 300, fallback: "default.png") %>

You can also set the URL to force the download of the uploaded file:

<%= link_to "Download", attachment_url(@user, :profile_image, force_download: true) %>

Use Refile.attachment_url if you already have attachment in your routes.

5. JavaScript library

Refile's JavaScript library is small but powerful.

Uploading files is slow, so anything we can do to speed up the process is going to lead to happier users. One way to cheat is to start uploading files directly after the user has chosen a file, instead of waiting until they hit the submit button. This provides a significantly better user experience. Implementing this is usually tricky, but thankfully Refile makes it very easy.

First, load the JavaScript file. If you're using the asset pipeline, you can simply include it like this:

//= require refile

Otherwise you can grab a copy here. Be sure to always update your copy of this file when you upgrade to the latest Refile version.

Now mark the field for direct upload:

<%= form.attachment_field :profile_image, direct: true %>

There is no step 3 ;)

The file is now uploaded to the cache immediately after the user chooses a file. If you try this in the browser, you'll notice that an AJAX request is fired as soon as you choose a file. Then when you submit to the server, the file is no longer submitted, only its id.

If you want to improve the experience of this, the JavaScript library fires a couple of custom DOM events. These events bubble, so you can also listen for them on the form for example:

form.addEventListener("upload:start", function() {
  // ...
});

form.addEventListener("upload:success", function() {
  // ...
});

input.addEventListener("upload:progress", function() {
  // ...
});

You can also listen for them with jQuery, even with event delegation:

$(document).on("upload:start", "form", function(e) {
  // ...
});

This way you could for example disable the submit button until all files have uploaded:

$(document).on("upload:start", "form", function(e) {
  $(this).find("input[type=submit]").attr("disabled", true)
});

$(document).on("upload:complete", "form", function(e) {
  if(!$(this).find("input.uploading").length) {
    $(this).find("input[type=submit]").removeAttr("disabled")
  }
});

Presigned uploads

Amazon S3 supports uploads directly from the browser to S3 buckets. With this feature you can bypass your application entirely; uploads never hit your application at all. Unfortunately the default configuration of S3 buckets does not allow cross site AJAX requests from posting to buckets. Fixing this is easy though.

• Open the AWS S3 console and locate your bucket
• Right click on it and choose "Properties"
• Open the "Permission" section
• Click "Add CORS Configuration"

The default configuration only allows "GET", you'll want to allow "POST" as well. You'll also want to permit the "Content-Type" and "Origin" headers.

It could look something like this:

<CORSConfiguration>
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
        <AllowedMethod>POST</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <AllowedHeader>Authorization</AllowedHeader>
        <AllowedHeader>Content-Type</AllowedHeader>
        <AllowedHeader>Origin</AllowedHeader>
    </CORSRule>
</CORSConfiguration>

If you're paranoid you can restrict the allowed origin to only your domain, but since your bucket is only writable with authentication anyway, this shouldn't be necessary. Note that you do not need to, and in fact you shouldn't, make your bucket world writable.

Once you've put in the new configuration, click "Save". After that it may take some time for the CORS setup to kick in (because of DNS propagation).

Now you can enable presigned uploads:

<%= form.attachment_field :profile_image, presigned: true %>

You can also enable both direct and presigned uploads, and it'll fall back to direct uploads if presigned uploads aren't available. This is useful if you're using the FileSystem backend in development or test mode and the S3 backend in production mode.

<%= form.attachment_field :profile_image, direct: true, presigned: true %>

Browser compatibility

Refile's JavaScript library requires HTML5 features which are unavailable on IE9 and earlier versions. All other major browsers are supported.

Authentication

URLs generated by Refile are cryptographically signed. This ensures that a file cannot be downloaded unless you hand someone the URL to that file. This is essentially equivalent to token base authentication.

Unfortunately a similar system is not in place for file uploads, meaning that direct file uploads are open to anyone. By default only the cache backend can be uploaded to, and you are encouraged to purge unused files from this backend periodically. This might seem insecure, but consider the fact that anyone who can access file uploads in your application will be able to upload files into it anyway.

Nevertheless, you can disable direct file uploads by setting:

Refile.allow_uploads_to = []

You also have the option of explicitly authenticating anyone who tries to access the Refile application. Since the Refile application is a Sinatra application, you can use Sinatra's before hooks to set up authentication like this:

Refile::App.before do
  halt 403 unless User.find_by(id: session[:user_id])
end

Additional metadata

In the quick start example above, we chose to only store the file id, but often it is useful to store the file's filename, size and content type as well. Refile makes it easy to extract this data and store it alongside the id. All you need to do is add columns for these:

class StoreMetadata < ActiveRecord::Migration
  def change
    add_column :users, :profile_image_filename, :string
    add_column :users, :profile_image_size, :integer
    add_column :users, :profile_image_content_type, :string
  end
end

These columns will now be filled automatically.

File type validations

Refile can check that attached files have a given content type or extension. This allows you to warn users if they try to upload an invalid file.

Important: You should regard this as a convenience feature for your users, not a security feature. Both file extension and content type can easily be spoofed.

In order to limit attachments to an extension or content type, you can provide them like this:

attachment :cv, extension: "pdf"
attachment :profile_image, content_type: "image/jpeg"

You can also provide a list of content types or extensions:

attachment :cv, extension: ["pdf", "doc"]
attachment :profile_image, content_type: ["image/jpeg", "image/png", "image/gif"]

Since the combination of JPEG, PNG and GIF is so common, you can also specify this more succinctly like this:

attachment :profile_image, type: :image

When a user uploads a file with an invalid extension or content type and submits the form, they'll be presented with a validation error.

If you use a particular content type or set of content types frequently you can define your own types like this:

Refile.types[:document] = Refile::Type.new(:document,
  content_type: %w[text/plain application/pdf]
)

Now you can use them like this:

attachment :profile_image, type: :document

Multiple file uploads

File input fields support the multiple attribute which allows users to attach multiple files at once. Refile supports this attribute. You can add the attribute to your attachment fields like this:

<%= form.attachment_field :images_files, multiple: true %>

Multiple file uploads also work nicely with direct and presigned uploads:

<%= form.attachment_field :images_files, multiple: true, direct: true, presigned: true %>

Note that you will get separate events for each uploaded file. So when you attach two files, the upload:start event and other events will be triggered twice, once for each file.

When you upload multiple files, your application will receive an array of files, instead of a single file. To map these files to model object, Refile's ActiveRecord integration ships with a nice macro that makes this trivial. Suppose you have an image model like this:

class Image < ActiveRecord::Base
  belongs_to :post
  attachment :file
end

Note it must be possible to persist images given only the associated post and a file. There must not be any other validations or constraints which prevent images from being saved.

From the post model, you can use the accepts_attachments_for macro:

class Post < ActiveRecord::Base
  has_many :images, dependent: :destroy, autosave: true
  accepts_attachments_for :images, attachment: :file
end

The attachment option defaults to :file, so we could have left it out in this case.

class Post < ActiveRecord::Base
  has_many :images, dependent: :destroy, autosave: true
  accepts_attachments_for :images
end

Note: Leaving out the autosave option will only save the attachments when the post is created.

You can add the attachment field to your post form:

<%= form_for @post do |form| %>
  <%= form.label :images_files %>
  <%= form.attachment_field :images_files, multiple: true %>
<% end %>

Now you only need to permit the generated accessor in your controller. Since images_files is an array, you need to tell Rails to allow array values for it:

def post_params
  params.require(:post).permit(images_files: [])
end

When editing a record with accepts_attachments_for, the default behaviour is to replace the entire list of attachments when new attachments are uploaded. It is also possible to append the new attachments to the list of attachments instead so that older attachments are kept. To enable this, set the append option to true.

class Post < ActiveRecord::Base
  has_many :images, dependent: :destroy
  accepts_attachments_for :images, append: true
end

Multiple file uploads for pure Ruby classes

You can also use accepts_attachments_for macro in pure Ruby classes.

Suppose you have a Document class to be associated with a Post class. A post has many documents, and each document has a file. First, you will need to use the attachment macro in the Document class to declare your :file attachment, and also implement a constructor to receive the attachment:

class Document
  extend Refile::Attachment
  attr_accessor :file_id

  attachment :file

  def initialize(attributes = {})
    self.file = attributes[:file]
  end
end

In the Post class, you will need a constructor to initialize the @documents variable and an attr_accessor :documents. Then you can use the accepts_attachments_for macro for declaring the :documents collection:

class Post
  extend Refile::Attachment
  include ActiveModel::Model

  attr_accessor :documents

  accepts_attachments_for(
    :documents,
    accessor_prefix: 'documents_files',
    collection_class: Document
  )

  def initialize(attributes = {})
    @documents = attributes[:documents] || []
  end
end

In this example, we specified the following options:

• collection_class is the attachments class, Document
• accessor_prefix gives a prefix to the generated accessors, documents_files.

Now you can append attachments with your HTML form in the following way:

<%= form_for @post do |form| %>
  <%= form.label :documents_files %>
  <%= form.attachment_field :documents_files, multiple: true %>
<% end %>

The default values for the accepts_attachments_for macro are { attachment: :file, append: false }. Everything else should be similar to the Active Record version of this macro.

Single file upload

File input fields support single file upload, allows users to attach one file at the time instead of the common multiple files feature. Let's suppose you have an image model like this:

class Image < ActiveRecord::Base
  belongs_to :post
  attachment :file
end

Note it must be possible to persist images given only the associated post and a file. There must not be any other validations or constraints which prevent images from being saved.

From the post model, you can use the accepts_attachments_for macro:

class Post < ActiveRecord::Base
  has_many :images, dependent: :destroy
  accepts_attachments_for :images, attachment: :file
end

The attachment option defaults to :file, so we could have left it out in this case.

class Post < ActiveRecord::Base
  has_many :images, dependent: :destroy
  accepts_attachments_for :images
end

You can add the attachment field to your post form without using the multiple attribute:

<%= form_for @post do |form| %>
  <%= form.label :images_files %>
  <%= form.attachment_field :images_files %>
<% end %>

Now you only need to permit the generated accessor in your controller. Since images_files is not an array, you need to tell Rails to only allow the attribute symbol:

def post_params
  params.require(:post).permit(:images_files)
end

Removing attached files

File input fields unfortunately do not have the option of removing an already uploaded file. This is problematic when editing a model which has a file attached and the user wants to remove this file. To work around this, Refile automatically adds an attribute to your model when you use the attachment method, which is designed to be used with a checkbox in a form.

<%= form_for @user do |form| %>
  <%= form.label :profile_image %>
  <%= form.attachment_field :profile_image %>

  <%= form.check_box :remove_profile_image %>
  <%= form.label :remove_profile_image %>
<% end %>

Don't forget to permit this attribute in your controller:

def user_params
  params.require(:user).permit(:profile_image, :remove_profile_image)
end

Now when you check this checkbox and submit the form, the previously attached file will be removed.

Fetching remote files by URL

You might want to give you users the option of uploading a file by its URL. This could be either just via a textfield or through some other interface. Refile makes it easy to fetch this file and upload it. Just add a field like this:

<%= form_for @user do |form| %>
  <%= form.label :profile_image, "Attach image" %>
  <%= form.attachment_field :profile_image %>

  <%= form.label :remote_profile_image_url, "Or specify URL" %>
  <%= form.text_field :remote_profile_image_url %>
<% end %>

Then permit this field in your controller:

def user_params
  params.require(:user).permit(:profile_image, :remote_profile_image_url)
end

Refile will now fetch the file from the given URL, following redirects if needed.

Cache expiry

Files will accumulate in your cache, and you'll probably want to remove them after some time.

The FileSystem backend does not currently provide any method of doing this. PRs welcome ;)

On S3 this can be conveniently handled through lifecycle rules. Exactly how depends a bit on your setup. If you are using the suggested setup of having one bucket with cache and store being directories in that bucket (or prefixes in S3 parlance), then follow the following steps, otherwise adapt them to your needs:

• Open the AWS S3 console and locate your bucket
• Right click on it and choose "Properties"
• Open the "Lifecycle" section
• Click "Add rule"
• Choose "Apply the rule to: A prefix"
• Enter "cache/" as the prefix (trailing slash!)
• Click "Configure rule"
• For "Action on Objects" you'll probably want to choose "Permanently Delete Only"
• Choose whatever number of days you're comfortable with, I chose "1"
• Click "Review" and finally "Create and activate Rule"

Testing

When testing your own classes that use Refile, you can use Refile::FileDouble objects instead of real files.

# app/models/post.rb
class Post < ActiveRecord::Base
  attachment :image, type: :image
end

# spec/models/post_spec.rb
require "rails_helper"
require "refile/file_double"

RSpec.describe Post, type: :model do
  it "allows attaching an image" do
    post = Post.new

    post.image = Refile::FileDouble.new("dummy", "logo.png", content_type: "image/png")
    post.save

    expect(post.image_id).not_to be_nil
  end

  it "doesn't allow attaching other files" do
    post = Post.new

    post.image = Refile::FileDouble.new("dummy", "file.txt", content_type: "text/plain")
    post.save

    expect(post.image_id).to be_nil
  end
end

simple_form

simple_form gem is also supported:

# in initializer or Gemfile
require 'refile/simple_form'

# in forms
<%= f.input :cover_image, as: :attachment, direct: true, presigned: true %>

License

MIT

refile-s3's Issues

Getting aws error after modifying image: Aws::S3::Errors::XAmzContentSHA256Mismatch: The provided 'x-amz-content-sha256' header does not match what was computed.

I have a repo that I would like to modify an image using minimagic. The aws sdk returns Aws::S3::Errors::XAmzContentSHA256Mismatch: The provided 'x-amz-content-sha256' header does not match what was computed. after I attempt to reassign the image after making a change to it. The example code is here https://github.com/marr/localpic2/blob/master/app/workers/image_filter.rb#L18 and relevant test https://github.com/marr/localpic2/blob/master/test/workers/image_filter_test.rb#L16

Any help would be appreciated

Mobile number hecking

Unique S3 Buckets

Given I have multiple users, each with a unique Account and each Account has a unique S3 bucket for that account.. How could I designate a specific S3 bucket not inside an initializer? For example, User A has a bucket called user_a_bucket and User XXZX has a bucket called user_xxzx_bucket. Obviously, those can't all go into an initializer. So what's the best strategy to handle that case, given it's not scalable to have each user's bucket names in an initializer.

Thanks!

Handle Missing Config

If the bucket, ID, or secret is missing, the app should refuse to start. Otherwise you only find out if you use presigned: true and load that view.

Rails 5 compatability

Rack gem version dependency needs to be upgraded to allow Rails 5 support

Bundler could not find compatible versions for gem "rack":
  In Gemfile:
    rails (< 5.1, >= 5.0.0.rc1) was resolved to 5.0.0.rc1, which depends on
      actioncable (= 5.0.0.rc1) was resolved to 5.0.0.rc1, which depends on
        actionpack (= 5.0.0.rc1) was resolved to 5.0.0.rc1, which depends on
          rack (~> 2.x)

    refile-s3 was resolved to 0.1.0, which depends on
      refile (~> 0.5.0) was resolved to 0.5.0, which depends on
        sinatra (~> 1.4.5) was resolved to 1.4.5, which depends on
          rack (~> 1.4)

Minimum Ruby version

It's a small thing, but I think you should specify spec.required_ruby_version = '>= 2.1.0' in the gemspec because of verify_uploadable def upload(uploadable). Methods return a symbol only on Ruby >= 2.1.

Error when uploading file from Remote URL

Since upgraded to the new, extracted refile-s3 gem, adding a profile photo via a remote URL gives me this error:

IOError (closed stream):
  app/controllers/users_controller.rb:52:in `block in update'

Traditional uploads via uploading a local file from my computer works fine

Unable to verify Ceph server identity when using self-signed cert

I've been able to successfully use refile-s3 to upload files to a private Ceph that uses a self-signed certificate by setting this before I initialize Refile:

Aws.config[:ssl_ca_bundle] = '/path/to/self-signed-cert.pem'

Now I'm running into an issue when attempting to download the same file because the gem uses OpenURI::OpenRead#open (https://github.com/refile/refile-s3/blob/master/lib/refile/s3.rb#L98) which doesn't know about the above cert. That method has an option to specify a cert via the :ssl_ca_cert option as listed in http://ruby-doc.org/stdlib-2.1.0/libdoc/open-uri/rdoc/OpenURI/OpenRead.html#method-i-open.

Would it make sense to modify Refile::S3#open to pick up Aws.config[:ssl_ca_bundle] if it's defined and then pass it on to OpenURI::OpenRead#open?

Not compatible with `aws-sdk v3`

I just upgraded to aws-sdk v3 and got a compatibility error.
As per the current master gemspec, they have locked the aws-sdk to version 2.

Can't install with Refile 0.6.0

I noticed Refile was tagged with version 0.6.0 today. This gem has a dependency of '~> 0.50' and therefore won't install with it. Can we safely bump that dependency?

Multipart copy

For files > 5GB in size, the S3 API requires using a multipart copy. I recently had to work around this by monkeypatching Refile, but I'd love to have official support for it.

Happy to work on it as well if you have an idea of how it should be implemented.

I ended up overwriting #upload to pass content_length and use_multipart_copy to #copy_from, but that just enables multipart for every copy. Not sure if that is desirable for Refile in general.

TypeError: 0 is not a symbol nor a string

Hi,
I am trying to use the refile/s3 gem and having some trouble during the upload. I have a test here in pictures_controller_test.rb https://github.com/marr/localpic2

The stacktrace doesn't show much more than this:

Api::V1::PicturesControllerTest#test_should_post_create:
TypeError: 0 is not a symbol nor a string
    app/controllers/api/v1/pictures_controller.rb:22:in `create'
    test/controllers/api/v1/pictures_controller_test.rb:17:in `block in <class:PicturesControllerTest>'

Any help would be appreciated, cc @hiromipaw

Retry download at least once

Sometimes the S3 timeouts the connection and then refile raises an exception because of that:

Errno::ETIMEDOUT

Connection timed out - connect(2) for "my-bucket.s3.amazonaws.com" port 443

I've seen this error a couple times before and just retrying the connection it works.

What you think?

We could retry at least once to ensure that is not a temporary failure.

SVG Support

Guys, I have to create a issue for this, because I couldn't find anything about it.

Well, I am using Rails 4 and Ruby 2.2.4 and I am uploading a SVG file to a S3 bucket.

I can upload the file and display correclty the SVG file while it's on cache folder, but when it's moved to store folder and lose the extension I try to display the image using the attachment_url(, :logo, filename: :svg) it loses the XML content for SVG file.

How can I use SVG files with Refile?

Migrate from Refile to ActiveStorage

Hi guys. We’re in the middle of migrating from Refile to ActiveStorage. We’ve been using Refile for the past 3-4 years now but recently decided to upgrade our Rails 5 application to Rails 6 and along comes ActiveStorage. Everything worked fine with Refile until we started to implement Action Mailbox which relies on ActiveStorage. To store the inbound emails we need to use the aws-sdk-s3 gem but when I run bundle install it can't find any compatible versions for aws-sdk-core:

Bundler could not find compatible versions for gem "aws-sdk-core":
  In snapshot (Gemfile.lock):
    aws-sdk-core (= 3.105.0)

  In Gemfile:
    refile-s3 was resolved to 0.2.0, which depends on
      aws-sdk (~> 2.0) was resolved to 2.0.1.pre, which depends on
        aws-sdk-resources (= 2.0.1.pre) was resolved to 2.0.1.pre, which depends on
          aws-sdk-core (= 2.0.1)

    aws-sdk-s3 was resolved to 1.79.1, which depends on
      aws-sdk-core (~> 3, >= 3.104.3)

Gemfile

gem 'refile', require: 'refile/rails', github: 'manfe/refile'
gem 'refile-memory'
gem 'refile-mini_magick'
gem 'refile-s3'

gem 'aws-sdk-s3', require: false

In order to migrate the data from Refile to ActiveStorage we need to have the two run alongside each other in the beginning.

Does anyone know what I'm doing wrong?

Bump aws-sdk version

Please update the aws sdk gem

Bundler could not find compatible versions for gem "aws-sdk-core":
  In snapshot (Gemfile.lock):
    aws-sdk-core (= 3.54.2)

  In Gemfile:
    dynamoid (~> 3.2) was resolved to 3.2.0, which depends on
      aws-sdk-dynamodb (~> 1) was resolved to 1.30.0, which depends on
        aws-sdk-core (~> 3, >= 3.53.0)

    refile-s3 was resolved to 0.2.0, which depends on
      aws-sdk (~> 2.0) was resolved to 2.0.1.pre, which depends on
        aws-sdk-resources (= 2.0.1.pre) was resolved to 2.0.1.pre, which depends on
          aws-sdk-core (= 2.0.1)

aws_access_id deprecation

You can see in the AWS::CredentialProvider docs the use of access_key_id is deprecated and being removed in the next dot release.

Location of the warning: https://github.com/refile/refile-s3/blob/master/lib/refile/s3.rb#L43-L54.

From Gemfile.lock:

    aws-sdk (2.1.23)
      aws-sdk-resources (= 2.1.23)
    aws-sdk-core (2.1.23)
      jmespath (~> 1.0)
    aws-sdk-resources (2.1.23)
      aws-sdk-core (= 2.1.23)
...
    refile (0.6.1)
      mime-types
      rest-client (~> 1.8)
      sinatra (~> 1.4.5)
    refile-mini_magick (0.2.0)
      mini_magick (~> 4.0)
      refile (~> 0.5)
    refile-s3 (0.2.0)
      aws-sdk (~> 2.0)
      refile (~> 0.6.0)

From rails c:

DEPRECATION WARNING: called deprecated method `access_key_id' of an Aws::CredentialProvider, use #credentials instead
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/refile-s3-0.2.0/lib/refile/s3.rb:48:in `initialize'
/srv/www/kast_app/releases/20151017150030/config/initializers/refile.rb:10:in `new'
/srv/www/kast_app/releases/20151017150030/config/initializers/refile.rb:10:in `<top (required)>'
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:268:in `load'
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:268:in `block in load'
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:240:in `load_dependency'
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:268:in `load'
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/railties-4.2.3/lib/rails/engine.rb:652:in `block in load_config_initializer'
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/activesupport-4.2.3/lib/active_support/notifications.rb:166:in `instrument'
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/railties-4.2.3/lib/rails/engine.rb:651:in `load_config_initializer'
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/railties-4.2.3/lib/rails/engine.rb:616:in `block (2 levels) in <class:Engine>'
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/railties-4.2.3/lib/rails/engine.rb:615:in `each'
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/railties-4.2.3/lib/rails/engine.rb:615:in `block in <class:Engine>'
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/railties-4.2.3/lib/rails/initializable.rb:30:in `instance_exec'
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/railties-4.2.3/lib/rails/initializable.rb:30:in `run'
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/railties-4.2.3/lib/rails/initializable.rb:55:in `block in run_initializers'
/usr/local/lib/ruby/2.2.0/tsort.rb:226:in `block in tsort_each'
/usr/local/lib/ruby/2.2.0/tsort.rb:348:in `block (2 levels) in each_strongly_connected_component'
/usr/local/lib/ruby/2.2.0/tsort.rb:420:in `block (2 levels) in each_strongly_connected_component_from'
/usr/local/lib/ruby/2.2.0/tsort.rb:429:in `each_strongly_connected_component_from'
/usr/local/lib/ruby/2.2.0/tsort.rb:419:in `block in each_strongly_connected_component_from'
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/railties-4.2.3/lib/rails/initializable.rb:44:in `each'
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/railties-4.2.3/lib/rails/initializable.rb:44:in `tsort_each_child'
/usr/local/lib/ruby/2.2.0/tsort.rb:413:in `call'
/usr/local/lib/ruby/2.2.0/tsort.rb:413:in `each_strongly_connected_component_from'
/usr/local/lib/ruby/2.2.0/tsort.rb:347:in `block in each_strongly_connected_component'
/usr/local/lib/ruby/2.2.0/tsort.rb:345:in `each'
/usr/local/lib/ruby/2.2.0/tsort.rb:345:in `call'
/usr/local/lib/ruby/2.2.0/tsort.rb:345:in `each_strongly_connected_component'
/usr/local/lib/ruby/2.2.0/tsort.rb:224:in `tsort_each'
/usr/local/lib/ruby/2.2.0/tsort.rb:203:in `tsort_each'
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/railties-4.2.3/lib/rails/initializable.rb:54:in `run_initializers'
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/railties-4.2.3/lib/rails/application.rb:352:in `initialize!'
/srv/www/kast_app/releases/20151017150030/config/environment.rb:5:in `<top (required)>'
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:274:in `require'
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:274:in `block in require'
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:240:in `load_dependency'
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:274:in `require'
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/railties-4.2.3/lib/rails/application.rb:328:in `require_environment!'
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/railties-4.2.3/lib/rails/commands/commands_tasks.rb:142:in `require_application_and_environment!'
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/railties-4.2.3/lib/rails/commands/commands_tasks.rb:67:in `console'
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/railties-4.2.3/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
/home/deploy/.bundler/kast_app/ruby/2.2.0/gems/railties-4.2.3/lib/rails/commands.rb:17:in `<top (required)>'
bin/rails:4:in `require'
bin/rails:4:in `<main>'

Missing bucket and bucket_name

Ran into a rather weird error today.

aws = {
  access_key_id: "xxx",
  secret_access_key_holder: "xxx",
  bucket: "xxx",
  region: "us-east-1",
}

[2] pry(main)> Refile::S3.new(prefix: "store", **aws)
=> #<Refile::S3:0x007fc30f4bc790
 @access_key_id="xxx",
 @bucket=,
 @bucket_name=,
 @hasher=#<Refile::RandomHasher:0x007fc30f4bc6a0>,
 @max_size=nil,
 @prefix="store",
 @s3=#<Aws::S3::Resource>,
 @s3_options={:region=>"us-east-1", :access_key_id=>"xxx", :secret_access_key=>"xxx"}>

As you can see, the @bucket and @bucket_name is empty even when bucket parameter has been supplied in the hash.

Funnily, changing :secret_access_key to something else, e.g. ::secret_access_key_temp makes the bucket & bucket_name reappear.

Server side encryption configuration option

Older Refile version, using AWS SDK V1, supported configuring server side encryption via the s3_options per S3 backend / bucket.

https://github.com/refile/refile/blob/bdc1fead72747a18f7120189d860f6368dbdc81e/lib/refile/backend/s3.rb#L37

AWS SDK V2 doesn't support configuring this option on the Aws::S3::Resource object.

https://github.com/refile/refile-s3/blob/master/lib/refile/s3.rb#L40

I think being able to define the encryption per bucket would be a rather essential feature. AWS SDK V2 requires this to be passed as part of the options argument for copy_from, put and presigned_post methods (for example server_side_encryption: 'aes256').

https://github.com/refile/refile-s3/blob/master/lib/refile/s3.rb#L56
https://github.com/refile/refile-s3/blob/master/lib/refile/s3.rb#L58
https://github.com/refile/refile-s3/blob/master/lib/refile/s3.rb#L140

http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Object.html#copy_from-instance_method
http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Object.html#put-instance_method
http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Object.html#presigned_post-instance_method

IMO these are some of the other "static" options that could be a deal breaker for some, but not relevant to my use case:

• storage_class
• sse_customer_algorithm
• sse_customer_key
• sse_customer_key_md5
• ssekms_key_id

add the possibility to use iam role instead of explicit using aws access keys

"key does not exist" after direct S3 upload

I'm running into issues with direct S3 uploads where the application is trying to save the model after the upload is complete but before the file upload has fully propagated across S3.

Is there any way to specify the number of retries or a timeout period when trying to copy from cache to store here https://github.com/refile/refile-s3/blob/master/lib/refile/s3.rb#L56?

Namespacing and structure

Food for thought

I'd like to think that this gem is the reference implementation for Refile 3rd party backend gems. From that point of view the current structure suggests placing backend implementations directly under Refile namespace is the preferred practice.

Hence I'd like to propose the following structure to avoid promoting pollution of the root Refile namespace by silly backend implementors releasing their own gems:

• /lib/refile/s3.rb
  • defines Refile::S3 module
  • requires refile/s3/version and refile/backend/s3
• /lib/refile/s3/version.rb
  • defines Refile::S3::VERSION
• /lib/refile/backend/s3.rb
  • implements the backend in Refile::Backend::S3 class

PS. I hope the tone of my post does not come through as patronizing - I'm certain @jnicklas has way more experience in structuring and gardening open source projects than I do. 😇

Pull request to fix the case with "cache" and "store" located in different buckets

In case with cache and store located in different buckets, the copy_source must refer to correct bucket_name.

In my pull request I fixed this issue #9

Please consider to merge that pull request.

My kindest regards,
Stan