choonkeat / attache

Yet another approach to file upload

No Plan.

attache's Issues

Support for upload via url

A similar feature available in Carrierwave is to supply a url, and the server would download the image for you. This is helpful when building crawlers.

The interface with the attache's API design philosophy might need a bit of thought.

Wasabi S3

I'm trying to get attache to work with wasabi ( https://wasabi.com ) as the store rather than amazon AWS.

I tried changing out the details in the vhost.yml file with my fingers crossed:

"images.myapp.com":
  "SECRET_KEY": ****************
  "FOG_CONFIG":
    "provider": AWS
    "aws_access_key_id": *************
    "aws_secret_access_key": **************
    "bucket": my-bucket
    "region": us-east-1
    "host": 's3.wasabisys.com'
  "BACKUP_CONFIG":
    "bucket": CHANGEME_BAK
    # only supports 1 key: `bucket`

but its rarely that simple(and didn't work)..

So I'm looking to find out - is it possible to use Wasabi? According to them it should be a drop in replacement for AWS.

How would I go about getting this to work?

Thanks,

johnny

Need to specify exactly what versions are "available"; being able to request for any size means service is open to DOS

Proposal 1 add GEOMETRY_WHITELIST key to vhost config, e.g.

"aws.example.com":
  "GEOMETRY_WHITELIST":
    - "64x64#"
    - "400x>"
    - "1024x>"

Though it may appear we regress into "predefined styles" land, but this is a late binding (as oppose to upfront early binding like Paperclip et al)

Proposal 2 Require all the resize urls be signed, so arbitrary sizes cannot be requested at all.

Impact on caching? browser js ux (upload and request for preview image)?

@janko-m @laurenceputra

[operations] change the image size on a high traffic production site

Though we can "pre-generate" (more accurately: warm the cache) sizes by tricking the browser to request for those sizes upon author's upload, but for the scenario where site design changes, we're back to cold cache problem.

For reference: for upload solutions that pre-generate thumbnails, the process is to

1. add new sizes to the current "array of sizes to pre-generate" and deploy (new sizes are defined for new uploads to generate but not used in views yet)
2. run offline process to iterate through all existing images to generate new thumbnails
3. make changes to views (html), remove obsoleted sizes from "array of sizes to pre-generate" and deploy
4. run offline process to iterate through all existing images and remove files in obsoleted size

@janko-m FYI

Running attach as a gem: Issues with config files

I tried running attache as a gem and I found a few issues with that workflow. The main issue is that there is no distinction between current project files and attache generated files.

Attache creates a few config files in the root of the project (Procfile, config/puma.rb...). However, if those files already exist (which is quite likely with a Rails app), attache just prints out a message like Using Procfile and tries to use the existing files - which probably are not compatible and attache fails.

I think a solution would be either to namespace the files like Procfile.attache or create a separate attache folder with all the config files.

Another small issue is that the gem dependencies are constrained so I might have to downgrade some gems to use attache as a gem. Eg: Sidekiq 4.0

Support rectangle crop plus resizing

Use case:

Allow user to upload an image, and then select a small section (crop + resize)

It seems that u're using PaperClip, according to the code here, although PaperClip use ImageMagic, it doesn't directly support crop image at a rectangle area.

One vhost but directories for multi-tenancy

https://github.com/choonkeat/attache/blob/master/config/vhost.example.yml

According to the config example, tenants are separated by domain.

I want just one domain https://storage.mycompany.com and then to use the directory path to distinguish tenants with distinct secrets and fog configs.

e.g.

https://storage.mycompany.com/myapp1
https://storage.mycompany.com/myapp1-staging
https://storage.mycompany.com/myapp2
https://storage.mycompany.com/myapp2-qa
https://storage.mycompany.com/...

Is this supported?

Latest 2.3.0 build doesn't run

~ » docker pull attache/attache
Using default tag: latest
latest: Pulling from attache/attache

efd26ecc9548: Already exists
a3ed95caeb02: Pull complete
d1784d73276e: Already exists
72e581645fc3: Already exists
9709ddcc4d24: Already exists
7013ecfd652c: Already exists
2bf97a8a48ea: Pull complete
689bf20a6192: Pull complete
82d4d971868c: Pull complete
14e5f7580b95: Pull complete
a56f8ab1730b: Pull complete
ae8b9f4ffaf4: Pull complete
49b3061d760a: Pull complete
74a6c7691c48: Pull complete
5bff24061d98: Pull complete
0e4fe6d00e8d: Pull complete
475e6f9cafab: Pull complete
e83ed25fa20a: Pull complete
c7fe10f1933d: Pull complete
1c80ee77a362: Pull complete
Digest: sha256:0012bfa34b4cb2095193903d0f91bf6e6185f0d7893b7fb6a1b2a368515ed363
Status: Downloaded newer image for attache/attache:latest
~ » docker run -it --rm attache/attache
Initialized config/vhost.yml
Initialized config/puma.rb
Initialized Procfile
Initialized Rakefile
Initialized config.ru
16:46:45 web.1    | started with pid 7
/usr/local/bundle/gems/foreman-0.80.1/lib/foreman/engine.rb:442:in `check_for_termination': undefined method `exitstatus' for nil:NilClass (NoMethodError)
    from /usr/local/bundle/gems/foreman-0.80.1/lib/foreman/engine.rb:420:in `block in wait_for_shutdown_or_child_termination'
    from /usr/local/bundle/gems/foreman-0.80.1/lib/foreman/engine.rb:415:in `loop'
    from /usr/local/bundle/gems/foreman-0.80.1/lib/foreman/engine.rb:415:in `wait_for_shutdown_or_child_termination'
    from /usr/local/bundle/gems/foreman-0.80.1/lib/foreman/engine.rb:60:in `start'
    from /usr/local/bundle/gems/foreman-0.80.1/lib/foreman/cli.rb:41:in `start'
    from /usr/local/bundle/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
    from /usr/local/bundle/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
    from /usr/local/bundle/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
    from /usr/local/bundle/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
    from /usr/local/bundle/gems/attache-2.3.0/exe/attache:61:in `<top (required)>'
    from /usr/local/bundle/bin/attache:22:in `load'
    from /usr/local/bundle/bin/attache:22:in `<main>'
~ »

LoadError: cannot load such file -- net/ssh

Actually not a bug in attache per se but attache will throw an error on run:

/.rvm/gems/ruby-2.2.3/gems/fog-1.35.0/lib/fog/joyent/compute.rb:3:in `require': cannot load such file -- net/ssh (LoadError)

Upon checking, the issue will happen if using fog-core ~> 1.33.0. It is caused because the net-ssh dependency was removed from fog-core in 1.33.0 but is still required by a component in fog. This has been fixed in fog/fog@ebfaa0c but still not released.

Ref similar issue on fog: fog/fog#3765

For a temporary fix, I am getting attache from the source code and edit the attache.gemspec to include net-ssh before fog:

...
s.add_runtime_dependency 'puma', '~> 2.14'
s.add_runtime_dependency 'net-ssh'
s.add_runtime_dependency 'fog', '~> 1.34'
...

metadata json response after upload is susceptible to malicious edit

the json is populated plainly in the html form (as hidden fields) - and these fields are stored in the db. thanks for @janko-m for headsup on this vector of attack

to prevent malicious changes to the json in the hidden field, we should sign the json to give the rails app confidence that the data is unmodified from attache server

Config attache with S3

I have created a vhost.yml file like this:

"localhost:9292":
  "SECRET_KEY": MY_KEY_HERE
  "FOG_CONFIG":
    "provider": AWS
    "aws_access_key_id": MY_KEY_ID
    "aws_secret_access_key": SAME_AS_SECRET_KEY
    "bucket": MY_BUCKET
    "region": us-west-2

I want to test this on my local machine first. After I successfully migrate my database from paperclip to attache, which means my image json is correct, I found, attache is still using local disk as storage, rather than fetching missing images from S3.

Any idea?

Current docker image doesn't work

It's trying to copy a vhost.example.yml file that doesn't exist because for some reason it's running off v1.0.0.

/usr/local/lib/ruby/2.2.0/fileutils.rb:1391:in `initialize': No such file or directory @ rb_sysopen - /usr/local/bundle/gems/attache-1.0.0/config/vhost.example.yml (Errno::ENOENT)

When I try to hack up my own Dockerfile changing the Gemfile generation to this:

ENV ATTACHE_VERSION 2.2.1

RUN gem install --no-ri --no-rdoc attache --version $ATTACHE_VERSION
RUN mkdir -p /app/src
WORKDIR /app/src
RUN echo 'source "https://rubygems.org"' > Gemfile
RUN echo "gem 'attache', '$ATTACHE_VERSION'" >> Gemfile && bundle

It gives me lovely bundler errors:

/usr/local/bin/bundle:22:in `load': cannot load such file -- /usr/local/lib/ruby/gems/2.3.0/gems/bundler-1.11.2/lib/gems/bundler-1.11.2/bin/bundle (LoadError)

Public docker images have no tags either so we can't roll back to anything.

When images did not get backup to config.storage

Upload was deem slow when uploads were backed up to cloud (e.g. s3) synchronously: file transfer from browser to server then server to s3. This would result in a timeout error for uploads, esp when hosted on heroku.

Thus the logic was https://github.com/choonkeat/attache/blob/master/lib/attache/upload.rb#L23-L28

• first, the file is written to local cache (and would fail hard if that didn't happen)
• then the browser would get a thumbsup and everything proceed
• background job will upload that file to cloud

So now, if the background job fail for some reason, after retries, what is the recourse?

Image cache

Hi, I'm new to this but from the docs it's not overly clear how to handle image caching, specifically cache-busting - if I replace an image with a different image (same filename) and request the same dimensions, how can I bust the cache in this scenario to ensure the new image is delivered?

Thanks in advance.

Allow to require hmac for downloads

It would be great if one could configure in then host file that he requires hmac authorized requests when downloading a file.

Combined with #17

One could define two vhost entries:

https://storage.mycompany.com/myapp1/public
Where hmac is not required for the downloads

And https://storage.mycompany.com/myapp1/private
Where hmac is required for the downloads

Both vhost entries would need to share the same secret to allow a unified upload and hmac signing mechanism.

Re-usable gem or what's a good means of distribution?

Currently this repo is not really a library to be included in another app. It is "checkout the code and run as an app" with a Deploy to Heroku button https://github.com/choonkeat/attache#readme

• I've considered making an entry in https://hub.docker.com and maybe will do that.
• Would distributing as a gem be useful? I guess attache init to setup the current directory and then attache run to run off the current directory (and config)?

fyi @kvz didn't want to pollute the other issue ;-)

401 unauthorized

Hi,
cannot make aws s3 configuration work - any clues ?
the upload starts - then i get a 401 unauthorized error in the form.
on the server logs nothing happens.

any clue ?

thanks !

An attacker can flood the storage by direct uploading files

Though attache presign uploads offers the same protection as S3 direct upload

The pre-signed URLs are valid only for the specified duration.

Within that duration, an attacker can still upload as many files.

To mitigate that, we can adopt the refile and shrine procedure of always uploading to cache then promote to store only when the client app sends a confirmation ping

Current proposal is for /promote to mimic the /delete endpoint

• require a pre-signed HTTP POST (but valid duration must be pretty short, like < 30s)
• params include a list of filenames to confirm (batch operation)
• image server perform promotion async, responds to client app immediately

@janko-m if async promotion fails in the background, what does a shrine user do?