shrinerb / shrine-url Goto Github PK

To avoid user-submitted data being able to download from any arbitrary url?

When url to the remote file is specified by https protocol then EOFError: end of file reached is raised from the Net::HTTP

Per the README's note that "Another advantage is that you can assign multiple remote URLs as multiple versions," is there a recommended way to do that that won't trigger the deprecation warning in assign_cached in the versions plugin?

https://github.com/shrinerb/shrine/blob/1f11094ac9e4966e86d6ca22b5af2b9161de272c/lib/shrine/plugins/versions.rb#L291

Hey @janko-m ,
Thanks for this gem - it's super useful but I have a one question - is it possible to support redirects? It would especially useful for links from dropbox like: https://www.dropbox.com/s/992urij4brdzckd/test_pdf.pdf?dl=1

Unfortunately those links redirect to single use URL so basically every call to exists?, download, etc. would require to generate a new link by following its redirect. I think it needs to remember how many redirect it should perform for download - otherwise it can invalidate the link by checking the response code, i.e.:

https://www.dropbox.com/s/992urij4brdzckd/test_pdf.pdf?dl=1 -> redirects to A

if you made a request to A (i.e. to check for additional redirect), you invalidate the A link so you can not use it to download the file.

i'm trying to use shrine-url with a signed S3 URL. Which is a pretty convenient usecase, importing from S3 (And I def want to do it in backgrounding, thus shrine-url).

However, it fails on too-long URLs, and signed S3 URLs are quite long (with query params).

The stack trace doesn't even involve shrine-url, I think it's maybe failing before it even gets to calling shrine url. Based on trying to make a filename based on id, which is the long URL.

Error performing Kithe::AssetPromoteJob (Job ID: 0979ac75-3611-4e99-a63a-be6f5db54957) from Async(default) in 5.67ms: Errno::ENAMETOOLONG (File name too long @ rb_sysopen - /Users/jrochkind/code/scihist_digicoll/tmp/shrine_storage_testing/store/cc65f8b0ebe79f859195410a8a71b025.tif?x-amz-algorithm=aws4-hmac-sha256&x-amz-credential=akiajorcl6txivhsdrhq%2f20181119%2fus-east-1%2fs3%2faws4_request&x-amz-date=20181119t182037z&x-amz-expires=14400&x-amz-signedheaders=host&x-amz-signature=fakefake45db944222adfadfadfadf7d20eacce3e6d062979f993db0ffa1e6e):
/Users/jrochkind/.gem/ruby/2.5.3/gems/shrine-2.13.0/lib/shrine/storage/file_system.rb:129:in `initialize'
/Users/jrochkind/.gem/ruby/2.5.3/gems/shrine-2.13.0/lib/shrine/storage/file_system.rb:129:in `copy_stream'
/Users/jrochkind/.gem/ruby/2.5.3/gems/shrine-2.13.0/lib/shrine/storage/file_system.rb:129:in `upload'
/Users/jrochkind/.gem/ruby/2.5.3/gems/shrine-2.13.0/lib/shrine.rb:348:in `copy'
/Users/jrochkind/.gem/ruby/2.5.3/gems/shrine-2.13.0/lib/shrine.rb:337:in `put'
/Users/jrochkind/.gem/ruby/2.5.3/gems/shrine-2.13.0/lib/shrine.rb:321:in `_store'
/Users/jrochkind/.gem/ruby/2.5.3/gems/shrine-2.13.0/lib/shrine.rb:248:in `store'
/Users/jrochkind/.gem/ruby/2.5.3/gems/shrine-2.13.0/lib/shrine/plugins/rack_file.rb:72:in `store'
/Users/jrochkind/.gem/ruby/2.5.3/gems/shrine-2.13.0/lib/shrine.rb:226:in `upload'
/Users/jrochkind/.gem/ruby/2.5.3/gems/shrine-2.13.0/lib/shrine/plugins/rack_file.rb:66:in `upload'
/Users/jrochkind/.gem/ruby/2.5.3/gems/shrine-2.13.0/lib/shrine.rb:658:in `store!'
/Users/jrochkind/.gem/ruby/2.5.3/gems/shrine-2.13.0/lib/shrine.rb:588:in `promote'
/Users/jrochkind/.gem/ruby/2.5.3/gems/shrine-2.13.0/lib/shrine/plugins/backgrounding.rb:152:in `promote'

I would try to add a failing test... but I notice now shrine-url doesn't even have any tests, so not sure where to add it. Also no idea how to fix this, since it happens before it even gets to shrine-url. I guess it would have to be fixed in shrine itself, to not try to make local files based on id... but it's not expecting id to be super-long like this, the way shrine-url uses it. I wonder if the whole design of shrine-url should be rethought?

Or wait, maybe it looks like the part of shrine that's extracting the filename suffix (to preserve it as it gets promoted), needs to be smart enough to ignore query parameter at the end, in order to be compatible with shrine-url? That might be it?

If you give me feedback and advice, I can try to send some PRs!

I like using WebMock in my tests.

When trying to write my own test against a shrine uploader using shrine-url, and using WebMock to mock out the HTTP request that shrine-url would be doing...

I get a mysterious error. I haven't yet been able to isolate this error into something that only uses down and not shrine-url, is why I'm reporting it here. It's really quite mysterious and I'm not sure what's going on, when I try to isolate it into a simple reproduction case, i end up not being able to reproduce.

Here's the error/stack trace as reported by my rspec, in case it gives you any ideas

     Failure/Error: response.connection.close unless @client.persistent?

     NoMethodError:
       undefined method `close' for #<HTTP::Response::Streamer:0x00007ff14aa89380>
       Did you mean?  clone
     # /Users/jrochkind/.gem/ruby/2.5.3/gems/down-4.7.0/lib/down/http.rb:109:in `stream_body'
     # /Users/jrochkind/.gem/ruby/2.5.3/gems/down-4.7.0/lib/down/chunked_io.rb:282:in `each'
     # /Users/jrochkind/.gem/ruby/2.5.3/gems/down-4.7.0/lib/down/chunked_io.rb:282:in `block in chunks_fiber'

(Does this also suggest it's using http_rb instead of net_http? Ah, I see, even though down says net-http is the default, shrine-url makes http_rb it's default. Any particular rationale?)