Status of chunking and resuming uploads about shrine

[janko]

It is done: https://github.com/janko-m/tus-ruby-server

I also created a demo app which integrates it with Shrine: https://github.com/janko-m/shrine-tus-demo

[janko]

I completely understand if you cannot use S3 or Azure; the goal of Shrine is to work for everyone, so we'll figure something out 😃

There is something called "chunked transfer encoding" which is part of HTTP 1.1, however, while it does allow sending data in chunks, I'm almost 100% sure that it doesn't let you specify the order. This means that you cannot upload chunks in parallel, or retry a chunk.

So, to give you more insight about RubyTus, it's a Rack application that accepts file uploads with the TUS protocol, and the end result is a file on the filesystem. That means that you can define Shrine's :cache directory to be the one where RubyTus saves the files, and then on client-side you can just add the location into Shrine's JSON representation of an attachment, the same way you do with direct S3 uploads. So if we can upload files to RubyTus, it should be good, right?

I think FineUploader wouldn't work in that case, as its protocol is pretty much sealed. Maybe there are apps which implement uploads with this protocol, just like RubyTus does for TUS. But instead of FineUploader you could just use a JS file upload library which supports TUS, probably best to go with the official tus-js-client.

Do you think that could work for you?

[openscript]

I've added a comment about the progress with the TUS implementation in FineUploader:
FineUploader/fine-uploader#1620 (comment)

[janko]

@openscript Just to report the status of the Shrine TUS demo.

So, Rubytus at the moment doesn't work with tus-js-client, I'm trying to fix it now. It is failing on the CORS preflight request, and I noticed that Rubytus was using old headers, so I tried copying over CORS headers from tusd, but for some reason it's still failing, I need to figure out why.

In the meanwhile I tried using tusd (a Go server), since it's the official one, and it's mature and up-to-date. And I managed to get things working. To really test resumability I tried using S3 backend for tusd (so that it's really uploading remotely), and it worked as well. However, the resumability didn't work, because tus-js-client only seems the remember the file once it's fully uploaded (it makes a request to tusd with the wrong ID when trying to resume). That's probably just a bug in tus-js-client, but that is irrelevant if we're going to use FineUploader.

So, once the TUS server finishes uploading, it returns the URL to the file, and I use shrine-url to store that URL as a cached file, which can then be promoted to any Shrine storage. So, the only part that's not working properly is uploading itself, which is the part that's unrelated to Shrine.

[janko]

Thank you for asking this, now I will not close this issue like Paperclip did, until we reach a satisfying result 😃

You're right, Shrine at the moment doesn't support chunked or resumable uploads. This is something that I thought about in the past, however, my conclusion was it makes much more sense to direct upload to an external service which does support these things.

Amazon S3 supports multipart uploads, and with that resumable uploads as well. From what I can see, FineUploader has support for uploading files directly to S3 and, unlike jQuery-File-Upload, it can do so in chunks by utilizing S3's multipart API. You can see Direct Uploads to S3 on how to setup direct S3 uploads with Shrine, along with some client-side tips. Would that be feasible for you?

In general I'm not closed to the idea of implementing chunked/resumable uploads in Shrine, but in that case instead of implementing a library-specific protocol like FineUploader imposes, it would be much better to implement TUS. Fortunately, there is already rubytus gem which gives you an app that implements the TUS protocol, storing files on the filesystem, and the author says that his company has been using it in production since 2013 (tus/tus.io#28 (comment)).

Let me know if any of these options would work for you.

[openscript]

Wow! Such an immediate response. Thank you :-)

Using S3 or Azure is not an option for this project, because of the data protection law, the project needs to be hosted in a specific country.

As far as I understand chunked uploads are specified in HTTP 1.1, whereas resumable uploads are not specified. Unfortunately FineUploader seems to use a library-specific protocol for chunked and resumable uploads.

Probably I need chunking, because of very big files. Then resuming is also super useful. I don't know yet, what I going to do.

[openscript]

Sorry for not responding lately. The weather was great and my motorbike smiled at me :-)

I really like the idea to use the well specified TUS protocol and I would love to go that way, if it doesn't consume too much time. I've been doing some research and I didn't find a popular uploading framework, which supports TUS, even though many of them have requests for TUS in their issue trackers (Plupload, FineUploader)

So I see two possibilities, which could work:

1. We could work together on that. We implement the support for TUS in Shrine and we implement in one of the JS uploading frameworks support for TUS. I think this would be a benefit for Shrine and makes it more popular, but I think it's more time consuming than the other possibility below. As I see we live in the same timezone and I've some time for that on the upcoming weekends. It would be fun for me doing something like that. The TUS protocol is already implemented in JavaScript and it doesn't have dependencies. I think it would be quite easy to implement it into FineUploader.
2. I implement the FineUploader protocol as a plugin in Shrine. I think it's not impossible and I estimate 20 hours of time for doing that. I'm so into FineUploader, because it's already in use in another project.

I've implemented the uploading today in my project with Shrine and FineUploader, but of course without chunking and resuming.

What do you think of all that?

[rnicholus]

Hey there, Fine Uploader developer here. The case where TUS support has been discussed in Fine Uploader can be found at FineUploader/fine-uploader#1620. In there, you'll also see historical information that explains why FU did not use TUS (TL;DR - chunking in FU came before TUS).

While it would be great to add optional TUS support to FU, I think the easier option, in the context of this case, is to add support for Fine Uploader chunked requests to shrine. Then again, I'm not familiar with shrine, so I could be way off-base with that assertion. Either way, I'm available to provide support and advice on the Fine Uploader side of things.

[janko]

@openscript If we decide to go for the 1st option, there is actually nothing that needs to be done on the Ruby side, because as I mentioned there is already a production-ready implementation of TUS, Rubytus. I think there would we very little benefit in writing a new TUS implementation as a Shrine plugin; you could just do direct uploads like to any other service. IMO it's much better to use an existing well-tested solution, and it can easily be hooked up with Shrine.

If we go with the 2nd option (@rnicholus thank you very much for your input), I think it would be also great to implement a generic server (like Rubytus), something like "fineupload-ruby-server". It would just be a Rack application (I personally prefer Roda, but Rubytus chose Goliath), which could then be either mounted in your router or run separately. I think it would cool to have that, as FineUploader looks like a really advanced JS file upload library.

Let me know if you would like to take a stab at it. Otherwise I could try taking a look into it the next week. Or we could work on it together; it's been so long since I last pair programmed 😃

Since this is technically not related to Shrine, I will close this issue. But feel free to continue the discussion here, or we can also switch to email.

[rnicholus]

My Ruby skills are pretty non-existent rusty. But if any FU-related questions come up, please do let me know. If this is a generic ruby server, perhaps this can be developed as a repo in the Fine Uploader org.

[janko]

@rnicholus Thanks, if we decide to build a ruby server for FineUploader, I also think that the "FineUploader" organization would be a great place for it.

I think in the long run FineUploader would greatly benefit from having support for TUS, because it's a standalone and fully specified protocol, and really many people collaborated to bring it to 1.0, with contributions from companies like Vimeo. I think it's great to agree on a generic protocol and make servers for it, and then have different JS libraries which support that same protocol.

@openscript However, FineUploader's protocol is also nicely specified, and many server implementations exist as well. And since FineUploader seems to have significant advantages over libraries like tus-js-client, it might be worthwhile to implement a Ruby server for it. Even though that also sounds like a very decent amount of work, for me personally it would be easier, because my Ruby skills are much hotter than my JavaScript skills.

But, there is actually a 3rd option that comes to mind. What if we created a Rack middleware that sits in front of Rubytus, and translates the FineUploader protocol into TUS? That way we get to leverage Rubytus, as I think it already solves a lot of problems which are protocol-agnostic, without having to change anything from FineUploader's side.

@openscript From these three options I'm now most inclined to start with the 3rd, as that seems like it would be the least amount of work, and we get to reuse existing generic components which is awesome. This way our "fineuploader server" would automatically benefit from any improvements that land on Rubytus. What do you think?

[janko]

Actually, I'm not so sure that the 3rd option would require less work than others, because it requires understanding FineUploader protocol, TUS protocol, and Rubytus itself. But I think it would be the more useful than implementing a standalone FineUploader server.

It's also worth noting that Rubytus doesn't seem to fully support TUS 1.0 yet (picocandy/rubytus#2), so it might not automatically work with tus-js-client at the moment.

There is also another option, I think, and that is to use an existing FineUploader server written in another language. fineuploader-go-server looks really good and would probably be the best bet, because from what I understood you don't need to have Go installed on the server to run a Go program. It might sound strange to use an app written in another language, but from the client side you don't actually care in what language an app is written in, you just care that it accepts requests and returns responses. And you shouldn't need to modify anything, just run it on some port like any other app.

To summarize, I think we have five options here:

1. Instead of FineUploader use tus-js-client with Rubytus
2. Add TUS support to FineUploader and use Rubytus (probably most useful in the long run)
3. Implement a standalone FineUploader-compliant Ruby server
4. Implement a Rack middleware for Rubytus which translates FineUploader requests into TUS and TUS responses into FineUploader (although I don't know if this is even possible)
5. Use an existing FineUploader-compliant server like fineuploader-go-server

@openscript Since I probably won't have time to help with 3rd or 4th option, it's up to you to choose whatever you think it's best for you.

[openscript]

I read through the whole issue again and tried to come to a conclusion, where to go from here. I've been working on several Rails projects since 2011 and handling uploaded files was always a pain. In my opinion there was never a rock solid and flexible solution for doing that. I've tried many combinations of Paperclip, Carrierwave, PluploadJS, jQuery File Upload or FineUploader. There was always something I wasn't happy about or something which needed some wizardry and tinkering. It would be great to have a strong, flexible, foolproof solution for uploading files to Ruby apps.

During today I made for all options a little diagram, so it's easier to get an overview. I wrote down the pros and cons as well:

Option 1 – FineUploader Ruby server

Pros:

• No changes in FineUploader
• In my opinion, this solution takes the least time

Cons:

• FineUploader doesn't get TUS support
• The RubyFineUploader endpoint has only a single purpose.
• There still won't be a strong file uploading framework with TUS support
• There are quite many components, which have to work together

Option 2 – Adding TUS to FineUploader

Pros:

• FineUploader gets TUS support.
• There will be a strong file uploading framework with TUS support.
• Not so many components. No new component. Building on existing projects and components.

Cons:

• Integrating TUS into FineUploader can be time consuming.

Option 3 – FineUploader-to-TUS middleware

Pros:

• No changes in FineUploader

Cons:

• FineUploader doesn't get TUS support
• The FUToTUS endpoint has only a single purpose.
• There still won't be a strong file uploading framework with TUS support
• There are quite many components, which have to work together.
• A lot of knowledge needed for the FUToTUS component: Ruby, FineUploader protocol, TUS, ..

Option 4 – TUS JS library

Pros:

• Almost nothing to implement (maybe extending RubyTUS a bit, but that needs to be done for the other options too) to get resumable uploads working. Just linking existing components together.

Cons:

• FineUploader doesn't get TUS support
• There still won't be a strong file uploading framework with TUS support
• A lot of features missing, which are already implemented (AND tested) in FineUploader like: Client side image resize, Folder drop, ..

Option 5 – FineUploader Go server

Pros:

• No changes in FineUploader
• In my opinion, this solution doesn't consume a lot of time too..

Cons:

• FineUploader doesn't get TUS support
• Make the whole thing more complex by bringing another technology into the game
• There still won't be a strong file uploading framework with TUS support
• There are quite many components, which have to work together

[openscript]

I want to go for option 2. I've already forked FineUploader and started integrating TUS. I've set up and got the TUS server written in go runningm so Ihave something to run FineUploader with TUS against. I think with some support from @rnicholus this is getting on track soon!

It would be great to have a sample project with Shrine and RubyTUS. I haven't been digging to deep into RubyTUS, but I've seen too, that the last commit is a year ago and TUS 1.0 is not complete. Probably we need to get RubyTUS to TUS 1.0, but I think that's also not so hard. @janko-m do you think you have the time to set that example project up? :-)

What do you think of all that? Do you think it's possible to get the option 2 implemented until the 21. August?

[rnicholus]

@openscript You've spent quite a bit of time outlining all of the options, very impressive. Since the change here will be to Fine Uploader (option 2), can you outline your proposed changes to Fine Uploader in FineUploader/fine-uploader#1620? Perhaps I can offer some advice or locate a potential issue before you get too deep into implementing your solution.

The chunking logic in Fine Uploader is quite complex, mostly due to the concurrent chunking feature I added a couple years ago. This feature allows bandwidth to be maximized when uploading a single file. Essentially, it breaks the file up into multiple chunks and sends as many of the chunks as possible (while respecting the maxConnections option) at once. This was quite difficult to implement, and I realized why other libraries punted on this feature in the past, but the speed gains are notable.

[janko]

@openscript Wow, so impressed with these diagrams and the pros & cons sections. I edited your comment to just note what each option relates to (I hope you don't mind). It's great that you're going for adding TUS support to FineUploader. @rnicholus It's awesome that you've implemented concurrent chunking feature into FineUploader, hopefully that will work the same way with TUS.

I will gladly set up an example project that integrates Shrine and Rubytus. I will first use tus-js-client and check whether Rubytus works with it, and try to make necessary PRs if it doesn't. And later once you manage to get TUS into FineUploader, we'll switch tus-js-client with FineUploader.

I would also like to get an opinion from @MarkMurphy, as I was reading his discussion around resumable uploads on Paperclip, and he seemed very knowledgeable about the topic. Mark, if you're not too busy, how did you end up implementing resumable uploads? Did you manage to integrate Rubytus with your Ruby application? If yes, what JS library did you use with it?

[MarkMurphy]

@janko-m Thanks, and yes I did help write some of the TUS protocol. I built a custom solution for a Ruby on Rails api. Some of it was inspired by code from Rubytus. I set it up so that it would easily couple with Paperclip. The api is currently only consumed by an iOS application so I can't speak for any client side libraries.

I don't mind sharing some of my api code.

Here's the gist:

https://gist.github.com/MarkMurphy/76dae9307cb67d56951e13a63df99b19

[janko]

@MarkMurphy Thank you very much for sharing this!

[janko]

@openscript So, I managed to get Rubytus working with tus-js-client, and now it works locally both creating and resuming (although there might be an edge case that I've missed). I made the following changes:

Rubytus changes

commit c8c83736994124db3c337e22b4195591c15de64a
Author: Janko Marohnić <[email protected]>
Date:   Sat Aug 13 02:38:06 2016 +0800

    Get things working

diff --git a/.ruby-version b/.ruby-version
deleted file mode 100644
index 68b3a4c..0000000
--- a/.ruby-version
+++ /dev/null
@@ -1 +0,0 @@
-1.9.3-p551
diff --git a/lib/rubytus/constants.rb b/lib/rubytus/constants.rb
index d93af20..49f63cd 100644
--- a/lib/rubytus/constants.rb
+++ b/lib/rubytus/constants.rb
@@ -16,7 +16,7 @@ module Rubytus
     DEFAULT_MAX_SIZE       = 1073741824

     SUPPORTED_VERSIONS     = ['1.0.0']
-    SUPPORTED_EXTENSIONS   = ['file-creation']
+    SUPPORTED_EXTENSIONS   = ['creation']

     STATUS_OK                  = 200
     STATUS_CREATED             = 201
diff --git a/lib/rubytus/helpers.rb b/lib/rubytus/helpers.rb
index 21dc232..9b33ffa 100644
--- a/lib/rubytus/helpers.rb
+++ b/lib/rubytus/helpers.rb
@@ -1,6 +1,7 @@
 require 'rubytus/constants'
 require 'rubytus/common'
 require 'rubytus/error'
+require 'base64'

 module Rubytus
   module Helpers
@@ -12,7 +13,7 @@ module Rubytus

       env['api.headers'].merge!(handle_cors(request, headers))

-      if request.collection? || request.resource?
+      if (request.collection? || request.resource?) && !(request.options? || request.get?)
         validates_supported_version(headers["Tus-Resumable"])
       end

@@ -27,7 +28,7 @@ module Rubytus

     def validates_offset(req_offset, info_offset)
       if req_offset > info_offset
-        error!(STATUS_FORBIDDEN, "Offset: #{req_offset} exceeds current offset: #{info_offset}")
+        error!(STATUS_FORBIDDEN, "Upload-Offset: #{req_offset} exceeds current offset: #{info_offset}")
       end
     end

@@ -71,18 +72,26 @@ module Rubytus

     def parse_metadata(metadata)
       return if metadata.nil?
-      arr = metadata.split(' ')
+      pairs = metadata.split(",").map { |string| string.split(" ") }

-      if (arr.length % 2 == 1)
+      if pairs.any? { |pair| pair.size != 2 }
         error!(STATUS_BAD_REQUEST, "Metadata must be a key-value pair")
       end

-      Hash[*arr].inject({}) do |h, (k, v)|
+      Hash[pairs].inject({}) do |h, (k, v)|
         h[k] = Base64.decode64(v)
         h
       end
     end

+    def serialize_metadata(metadata)
+      encoded_list = metadata.map do |key, value|
+        "#{key} #{Base64.encode64(value)}"
+      end
+
+      encoded_list.join(",")
+    end
+
     def handle_cors(request, headers)
       origin = headers['Origin']

@@ -92,11 +101,11 @@ module Rubytus
       cors_headers['Access-Control-Allow-Origin'] = origin

       if request.options?
-        cors_headers['Access-Control-Allow-Methods']  = "POST, HEAD, PATCH, OPTIONS"
-        cors_headers['Access-Control-Allow-Headers']  = "Origin, X-Requested-With, Content-Type, Entity-Length, Offset, TUS-Resumable"
+        cors_headers['Access-Control-Allow-Methods']  = "POST, GET, HEAD, PATCH, DELETE, OPTIONS"
+        cors_headers['Access-Control-Allow-Headers']  = "Origin, X-Requested-With, Content-Type, Upload-Length, Upload-Offset, Tus-Resumable, Upload-Metadata"
         cors_headers['Access-Control-Max-Age']        = "86400"
       else
-        cors_headers['Access-Control-Expose-Headers'] = "Offset, Location, Entity-Length, TUS-Version, TUS-Resumable, TUS-Max-Size, TUS-Extension"
+        cors_headers['Access-Control-Expose-Headers'] = "Upload-Offset, Location, Upload-Length, Tus-Version, Tus-Resumable, Tus-Max-Size, Tus-Extension, Upload-Metadata"
       end

       cors_headers
@@ -117,7 +126,7 @@ module Rubytus
         env['api.uid']           = uid
         env['api.entity_length'] = request.entity_length
         env['api.resource_url']  = request.resource_url(uid)
-        env['api.metadata']      = parse_metadata(headers['Metadata'])
+        env['api.metadata']      = parse_metadata(headers['Upload-Metadata'])
       end
     end

@@ -134,6 +143,10 @@ module Rubytus
         env['api.action']  = :patch
         env['api.buffers'] = ''
         env['api.offset']  = request.offset
+
+        env['api.headers'].merge!(
+          'Upload-Offset' => (request.offset + request.content_length).to_s
+        )
       end

       if request.get?
diff --git a/lib/rubytus/info.rb b/lib/rubytus/info.rb
index 7e41e0d..ad3cbd7 100644
--- a/lib/rubytus/info.rb
+++ b/lib/rubytus/info.rb
@@ -3,25 +3,33 @@ require 'json'
 module Rubytus
   class Info < Hash
     def initialize(args = {})
-      self['Offset']       = args[:offset]        || 0
-      self['EntityLength'] = args[:entity_length] || 0
-      self['Meta']         = args[:meta]          || nil
+      self['Upload-Offset']   = args[:offset]        || 0
+      self['Upload-Length']   = args[:entity_length] || 0
+      self['Upload-Metadata'] = args[:meta]          || nil
     end

     def offset=(value)
-      self['Offset'] = value.to_i
+      self['Upload-Offset'] = value.to_i
     end

     def offset
-      self['Offset']
+      self['Upload-Offset']
     end

     def entity_length=(value)
-      self['EntityLength'] = value.to_i
+      self['Upload-Length'] = value.to_i
     end

     def entity_length
-      self['EntityLength']
+      self['Upload-Length']
+    end
+
+    def metadata=(value)
+      self['Upload-Metadata'] = value
+    end
+
+    def metadata
+      self['Upload-Metadata']
     end

     def remaining_length
diff --git a/lib/rubytus/middlewares/storage_barrier.rb b/lib/rubytus/middlewares/storage_barrier.rb
index 786d3c6..7c36537 100644
--- a/lib/rubytus/middlewares/storage_barrier.rb
+++ b/lib/rubytus/middlewares/storage_barrier.rb
@@ -1,9 +1,12 @@
+require 'rubytus/constants'
+require 'rubytus/helpers'
 require 'rubytus/error'

 module Rubytus
   module Middlewares
     class StorageBarrier
       include Rubytus::Constants
+      include Rubytus::Helpers
       include Goliath::Rack::AsyncMiddleware

       def post_process(env, status, headers, body)
@@ -16,7 +19,7 @@ module Rubytus
           when :create
             status = STATUS_CREATED
             headers['Location'] = env['api.resource_url']
-            storage.create_file(env['api.uid'], env['api.entity_length'])
+            storage.create_file(env['api.uid'], env['api.entity_length'], env['api.metadata'])

           when :head
             info = storage.read_info(env['api.uid'])
@@ -24,11 +27,28 @@ module Rubytus
             if info.nil?
               status = STATUS_NOT_FOUND
             else
-              headers['Offset'] = info.offset.to_s
+              headers['Upload-Offset']   = info.offset.to_s
+              headers['Upload-Length']   = info.entity_length.to_s
+              headers['Upload-Metadata'] = serialize_metadata(info.metadata) if info.metadata
             end

           when :get
-            body = storage.read_file(env['api.uid'])
+            path = storage.file_path(env['api.uid'])
+            info = storage.read_info(env['api.uid'])
+
+            if metadata = info.metadata
+              headers['Content-Disposition'] = "attachment; filename=\"#{metadata['filename']}\"" if metadata['filename']
+              headers['Content-Type'] = metadata['content_type'] if metadata['content_type']
+            end
+
+            file = ::Rack::File.new(File.dirname(path))
+            file.path = path
+
+            result = file.serving(env)
+
+            status  = result[0]
+            headers = result[1].merge(headers)
+            body    = result[2]
           end
         rescue PermissionError => e
           raise Goliath::Validation::Error.new(500, e.message)
diff --git a/lib/rubytus/request.rb b/lib/rubytus/request.rb
index 0b69abe..c8d9f1f 100644
--- a/lib/rubytus/request.rb
+++ b/lib/rubytus/request.rb
@@ -43,11 +43,11 @@ module Rubytus
     end

     def entity_length
-      fetch_positive_header('HTTP_ENTITY_LENGTH')
+      fetch_positive_header('HTTP_UPLOAD_LENGTH')
     end

     def offset
-      fetch_positive_header('HTTP_OFFSET')
+      fetch_positive_header('HTTP_UPLOAD_OFFSET')
     end

     def base_path
@@ -88,8 +88,8 @@ module Rubytus
         error!(STATUS_BAD_REQUEST, "#{header_orig} header must not be empty")
       end

-      if (header_name == 'HTTP_ENTITY_LENGTH') && (value < 0)
-        error!(STATUS_BAD_REQUEST, "Invalid Entity-Length: #{value}. It should non-negative integer or string 'streaming'")
+      if (header_name == 'HTTP_UPLOAD_LENGTH') && (value < 0)
+        error!(STATUS_BAD_REQUEST, "Invalid Upload-Length: #{value}. It should non-negative integer or string 'streaming'")
       end

       value
diff --git a/lib/rubytus/storage.rb b/lib/rubytus/storage.rb
index c00a3c1..a0fe2f5 100644
--- a/lib/rubytus/storage.rb
+++ b/lib/rubytus/storage.rb
@@ -48,8 +48,9 @@ module Rubytus
     def create_file(uid, entity_length, metadata = {})
       fpath = file_path(uid)
       ipath = info_path(uid)
-      info  = Rubytus::Info.new(metadata)
+      info  = Rubytus::Info.new
       info.entity_length = entity_length
+      info.metadata = metadata if metadata

       begin
         File.open(fpath, 'w') {}
@@ -94,7 +95,7 @@ module Rubytus

       begin
         data = File.open(ipath, 'r') { |f| f.read }
-        JSON.parse(data, :object_class => Rubytus::Info)
+        parse_info(data)
       rescue SystemCallError => e
         raise(PermissionError, e.message) if e.class.name.start_with?('Errno::')
       end
@@ -113,5 +114,13 @@ module Rubytus
         raise(PermissionError, e.message) if e.class.name.start_with?('Errno::')
       end
     end
+
+    private
+
+    def parse_info(data)
+      info = Rubytus::Info.new
+      info.replace(JSON.parse(data))
+      info
+    end
   end
 end
diff --git a/test/rubytus/test_command.rb b/test/rubytus/test_command.rb
index 368da77..48f668a 100644
--- a/test/rubytus/test_command.rb
+++ b/test/rubytus/test_command.rb
@@ -204,7 +204,7 @@ class TestRubytusCommand < MiniTest::Test
       :path => "/uploads/#{uid}",
       :body => 'abc',
       :head => protocol_header.merge({
-        'Offset'        => '0',
+        'Upload-Offset' => '0',
         'Entity-Length' => '3',
         'Content-Type'  => 'plain/text'
       })
@@ -230,7 +230,7 @@ class TestRubytusCommand < MiniTest::Test
       :path => "/uploads/#{ruid}",
       :body => 'abc',
       :head => protocol_header.merge({
-        'Offset'        => '0',
+        'Upload-Offset' => '0',
         'Entity-Length' => '3',
         'Content-Type'  => 'application/offset+octet-stream'
       })
@@ -256,7 +256,7 @@ class TestRubytusCommand < MiniTest::Test
       :path => "/uploads/#{ruid}",
       :body => 'abc',
       :head => protocol_header.merge({
-        'Offset'        => '3',
+        'Upload-Offset' => '3',
         'Entity-Length' => '3',
         'Content-Type'  => 'application/offset+octet-stream'
       })
@@ -281,8 +281,8 @@ class TestRubytusCommand < MiniTest::Test
       :path => "/uploads/#{ruid}",
       :body => 'abcdef',
       :head => protocol_header.merge({
-        'Offset'        => '0',
-        'Entity-Length' => '6',
+        'Upload-Offset' => '0',
+        'Upload-Length' => '6',
         'Content-Type'  => 'application/offset+octet-stream'
       })
     }
@@ -300,8 +300,8 @@ class TestRubytusCommand < MiniTest::Test
       :path => "/uploads/#{uid}",
       :body => 'abc',
       :head => protocol_header.merge({
-        'Offset'        => '0',
-        'Entity-Length' => '3',
+        'Upload-Offset' => '0',
+        'Upload-Length' => '3',
         'Content-Type'  => 'application/offset+octet-stream'
       })
     }
diff --git a/test/rubytus/test_storage.rb b/test/rubytus/test_storage.rb
index b36fbee..7fe0584 100644
--- a/test/rubytus/test_storage.rb
+++ b/test/rubytus/test_storage.rb
@@ -48,12 +48,12 @@ class TestStorage < MiniTest::Test

   def test_read_info
     File.open(@storage.info_path(@uid), 'w') do |f|
-      f.write('{"Offset":100,"EntityLength":500,"Meta":null}')
+      f.write('{"Upload-Offset":100,"Upload-Length":500,"Meta":null}')
     end

     info = @storage.read_info(@uid)
     assert_kind_of Hash, info
-    assert_equal 100, info['Offset']
+    assert_equal 100, info['Upload-Offset']
   end

   def test_read_info_failure
@@ -63,19 +63,19 @@ class TestStorage < MiniTest::Test

   def test_update_info
     File.open(@storage.info_path(@uid), 'w') do |f|
-      f.write('{"Offset":100,"EntityLength":500,"Meta":null}')
+      f.write('{"Upload-Offset":100,"Upload-Length":500,"Meta":null}')
     end

     @storage.update_info(@uid, 250)

     info = @storage.read_info(@uid)
     assert_kind_of Hash, info
-    assert_equal 250, info['Offset']
+    assert_equal 250, info['Upload-Offset']
   end

   def test_update_info_failure
     storage = Rubytus::Storage.new(read_only_options)
-    stub(storage).read_info(@uid) { Rubytus::Info.new('Offset' => 100) }
+    stub(storage).read_info(@uid) { Rubytus::Info.new('Upload-Offset' => 100) }
     assert_raises(Rubytus::PermissionError) { storage.update_info(@uid, 250) }
   end

However, when attempting to make a pull request with organized commits, I realized that there are still some things in the specification that are not part of Rubytus, but I had troubles implementing it because Goliath is a complex web framework 😃

In Goliath you can return early by raising a validation error, but there currently isn't a way to set headers when raising an error. I made a PR to Goliath, which should get merged soon, but I'm not sure when a new version of Goliath will be released (the last version was released in June 2014), and we would need a version to put it in rubytus.gemspec.

However, I feel like using Goliath is really an overkill here, the asynchronicity makes it much more difficult to work with than e.g. Roda. And I feel that EventMachine only shines when you do slow IO, and here we're just writing to filesystem. Even after that, Rubytus is still missing some TUS extensions (notably concatenation, which I find really important because it enables parallel chunked uploads).

So, now that read the whole TUS 1.0 thoroughly, I decided that it makes much more sense to make my own implementation in Roda. That would make it much easier to maintain, since not many people are familiar with Goliath. And also current Rubytus code is pretty difficult to follow, since it's scattered across middlewares and subclasses. I feel like it would it would take similar amount of time to make Rubytus work fully by the specification than it would take to make a Roda-based TUS 1.0 implementation. And it would bring me a lot of joy to do this, because I absolutely love Roda. 😃

[erikdahlstrand]

That is truly awesome @janko-m!