tus / tusd

Reference server implementation in Go of tus: the open protocol for resumable file uploads

tusd

tus is a protocol based on HTTP for resumable file uploads. Resumable means that an upload can be interrupted at any moment and can be resumed without re-uploading the previous data again. An interruption may happen willingly, if the user wants to pause, or by accident in case of a network issue or server outage.

tusd is the official reference implementation of the tus resumable upload protocol. The protocol specifies a flexible method to upload files to remote servers using HTTP. The special feature is the ability to pause and resume uploads at any moment allowing to continue seamlessly after e.g. network interruptions.

It is capable of accepting uploads with arbitrary sizes and storing them locally on disk, on Google Cloud Storage or on AWS S3 (or any other S3-compatible storage system). Due to its modularization and extensibility, support for nearly any other cloud provider could easily be added to tusd.

Protocol version: 1.0.0

This branch contains tusd v2. If you are looking for the previous major release, after which breaking changes have been introduced, please look at the 1.13.0 tag.

Documentation

The entire documentation, including guides on installing, using, and configuring tusd can be found on the website: tus.github.io/tusd.

Build status

License

This project is licensed under the MIT license, see LICENSE.txt.

tusd's Issues

Failed because: Error: tus: failed to upload chunk at offset NUMBER

I´m testing tus-js-client with tusd.
The tusd is running in Ubuntu Desktop 16.04 in Virtualbox and tus-js-cliente demo in Windows host with Firefox. I put the tusd in a AWS EC2 server and the same error occour.
The tusd is configured with parameter -s3-bucket,
All works fine, but an error always happen in upload process in big files (~ 300Mb). When click in "Ok" the process continues but after some minutes the error occurs again.

"Failed because: Error: tus: failed to upload chunk at offset 62914560
Do you want to retry?"

After many click´s in "Ok" button, the upload is completed and file is correct in AWS S3

What might is happening? How might I discover wher is the problem?

Transfer over websockets

at @Wayla, we're heavily using websockets for file uploads & downloads. I haven't been through all of the @tus docs yet, but has there been any discussion of using websockets for transport?

Upload-Offset should not be returned for partial uploads

When using the concatenation extension, partial uploads should not have an offset, which also implies that they should be PATCHed from offset 0 regardless of how many bytes have been received so far.:

https://github.com/tus/tus-resumable-upload-protocol/blob/master/protocol.md#concatenation

Review tus.io protocol section

Streamable file downloads

Right now GET doesn't work until the full file is received. It'd be awesome if it would stream the amount of data that has been received so far, so that data can be streamed to the downloader as it's being uploaded!

Deleting .info files and moving uploaded file to another directory after upload complete

After successful upload, I need to move the uploaded file(.bin) to another directory and delete the .info file which is not need after that. Is there any call back function I can use to perform this operation.
And please let me know whether the following is feasible:
I have 'n' number of clients uploading files; I want to store the files in separate directory with respect to the client uploading it. This may be a feature of file server; but I foresee tus as a upload file server rather than just a uploading protocol.

Bad code at filestore.go

In the lines above the path is concatenated with /.

https://github.com/tus/tusd/blob/master/filestore/filestore.go#L187
https://github.com/tus/tusd/blob/master/filestore/filestore.go#L192

I think this will fail on Windows. And if I set the Path with slash it will add two slash. Most Linux don't break because of this, but the correct way is to use path.Join function to concat paths.

If there's no specific reason for this I can send a PR to fix it.

Access-Control-Allow-Origin

i am a newbie to tus (and github), hope to get a right place to ask.

i am trying to start a tusd on an aws ec2 instance and client on my local computer.i have tried to use the binary and everything all right.

However, i would like to config the datastore path, so i used the sample code here and the below error occur,

Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.

checked i have to allow the access of the ec2 from other origin, but i am not sure how to achieve. thanks for any reply.

Changing upload direcory (not sure is an issue)...

Hi!
I have just succeded in tusd (go application) installation on an ubuntu machine. My idea is to to test "resumable file upload funcionality" with an Android client (using your sample Android application for now).

I have succesfully uploaded some files to my server througth Android application (pausing and resuming it).
It works like a charme! :)

Then I have stopped the tusd application and I have changed the upload directory from "data" to "uploadDirectory" in main.go.
I have started the server again and I have succesfully uploaded some OTHER files.

But when I try to upload one file that I have already uploaded in the previous tusd session (with upload dir="data") I have receiveed an "unexpected status code (404) while resuming upload".

I'm completly newby on go so forgive me in advance if I have miss some important details (and for my bad English).

Where is my mistake?

Is this the rigth place to post this questions?

Thanks in advance.

Release tags not complied with SemVer spec

I was using the newest gb tool to fetch tusd, but it failed. Tusd is declared in the depfile. It turns out that tusd repo tags is not complied with the SemVer spec, so the gb tool failed. Can you guys tag the
tusd with a SemVer version?

Issue in using tusd with HTTPs

Hello,

I would like to use tusd on Https protocol. I didn't success to used it. Is there possible to use this protocol. If it's possible, how do it this ?

Thank you for helping.

refactor / clean up

The current code could need a little more structure / types, which would also help with testing.

Switch the usage of `UseIn`

Hi,

I find strange that store.UseIn(composer) only acts on the composer (and not on the store).

It might be possible to create a UseAllExtensions method on the composer using some tests to know what the store implements (see https://stackoverflow.com/a/20315155/3207406)

For instance:

func (composer *tusd.StoreComposer) UseAllExtensions(store interface{}) {
  var ok bool
  if _, ok = store.(DataStore);ok {
    composer.UseCore(store)
  }
  if _, ok = store.(LockerDataStore);ok {
    composer.UseLocker(store)
  }
  // ...
}

Some questions about <some.long.alfanumeric.string>.info files content...

During my test, I have earned how to add key-value pairs into MetaData field contained into .info files.
But I'm not sure to have well understood the meaning of the field "IsFinal": I have read the comments into filestore.go, but I have seen these field always set to false, even if the upload is correctly finished. On the other hand the field "IsPartial" is always false even if I stop (restart and stop again) the upload before it is completed.
What I'm misunderstanding?
Who is responsible (which "go" class) to write the content of the .info files?

Again, thanks a lot in advance.

Resume capability for HTTP GET

It seams that tusd does not support "Range" headers for resuming interrupted downloads, or using a download manager to faster download the files.

It would be very nice to have this feature ;-)

start html5 client

a separate repository might be good for this:

• 1 repo per client library
• Has it's own README.md we can use for docs for now
• Can use a different license (something we also need to figure out, we could go AGPL on the client libs as well and sell cheap commercial licenses)

Provide upstart skeleton?

At Transloadit, we're currently generating the following Upstart file to launch tusd:

$ cat /etc/init/tusd.conf
stop on runlevel [016]

respawn
respawn limit 10 5

limit nofile 32768 32768

pre-stop exec status "tusd" | grep -q "stop/waiting" && initctl emit --no-wait stopped JOB="tusd" || true

script
  set -e
  rm -f "/srv/shared/tusd-log-fifo"
  mkfifo "/srv/shared/tusd-log-fifo"
  ( logger --tag "tusd" <"/srv/shared/tusd-log-fifo" & )
  exec >"/srv/shared/tusd-log-fifo"
  rm -f "/srv/shared/tusd-log-fifo"
  exec bash -c "cd /srv/current \
    && source env.sh \
    && exec sudo -HEu transloadit-api2 PATH=\${PATH} LD_LIBRARY_PATH=\${LD_LIBRARY_PATH} \
      /srv/current/stack/bin/tusd \
        -host 127.0.0.1 \
        -port 1080 \
        -base-path /resumable/ \
        -hooks-dir /srv/current/api2/bin/tusd-hooks \
        -max-size 42949672960 \
        -s3-bucket tmp.transloadit.com \
      2>&1"
end script

Then our rsyslog is configured like so:

$ cat /etc/rsyslog.d/49-tusd.conf
if $programname == 'tusd' then /srv/shared/log/tusd.log
if $programname == 'tusd' then @logs.papertrailapp.com:34719
if $programname == 'tusd' then stop # no other actions to avoid duplicates

This handles things like logging (both to local file via syslog as well as centralized to papertrail), running tusd as a non-privileged user, injecting environments with a privileged user, and respawns.

Now this config may not work for everyone. I would also not be surprised if people in the tus community know better ways. And so the thought arises, maybe it's useful to offer a few community curated startup templates, initially for both Upstart and systemd (but that could later be expanded to support other platforms), so that the binary is easier deployed in production environments, and users run tus in a safer, and pretty much all in an identical way.

This could be provided by the tus community either in the form of:

• A developers's guide with some snippets in the best practices, ready for copy/paste.
• In a ./templates/ directory or similar
• In code, so that one could type tusd os_startup or similar, and it would ask some questions, and output the appropriate startup file, that the user could | sudo tee /etc/to/the/right/place (avoiding tusd writing itself and having to ever run it as root)

Thoughts?

Official s3 storage plugin

Would be awesome to get an officially maintained s3 storage plug using tusd.DataStore.

config: expose more options

For now we should probably make the most basic things configurable as environment variables:

• http port - done
• tus_data dir
• tus_data dir max size

implement out file storage

We need a way to locally store files, which may be coming in as partial chunks. These chunks may overlap, which must cause no problems if the overlapping parts hold the same data.

My idea: For the first request create a file id, then create tus_data/<fileId> and tus_data/<fileId>.log. The first file has the full size of the expected is empty, and gets filled as data comes in. The second file is an append-only log that lists all chunks of the file that have been received. The format for this could be new-line separated JSON.

Reverse proxy support

Now UnroutedHandler.absFileURL function uses r.Host and scheme of server where the tusd handler is embedded. It would be great if the handler automatically detects proxied request using Forwarded/X-Forwarded-Host headers. I have to manually set r.Host before calling tusd handler to workaround the cross-origin error.

Standalone server

Hello,

As the README says:
"In the future tusd may be extended with additional functionality to make it suitable as a standalone production upload server, but for now this is not a priority."

This issue deals with the standalone version of tusd.

tus client for Go

Hi @Acconut, as mentioned I made a version of tus client for go lang.

It was tested with 1.6, 1.7 and 1.8 (tip).

The code can be found in github.com/eventials/go-tus.

Feel free to check out, comment, or even send a PR. The lib was based a little in the JS version, but I want to add more features soon.

gc: delete uploads as needed

Otherwise we may fill up the disk on our demo server too quickly.

Basic plan: Trigger GC() after each upload, if total file size is too big, delete old files until enough space is available.

store content-type

Need to figure out where to put it. Potentially as the first entry into the log file?

Breaking internet connection does not stop uploading

Hello,

Few days ago I started port tus java client to C#. When I finished I tested it on mobile devices. I have tested 2 scenarios:

• Closing stream connection by killing application (S1)
• Closing internet connection by turning off WiFi (S2)

In S1 Golang server close connection correctly, copy already send data to the file and set correct offset for future requests.
In S2 Golang server is waiting for resuming connection (it looks like). As far as I know it has default infinity timeout and is waiting for resuming connection too long. In that case If I want to re-upload the file I must start from the scratch (HEAD requests gives me 0 offset).

I don't know Golang so I'm asking you: Is there any way you can handle this scenario?

tusd uploaded files permissions executable

Great project, we are just starting to use it. Hope uploading implementations are a thing of the past now 👍

The uploaded files are created with the executable bit true. For everyone! -rwxr-xr-x
I would prefer them not executable.

For me this could be one part of an attack strategy with some trojan. It's clear that there has to be a second exploit at least to complete the big picture. But early intervention is king here.

I don't know exactly if this could be changed by the os configuration or is caused by it (Ubuntu 16.04, pretty default install). Other uploads and tmp outside of tusd are not executable.

But maybe this should be cared about by tusd in every case to prevent people running into it.

A welcome message would be nice

When you GET on the root or the basePath, it would be nice to display:

Welcome to tusd version x.y.z. For documentation and bug reports, please check https://github.com/tus/tusd

Instead of

Just more friendly and to let admins know they did something right, vs wrong.

Rename basePath to urlPrefix

urlPrefix is more specific / less generic imho

add integration tests

This depends on #9, once we have things refactored, we could easily spin up a test server for integration tests.

Error: Cannot find module 'google-cloud'

I can't run the server, nor demo files. I keep getting Error: Cannot find module 'google-cloud' from node.
Is there any workaround? Am I doing something wrong?

How to enable checksum?

Hi,

Looking into the code and the go docs I couldn't figure out how to enable checksum extension.

The only place that mention this extension is the specification.

Is this supported by tusd? How can I enable ou start implementing this?

add logging

this way we can tell what files were uploaded, how fast, etc.

Information about high data transfers

This is about programs --some of them with the protocol explained-- that got great speed to upload files:

http://www.tcnj.edu/~bush/uftp.html
http://udt.sourceforge.net/
http://tsunami-udp.sourceforge.net/

And this information is to tune TCP to get high performance data transfers:

http://www.psc.edu/index.php/networking/641-tcp-tune

I hope it can be useful at some point.

Anyway to change the owner of uploaded files?

I know this may be a bit off-topic, but i have researched a lot and no clue.
The case is that I am working on a simple web application that allows people to upload files and edit them.
However I found that the uploaded files are owned as root and with permission 644, which cannot be edited by apache.
I have no idea how to allow apache edit the file that is not owned by it. Is it possible to change the owner or permission once tusd received the file?
many thanks!

upload broken

my local upload demo is broken (upload gets stuck / is not resumable), need to figure it out / fix it.

GridFS support

Hello,
I'm currently working on MongoDB and I need to upload some large files into DB. Is there a plan to support GridFS storage?

Where is the uploaded file?

Another issue (closed) says, regarding uploaded files being renamed to *.bin and *.info: "That's a good use case but a very specialized one. In general, I think, people will need more information than just the original file name and therefore have to read the additional data anyway. So I am sorry, but your suggestion won't make it into tusd."

I believe this is a major shortcoming for a "file uploader" - the uploaded file is essentially not there at all; this is a huge use case, not specialized in the least.

Is there a procedure for transferring "x.jpg" for example, to the server as "x.jpg"? Or, perhaps, a config hook to rename the file upon upload completion?

Remove locking from handler implementation

Hi!
I'm experimenting with the tusd implementation as the basis for a production-grade TUS server. One of the things I've noticed in our tests is that at times the tusd.Handler can get files to stay in locked state, with no way of being unlocked. We don't think we need this locking behavior in the handler, and can delegate this to the file store (we're using a DB as our back-end, which handles concurrent read/writes).

I have a (forthcoming) pull request which refactors the locking behavior to the filestore.Filestore type.

Best,
Adam

> 4.0 GB of file

Hi all,

I'm trying to upload a file that have size of 4.6 GB via tus. But after file is about 4.0 GB the PATCH method return an error message

read tcp 127.0.0.1:1080->127.0.0.1:61596: i/o timeout

Any idea?

Wrong code returned for HEAD calls in unrouted handler

The protocol shows HEAD calls as returning 200 codes, which is also how the TUSKit library is implemented and what the HTTP spec defines.

https://github.com/tus/tus-resumable-upload-protocol/blob/master/protocol.md#example
https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

Issues in configuring Tusd Server

Hi,

I want to use resumable upload functionality with PHP server to make it work over server, there are few things need like below -

1. tusd server - Uploaded files will be saved over tusd server(it will be endpoint of the tusd client, if i'm not wrong.)
2. tusd client - From where user can upload files, abort the upload, resume the uploads.

I have configured tusd client but when installing/configuring tusd server in my local machine, it is not working as it is giving error when I'm trying to run main.go using below command -
go run cmd/tusd/main.go

Please try to help me out to solve the issue.

Thanks!

Make /metrics endpoint configurable

Currently, /files is configurable.

If you where to place tusd behind a proxy, and route all URLs prefixed /resumable/ to your tusd server (and you're not stripping this prefix, because tusd needs to know and return the full path to its files), you might end with:

tusd \
  -base-path /resumable/files/

However, since the metrics path is hardcoded under /metrics this will now have become inaccessible, unless you are hacking around this with conditional stripping of prefixes.

Long story short we could solve this by making the metrics endpoint path configurable. And in our case, mount it to /resumable/metrics and then redirect all /resumable/* traffic to tusd without stripping or rewriting anything conditionally:

tusd \
  -base-path /resumable/files/ \
  -metrics-path /resumable/metrics

Or perhaps, if we're going down this path anyway, maybe we can make it apparent we are not talking about local paths, and also be more specific than base:

tusd \
  -files-endpoint /resumable/files/ \
  -metrics-endpoint /resumable/metrics

Review tus.io faq section

Default maximum file size?

Hi All,

I'm sorry if this is a noob question, but I can't find the default maximum file upload size for Tusd. I am not versed in Go so am unable to understand the flow of the binary.

Can someone help? Thanks!

implement remaining http routes

We need all of them : )

About the file name

Hi all,

I tried to run the official binary distribution of Go language edition of tud with official Android sample code (client). I found a file name problem (or may be an enhancement / feature request) of current sample codes of server side and client side.

The uploaded file's file name on the server is convert to a random hashed file name (e.g. 5dcecfd73c6d77bf7dfcf4fbdd75275e.bin with 5dcecfd73c6d77bf7dfcf4fbdd75275e.info )
It's a good design to avoid the duplicate of file name of uploaded files.
But the server side do not know the original file name of uploaded file either in .info file. (.info file do not have the information of original file name)
The client side is also do not know the new file name of uploaded file.
So it is difficult to make the relationship of uploaded file name with original file name for further application (e.g. client side can call the server API to tell the server the uploaded file name for server to fetch the file for further processing after client side uploaded the file)
The file extensions (e.g. .png, .zip) are also changed to .bin. The server side is also difficult to know the original file extension name to do further use.

In conclusion, my suggestions are as below:

1. Server side (tud):
   a. change the file name to the format such like the "original_file_name_random_hash.original_file_extension" (e.g. MyPicture_e4fe1bebdfcdc5efd402de1d2bfa6ac4.png)
   b. record the original file name information and new file name information in the .info file for further processing
   c. return the new file name to the client side via the protocol or some mechanism
2. Client side (sample codes)
   a. receive and show the "new file name" information of uploaded file

Return a custom HTTP error from the datastore

How can I return a custom http error code from a custom datastore ?

For instance, if the file extension is not allowed, I would like to send back a http/403 with a reason file extension not allowed.
From the source, if my datastore returns an errors.New("403: File extension not allowed") it will be returned as a 500 error.

Could the unrouted_handler provide an error structure to customize the status code ?
Like return tusd.NewHttpError(403, "File extension not allowed") (if the error implements the HttpError interface, the status code is custom, otherwise it is set to 500).

The unrouted handler could also benefit from this (to remove the ErrStatusCodes map):

ErrUnsupportedVersion  = NewHttpError(http.StatusPreconditionFailed, "unsupported version")
ErrMaxSizeExceeded     = NewHttpError(http.StatusRequestEntityTooLarge, "maximum size exceeded")
...

S3 AWS package incompatibility

Hey guys,

So I am trying to use TUS with S3 in my own project, since there were no examples, I looked into comments/code and tried came up with this piece of code:

        cfg := &aws.Config{
            Credentials: credentials.NewStaticCredentials(
                viper.GetString("s3_access_key_id"),
                viper.GetString("s3_secret_access_key"),
                ""),
        }
        s3Client := s3.New(session.New(), cfg)
        store := s3store.New(viper.GetString("s3_bucket"), s3Client)
        store.UseIn(composer) 

When compiling this piece of code I get compile errors

cannot use s3Client (type *"github.com/aws/aws-sdk-go/service/s3".S3) as type s3iface.S3API in argument to s3store.New:
    *"github.com/aws/aws-sdk-go/service/s3".S3 does not implement s3iface.S3API (wrong type for AbortMultipartUpload method)
        have AbortMultipartUpload(*"github.com/aws/aws-sdk-go/service/s3".AbortMultipartUploadInput) (*"github.com/aws/aws-sdk-go/service/s3".AbortMultipartUploadOutput, error)
        want AbortMultipartUpload(*"github.com/tus/tusd/vendor/github.com/aws/aws-sdk-go/service/s3".AbortMultipartUploadInput) (*"github.com/tus/tusd/vendor/github.com/aws/aws-sdk-go/service/s3".AbortMultipartUploadOutput, error)

What am I doing wrong here? Is it a bug? How can I fix this?

Better error message when Content-Length does not match Content-Range

Right now it gives a 500 error with the explanation EOF which is confusing