projectbismark / bismark-data-server

This repository does not currently contain any code of its own -- it is a container for project-wide issues and wiki pages to be stored. Code is stored in many of the bismark-* repositories.

Wiki: https://github.com/projectbismark/projectbismark/wiki

Project-wide issues: https://github.com/projectbismark/projectbismark/issues

Overall issue dashboard: https://github.com/organizations/projectbismark/dashboard/issues

Project Page and Dataset: http://projectbismark.net.s3.us-east-2.amazonaws.com/index.html

bismark-data-server's Issues

Nothing is logged

Uploads and failures aren't logged anywhere. We're using django, so this could easily go into a database.

Switch from apache to nginx

nginx should scale much better. Setting up will require exploring various ways to get nginx talking with Django.

Keep-alive doesn't work

Not sure where it's a client or server problem. First step to diagnose is run our Apache instance in HTTP mode and watch the traffic with Wireshark.

Implement rate limiting

We currently limit the size of each uploaded file, but don't limit the rate at which they're uploaded, so someone could easily DoS the server by uploading many small files.

We should implement a token bucket rate limiter with the following parameters:

• Token size: 1 byte
• Bucket size: router_uploads_buffer_size + max_upload_file_size*retry_interval/expected_upload_interval
• Refill rate: max_upload_file_size / expected_upload_interval