projectbismark / bismark-data-transmit

data transmit using cURL

This software monitors a set of directories (the "upload directories") for new files and uploads them to a server using a persistent HTTP connection.

How to use:

1. Build a copy of bismark-data-transmit. See bismark-packages for the companion OpenWRT package Makefile.
2. Install bismark-data-transmit on your BISmark router.
3. Create the UPLOAD_ROOT directory. By default, this is /tmp/bismark-uploads but you can change it using the package Makefile.
4. Create subdirectories of UPLOAD_ROOT. bismark-data-transmit will monitor these for new files to upload. Each subdirectory should represent a certain type of data to be processed on the server side. For example, I created /tmp/bismark-uploads/passive for bismark-passive log files and /tmp/bismark-uploads/passive-frequent for bismark-passive's frequent uCap-specific updates. Data from these directories will be processed by different entities on the server side. Alternatively, you can place directory names in /etc/config/bismark-data-transmit and they will be automatically created before starting.
5. Restart bismark-data-transmit for it to rescan UPLOAD_ROOT and begin accepting uploads from the new directories.
6. To upload files, move files into the subdirectories of UPLOAD_ROOT. Do not create new files directly inside subdirectories of UPLOAD_ROOT. They will not get uploaded in a timely fashion. Instead, create files somewhere else and mv them into the desired subdirectory.
7. Collect your files on the server side. Files are not guaranteed to arrive in any particular order, since files that fail to upload are retried after a delay (3 minutes by default).
8. There is a limited buffer for storing uploads. If more than 5 MB of pending uploads accumulate, bismark-data-transmit will start deleting the oldest uploads. It counts the number of files it's deleted and writes the counters to /tmp/bismark-data-transmit-failures.log.

Point 6 deserves repetition: Do not create new files directly inside /tmp/bismark-uploads. Instead, create the files elsewhere and mv them into /tmp/bismark-uploads/.

bismark-data-transmit's Issues

Uploads URL isn't configurable

It should be configurable using a command line parameter.

Retries and garbage collection are untested

We're using two threads and two connections

Stop using pthreads and (thereby) multiple cURL handles. This should dramatically cut memory usage.

Failed uploads are retried forever

Failed uploads are retried every 30 minutes until they succeed. This could take an infinite amount of time.

Log when files get dropped

bismark-data-transmit-collect doesn't log when it removes old files.

Explore and implement rate limiting

We should figure out if we need to limit the bitrate of the uploader, and implement if so.

Rate limiting would be useful after long periods of uplink unavailability, when lots of files need to be uploaded.

Only one connection open at a time

If a server fails we should use a failover server

tmpfs could fill up

Failed uploads should be deleted when tmpfs is almost full, not when they get old.

Nothing is logged

Should add fixed-size rotating log.

Application support for dropped file detection

We should log the number of dropped files in each directory in a machine-readable format. This will let processes like bismark-passive detect when data has been lost and act accordingly.

Keep-alive doesn't work

Not sure whether it's a client or server problem. First step to diagnose is to run our Apache instance in HTTP mode and watch traffic on Wireshark.