Streaming get / post about tantalum

[paulirotta]

It is a worthy expansion.

When we do that, the complexity to watch for obscure hang states and performance suck is between the Worker thread, the Worker queue, and the source of the stream. On some devices we only use 2 Worker threads so locking a Worker for an indeterminate amount of time might be expensive.

Several approaches come to mind:

1. reverse logic a bit and the stream source drives the process. Some more thought needed here
2. create a dedicated thread for the duration of the streaming operation
3. instead of stream, break the stream of data into a series of fixed-size byte[] buffer chunks, similar to iphone media framework and others. Here the risk is heap memory growth from queing, but proper dance between threads can keep that down.

[vivainio]

We should specify "writeReady" so that it should not block beyond what it takes to write to the OutputStream (i.e. to the http connection). That way, there is no more blocking than there is with the variant that takes byte[].

[vivainio]

Assigning to @jnynas as he has a working implementation is his own app already

[paulirotta]

I've shifted the default number of worker threads from 4 to 2 in large part because flash writes seem to suck CPU cycles on some phones. For algorithm reasons we can't really run with less that 2 workers. A nice implementation thus might need to add a Thread per stream and Task.fork(Task.SERIAL_PRIORITY) to drop ready chunks of data into flash memory.

[vivainio]

On not blocking when reading; yeah, correct user code will have to read all of the stream and enqueue the data for background writing. The advantage of stream would mainly be to only store the data that needs to be stored; e.g. in case of base64 encoded content, we would stream in the data as one decoded memory block that then gets queued for writing

... or indeed manually fork a new Task to handle that stream, for each request

[paulirotta]

Streaming large objects to flash memory file system could be quite useful, for example storing songs and videos on the memory card.

Let's keep this in the backlog

[vivainio]

Reopened, javadocs need to be added before closing the issue.

[paulirotta]

Did some JavaDoc and minor thread hardening.

A unit test or example to validate is still needed