Protocol: File transfer outside of JSON-RPC about core

[44203]

@robertsdotpm

[44203]

https://github.com/mscdex/base91.js

[robertsdotpm]

It's still no where near as efficient as pure binary data but I'd consider it a good compromise for short - medium term.

My concern with this current design is that most servers aren't exactly optimised to accept huge amounts of data in the POST request - the file field is normally used and then the data can be easily buffered to /tmp. But with something like POST, its potentially all just in memory and that could add up to bad concurrency / scaling if the shards are like 128 mbs each (or if you're doing multiple transfers at once.) So we probably still do need to use something different eventually.

[44203]

Yes, this doesn't work well for large chunks. Let's think about other options. Maybe a separate data channel is needed.

Or maybe websockets works for this since we are already over HTTP.

[44203]

The CONSIGN RPC could just exchange a token for upgrading the HTTP connection to WS and pump the raw binary data over.

[44203]

http://binaryjs.com/