Chunk encoding instead of content-length? about watsonwebserver

[xmike402]

Wow, you are one cool dude, thanks for taking this on! As I said in my original post right now I am trying to recreate that node server for streaming live mpeg1. The input is from ffmpeg and the output is http using the upgrade call. I actually got it working right now where I buffer about 2 seconds to disk and then read that buffer to each viewer, but it sucks as they need to reconnect and there is a small gap. Since it sounds like I will be able to do this so much better with version 2 or perhaps version 3 of Watson I will wait until then before dropping it onto github. The jsmpeg folks should be pretty happy about all of your work! Thanks again

[jchristn]

Hi @winkmichael that's a great question! I haven't implemented support for that yet and looking into what's required it would also require persistent TCP connections. I'll add it to my list and see if it's something I can do (alternatively if you'd like to take a swing at it and send a pull request, that would be awesome too)

An alternative to chunk transfer encoding would be to do range reads and range writes. This would require a layer of abstraction on top of the webserver to accept a PUT/POST to an object, write the range of data, and also accept GET to an object with a specified range (but you'd have to maintain a position/index on your reads/writes). That's something you could do with https://github.com/kvpbase/storage-server (specifically see https://github.com/kvpbase/storage-server/wiki/4.-Objects)

[jchristn]

Hi @winkmichael sorry for the delay in getting back to you. I have a prototype ready and was curious if you would be open to testing it? Also I was wanting to know if you used signature calculation on the client-side (and if so, which algorithm) and perhaps an example client request I could examine?

[winkmichael]

Yes, yes and yes! That is fantastic news, thank you! I would really love to test it out.

I've been back and forth between modifying Watson and forcing the header (which jams me on file locks) to Nginx proxying in front of Watson (which is inconsistent because sometimes the state just closes). So yes I have a good environment setup and would really love to test it out and should be able to give some good feedback. Thank you again!

[jchristn]

Awesome! I created a beta branch for v2.1 (see https://github.com/jchristn/WatsonWebserver/tree/beta-2.1). It is basic chunked transfer encoding, expecting chunked to be set in the Transfer-Encoding header. I've added variables as placeholders for gzip and deflate as well.

The implementation is quite basic and assumes no chunk-signatures, i.e. following directly the model on Wikipedia.

Are you using any type of chunk signatures?

Please feel free to clone/build/test and let me know if it works for you!

Also - an alternative approach to this would be to have a callback for each chunk. Right now it's building the datastream and returning after completion rather than chunk by chunk. But, the resultant HttpResponse will have a populated ContentLength and either Data or DataStream.

[jchristn]

Found some issues I need to resolve, will work on these today/tomorrow. Wanted to give you a heads-up before you test. Details on the questions above will definitely be helpful, too! Thanks!

[jchristn]

Hi @winkmichael I just pushed a new commit that seems to work. Please let me know if any issues. Note: it does not include gzip or deflate as yet.

For reference, the first simple test I used was:

POST /
Request body:
6<crlf>
Hello_<crlf>  // _ was a space
5<crlf>
World<crlf>
0<crlf>
<crlf>

This seemed to be correct according to https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding

Thanks!

[jchristn]

Also I isolated the decoding logic into another project (avail in NuGet: https://www.nuget.org/packages/ChunkDecoder/1.0.0) in anticipation of needing to handle chunk signature validation, extra callbacks, etc.

[jchristn]

And the latest version now has callbacks for validating signatures/other key-value pairs against the associated chunk, and callbacks for when a chunk is received. https://www.nuget.org/packages/ChunkDecoder/

[jchristn]

I tested this out on a couple of other projects and it seems to be working well. I removed the explicit code from within Watson and used the ChunkDecoder library internally, and pushed a commit to branch beta-2.1, which I'll be merging into master now. Pushing this version to NuGet - this release won't callback to validate each chunk (against supplied signatures, which ChunkDecoder can do), and also doesn't yet handle gzip/deflate transfer encoding (let me know if you need these). If you encounter any issues, please let me know!

[winkmichael]

Good Evening Joel

I was traveling this weekend but started to look at this a bit earlier this afternoon. The .net 2.2 core threw my linux and OSX environments for a loop. I am just waiting for a few hundreds of Windows Updates to finish on my new Windows 10 install to expedite this.

My initial review of the code thinks that ~485 of WatsonWebServer.cs will need to be tweaked to check the file length has changed / or if we are really at the end. The idea being that chunked encoding will deliver a file of unknown length, or a file that is actively being written to.

I am seeing you added a CORS concepts with the Access-Control *. If I understand the code (again haven't compiled) I think this might run into an issue with too many headers (you can only send it once per session)

Once I get my Windows box up I will get moving on the promised testing. Thank you yet again from what I see here this is looking really sweet!

[jchristn]

Thanks for the feedback! Reopening and will close once tested/resolved etc. Cheers!

[jchristn]

Hi @winkmichael I just tested on my Mac using the Test.Default project - had to do separate builds for WatsonWebserver (dotnet build -f netstandard2.0) and Test.Default (dotnet build -f netcoreapp2.2) and tested with both standard POST data as well as chunk-encoded and both seemed to work on my end.

The ContentLength64 is set explicitly by the response. With the way I've implemented here (in Watson) the server will handle the chunk-encoding and dump a single HttpRequest object to the callback, rather than call the callback for each chunk.

Do you prefer a model whereby you have a callback for each of the chunks? I can do that (and it would be relatively easy to expose since Watson is now using ChunkDecoder, which has this).

Looking forward to your further testing. Cheers, Joel

[jchristn]

Hi @winkmichael any update from your end? Cheers

[winkmichael]

I am very sorry for the lack of response, had a family emergency. I will look at this one this weekend and report back. Thanks again!

[jchristn]

No sweat @winkmichael :)

[jchristn]

Hoping everything worked out with the family emergency. Please re-open if you encounter issues. Thanks

[winkmichael]

Hello Joel,

Apologies for the slumber on the response.

Do you prefer a model whereby you have a callback for each of the chunks? I can do that (and it would be relatively easy to expose since Watson is now using ChunkDecoder, which has this).

Can this be a chunk encoder where I am able to continually add input to the output?

Thanks,
Mike

[jchristn]

Hi @winkmichael I'm thinking of a model where something slightly different from HttpRequest and HttpResponse are used. One where, within the callback, you'd be able to explicitly call ReadChunk and WriteChunk, thereby allowing bidirectional chunked data exchanges. I haven't though through the implementation quite yet (there was another gentleman @xmike402 that had a use case that would also demand this type of function).

[jchristn]

FYI I've started working on a solution to this. It's going to demand that I make a few breaking changes but will better position the project to support future enhancements as well. I'll keep you both posted!

[winkmichael]

That is super, thank you very much in advance! I've read over your comment in regards to a callback, are you proposing a loop method where data is sent and then a callback is tiggered or am I misunderstanding? Breaking is fine by me, I've got a number of small projects using watson but would be more than happy to update for this type of advanced functionality. Web sockets are great, but being able to do this would be preferable to many I suspect.

[jchristn]

Hi @winkmichael I'm thinking of a model where the callback receives an HttpRequest and is expected to return an HttpResponse object with data, or, manually control sending the headers from the HttpRequest followed by a series of chunks. Similarly when extracting data from the HttpRequest either reading directly from the stream (if the request is not chunk encoded) or having a method along the lines of .ReadNextChunk() that would pull data from the stream chunk by chunk. Long way to go on it :)

[jchristn]

@winkmichael and @xmike402 I'd love to get your input on this.

Obviously there's a simplicity in having callbacks of the form public HttpResponse ProcessMyRequest(HttpRequest req). I get an HttpRequest and return an HttpResponse.

However for a chunked transfer (send or receive, either way), the consuming code will be responsible for interacting with the stream in both directions. The implications here are that:

• HttpResponse must be constructed before sending to the consuming code and have the HTTP elements initialized beforehand
• The consuming code would become responsible for sending data, chunk by chunk. Internally, HttpResponse would need to track whether or not the headers for the response had been sent
• When finished, the HttpResponse would need to be closed so that streams could be closed/disposed and such

This would likely result in a callback of the form public Task<bool> ProcessMyRequest(HttpRequest req, HttpResponse resp).

I think it's straight forward to do it this way.

The heartburn I have is that I should migrate to this model wholesale, but then anyone that has integrated up to v2.x (this new version would definitely need to be a 3.0) would have a pretty big change on their hands.

I think it's the right way to go, but it feels like a heavy lift, particularly for those upgrading from v2.x to v3.x, and I don't see a way to support both callback signatures going forward since a lot of the logic would need to go into HttpRequest itself.

Alternatives I've also considered but am leaning away from include:

• Doing a separate project specifically to address chunked transfers
• Using separate routes for chunked APIs so that existing routes with existing callback signatures could be preserved

Of these two, the first one is ugly, and doesn't allow both chunked and non-chunked in the same library. Which I really don't like. The second one would require some gymnastics internally within the code and could create unforseen future technical debt. And, I'd need to be worried about handling static, dynamic, content routes for both chunked and non-chunked, and they would be separate.

It seems the right path forward is the wholesale migration to the new callback signature. Would you have a moment to share your opinion on this?

Thanks!
Joel

[winkmichael]

Ok, I haven't fully thought through exactly how you are thinking of implementing it, but I wanted to response as I am sorry about asking for your help and then going silent! I'll respond right now in a non-technical way elaborating on my two hoped for use cases with Chunk transfer. #1 Low Latency HLS Delivery, this is basically the idea where a process such as ffmpeg writes h264 files .ts direct to the disk and Watson will be able to serve fully written files and files actively being written to (where the length is unknown hence the chunk) #2 open a http session and read and write semi indefinitely, even with idle periods - basically websockets I guess would a simplified way of saying it however over http without the need to reconnect.

I'll put some more thought into it and write back, now that I've started to think it through I don't think there is simple answer on how to implement a "streaming http server" as that might be what it would be called.

[jchristn]

Thanks @winkmichael - this is really great information. I think the approach above would satisfy both use case #1 and #2. With both cases, you'd be able to have a filestream open against the local file, and as new data is read from the filestream, you could send it as a chunk, keeping the connection open. This would work both from the perspective of the endpoint sending to Watson as well as from Watson sending out to recipients.

Cheers, Joel

[jchristn]

Hi @winkmichael and @xmike402 great news! I have a beta branch of v3.0 posted (see https://github.com/jchristn/WatsonWebserver/tree/beta-3.0) which is a pretty massive refactor with lots of breaking changes but also much, much better support for chunked transfer-encoding.

I'll be working on the README for a bit but hope to get it updated in the beta3.0 branch in the next day (maybe sooner, depends on how this evening plays out).

The biggest change is that now callbacks have a signature of Task MyRouteProcessor(HttpContext ctx). HttpContext now includes both the HttpRequest and the HttpResponse objects. Both are initialized prior to the callback, so instead of returning a populated HttpResponse you'll modify the parameters in the context object, and then use the various .Send() methods on HttpContext.HttpResponse to actually send the data.

I'm going to have to be very careful with the NuGet package description on this one to inform people upgrading that there's a pretty large shift to upgrade to 3.0.

For a working example of chunked transfer-encoding, please refer to the project Test.ChunkServer and more specifically https://github.com/jchristn/WatsonWebserver/blob/beta-3.0/Test.ChunkServer/Program.cs

You'll see that if a POST/PUT is received with chunked transfer encoding, it reads out chunk by chunk.

Similarly, if a GET is done against /img/watson.jpg or txt/test.txt the files are read and sent in chunks.

I tested both receive and send. Receive using Postman (a hand-crafted requested body) and send using Chrome/Postman/Firefox. It seems to be working really well.

For sending chunks, there's .SendChunk() and .SendFinalChunk(). SendFinalChunk will send the final 0<crlf><crlf> to indicate the end of transmission.

I'll keep you posted - but if you have any feedback before then please let me know! Thanks, Joel

Edit:
This refactor will also make the project a lot friendlier to some of the future/planned enhancements. So I'm glad you both gave me the push to go this direction. It's probably better that I make this type of major change now while the number of integrations is still modest rather than down the road when there are a bajillion people using it. Also, with a context object, it's starting to look more like HttpListenerContext :-D

[jchristn]

Updated README here: https://github.com/jchristn/WatsonWebserver/tree/beta-3.0

Code has been documented, improved protection levels on certain class members (many were public when they should have been internal).

Feedback would be hugely valuable!

[winkmichael]

Thank you Joel, its rather late in my time zone so I won't have a chance to experiment tonight. I did chuckle to myself and a bit out loud when I saw the github email notification - I was discussing with my wife at dinner the changes in what a web server does these days or is expected to do, with the frame work of providing a better response from my original response of, "hey it would be great if you could support these things I want to do..." Lo and behold you already have what appears to be a working beta that covers everything I mentioned and then some. From your example code it appears you've actually done the bulk of the heavy lifting in terms of my low latency hls objective, so I am pretty sure I will have a working prototype by tomorrow. I will post back asap. Thank you again, that really is a slick looking implementation.

[xmike402]

I'm trying to use ws:// instead of http:// on a request and not finding much joy. Should I not expect to be able to use these protocols interchangeably now?
Thanks
Mike

[jchristn]

Hi @xmike402 WatsonWebserver never supported websockets. Are you saying that you were able to issue requests using ws:// previously, i.e. in v2.x?

[xmike402]

No sir, I've never been able to use it with either. I misunderstood a little, I thought that the "upgrade" method was added and that would allow me to use ws://. I am trying to accept http from ffmpeg, which is actually working good with the chunk method added! I ended up making the broadcast method using multicast, a bit hacky but I receive the data from ffmpeg and all the viewers get it back, but I didn't realize I can't use ws:// The jsmpeg system uses ws:// wss:// to receive the data. I've been just started looking at your WatsonWebsocket project. Thanks

[jchristn]

Makes sense @xmike402 perhaps the right approach is a backend that uses both.

Thanks for getting back to me - I'll close this issue and merge beta-3.0 into master today and get a NuGet update going.

Thanks!