New NBD protocol based on HTTP/2 about nbd

[socketpair]

(Sorry, I have very old laptop, so I can't edit messages)
Also, HTTP/2 reduces RTT, has internal PING, allows to read as a stream, say, 42 GB without blocking any other operations, has dependency tree of requests and priorities.

[abligh]

Your comparisons seem to be against HTTP. I don't immediately see why HTTP/2 would help NBD (or indeed NFS or any other native binary protocol). HTTP/2 doesn't provide a similar command set either. On the plus side, NBD will run over any connection oriented protocol. So there is no reason you couldn't tunnel NBD inside HTTP/2, though I can't immediately see the application.

…

On 14 Oct 2017, at 16:26, Коренберг Марк ***@***.***> wrote: (Sorry, I have very old laptop, so I can't edit messages) Also, HTTP/2 reduces RTT, has internal PING, allows to read as a stream, say, 42 GB without blocking any other operations, has dependency tree of requests and priorities. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

-- Alex Bligh

[socketpair]

No, HTTP/2 is completely different protocol. Yes it’s slightly related to HTTP/1.1. But actually, it’s generic stream multiplexer.

For example, two simultaneous write commands should look like two parallel streams inside one TCP-connection, multiplexed by HTTP/2. So, I don't mean wrapping current protocol in HTTP/2. I mean implementing same semantic as in current NBD, but using framing from HTTP/2. Just like gRPC does.

Do you familiar with details of HTTP/2 ?

[abligh]

Yes I am reasonably familiar with HTTP/2 / RFC7540; I am just not entirely clear what you want to do with it. I am aware that it gives you stream multiplexing (though I believe it's actually more HTTP specific than that, e.g. header compression). However it also does not support many things already in the NBD protocol (for instance the negotiation phase), and adds significant complexity; therefore in order to support it there would need to be a really good reason. HTTP/2's multiplexing seems to me only really a benefit for short lived sessions where the alternative would be opening multiple separate TCP connections. NBD sessions are typically long lasting, and there is nothing to stop you opening more than one of them. However, I am interested in learning more about the use case where stream multiplexing NBD requests would give significant performance improvements over conventional NBD (possibly with multiple sessions); could you tell me more about that? For rotational media (the norm when NBD was first designed) this is normally an antipattern, but I accept that things have changed since then. However, I still can't see it buying much as in general requests are serialized by the server at some level in any case. What does HTTP/2 framing actually buy you?

…

On 14 Oct 2017, at 19:32, Коренберг Марк ***@***.***> wrote: No, HTTP/2 is completely different protocol. Yes it’s slightly related to HTTP/1.1. But actually, it’s generic stream multiplexer. For example, two simultaneous write commands should look like two parallel streams inside one TCP-connection, multiplexed by HTTP/2. So, I don't mean wrapping current protocol in HTTP/2. I mean implementing same semantic as in current NBD, but using framing from HTTP/2. Just like gRPC does. Do you familiar with details of HTTP/2 ? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

-- Alex Bligh

[socketpair]

Well, I use NBD with qemu-nbd to write qcow2 images from CEPH. In order to increase performance, I read from CEPH with 10 streams in parallel. In my code, it writes to NBD in parallel in one connection. And I need to wait while really big chunk is written to socket (4 MB) before small chunk from another stream can be written. 4 MB is the standard chunk in RBD. And if there are 9 big chunks in queue, everyting is worth.

Yes, I can open 10 NBD connections, but what is purpose of multiplexing in current NBD then?

[socketpair]

What does HTTP/2 framing actually buy you?

Because server can limit window size for each stream (i.e. each IO request) and allow another IO to be submitted.

[yoe]

The trouble with framing NBD over HTTP/2 is that you'd have to implement HTTP/2 in the kernel. Not sure that's a terribly good idea; it requires a lot of baggage that we really don't need for NBD.

There are some other, lower layer, protocols such as SCTP, which might be worthwhile to layer NBD on top of; but HTTP/2 doesn't seem like a very good idea.

[yoe]

Additionally, once structured replies have been implemented, a server should be able to multiplex multiple read replies in the same server, too.