HBONE Specification? about ztunnel

[kyessenov]

This might be intentional since we don't want Istio to reinvent the wheel. The standardization effort is done by https://www.ietf.org/archive/id/draft-schinazi-masque-02.html. The specific instance used by Istio differs in two ways:

• it's mTLS, not TLS - QUIC implementations are not ready for mutual yet
• it's only TCP - hence it's regular HTTP CONNECT with authority populated (no need for extended or UDP yet)

The operational concerns (stream management, flow control) are handled via regular HTTP means.

The one extra header (baggage) has been removed from the protocol specification after some extensive discussion.

[kyessenov]

Based on my (admittedly limited) understanding of Rust, and in line with what you've written above, it seems that the transport connection is H2 frames over TCP. Identical (i.e., based on client and server identities) connection are pooled into a single ztunnel (TCP socket). Is that correct?

Yes. This is the same for Envoy, and should not be Rust specific, really.

Also, which identities (i.e., certificates) are used between ztunnels? I think it the workload's identities upon each TCP connection establishment, but would like to confirm. In addition, since it is H2/TCP, it would seem that mTLS is supported, but I may have misunderstood the Rust TLS acceptor set up.

Ztunnel doesn't present its own identity, it presents the identities of the source/destination pods on their behalf. This is a limitation, but it works for now.

I understand how the operational concerns of different (i.e., "unpooled") connections between ztunnel gateways are handled, as each is its own TCP connection and the kernel stack would handle fairness, congestion, etc. It is less clear to me how operational concerns of multiple client-server connections, using the same tunnel, are handled. Is the ztunnel implementation responsible for allocating fair share of the TCP stream between multiple pooled streams?

Yes, it should be responsible for fair-sharing network resources and handling the noisy neighbors. This is my main concern about production-readiness of ambient - I don't think we have enough evidence the current settings are adequate.

What was the rationale for pooling (as opposed to opening a new ztunnel socket for each client-server)? Was it resource usage, latency in establishing new streams (the current scheme may avoid roundtrips used for TCP establishment), or something else?

The main benefit of pooling is reducing the overhead of the TLS handshake. I don't know how ztunnel pools, but in Envoy we set max streams to 100 so ensure better CPU utilization.

Is there a general timeline for switching over to MASQUE/QUIC (via https://www.ietf.org/archive/id/draft-schinazi-masque-02.html or any of the more the specific encoding specified under the WG)?

You don't need full MASQUE if all traffic is TCP. We will need it when UDP or arbitrary IP traffic becomes in scope for Istio.

[keithmattix]

The closest thing we have is being submitted in a PR: istio/istio#46472. Take a read through that and see what gaps exist in the doc and we'll be happy to add it

[elevran]

thanks @keithmattix !
I'll go through that, noting gaps (if any) in the PR discussion

[kyessenov]

Client authorization is done in a simplistic way: source IP and identity is forged by ztunnel. There's no delegation (separate ztunnel identity) or client forwarding (XFF) that are needed right now, but could be easily added.

[bleggett]

This might be intentional since we don't want Istio to reinvent the wheel.

Well, we invented a codename/acronym, and we have to at least contend with that :D

My $0.02 is just put @kyessenov 's explanation above into architecture/HBONE.md and call it a day.

[elevran]

@kyessenov - apologies for not replying earlier, hope you are ok restarting this conversation.

I've taken the time to read through the ztunnel architecture doc and the code.

This might be intentional since we don't want Istio to reinvent the wheel. The standardization effort is done by https://www.ietf.org/archive/id/draft-schinazi-masque-02.html. The specific instance used by Istio differs in two ways:

• it's mTLS, not TLS - QUIC implementations are not ready for mutual yet
• it's only TCP - hence it's regular HTTP CONNECT with authority populated (no need for extended or UDP yet)

Based on my (admittedly limited) understanding of Rust, and in line with what you've written above, it seems that the transport connection is H2 frames over TCP. Identical (i.e., based on client and server identities) connection are pooled into a single ztunnel (TCP socket). Is that correct?

Also, which identities (i.e., certificates) are used between ztunnels? I think it the workload's identities upon each TCP connection establishment, but would like to confirm. In addition, since it is H2/TCP, it would seem that mTLS is supported, but I may have misunderstood the Rust TLS acceptor set up.

The operational concerns (stream management, flow control) are handled via regular HTTP means.

I understand how the operational concerns of different (i.e., "unpooled") connections between ztunnel gateways are handled, as each is its own TCP connection and the kernel stack would handle fairness, congestion, etc.
It is less clear to me how operational concerns of multiple client-server connections, using the same tunnel, are handled. Is the ztunnel implementation responsible for allocating fair share of the TCP stream between multiple pooled streams?

What was the rationale for pooling (as opposed to opening a new ztunnel socket for each client-server)? Was it resource usage, latency in establishing new streams (the current scheme may avoid roundtrips used for TCP establishment), or something else?

Is there a general timeline for switching over to MASQUE/QUIC (via https://www.ietf.org/archive/id/draft-schinazi-masque-02.html or any of the more the specific encoding specified under the WG)?

[craigbox]

https://github.com/istio/istio/blob/master/architecture/ambient/ztunnel.md#hbone

enough there now?

[bleggett]

https://istio.io/latest/docs/ambient/architecture/hbone/ is published now and is actually likely the most complete description of the protocol at this point, inward-or-outward-facing - between these two it's as described as it needs to be IMO.

[elevran]

Thanks!
Between https://github.com/istio/istio/blob/master/architecture/ambient/ztunnel.md#hbone and https://istio.io/latest/docs/ambient/architecture/hbone/ there's enough info, IMHO.
You may close.

[howardjohn]

thanks!