Implement connection pooling about ztunnel

[howardjohn]

@EItanya is working on this I think, but cannot assign since they are not in the org yet

[stevenctl]

HBONE Connections between zTunnels from the same source and destination Pods/IPs should be reusable across multiple underlying client connections (streams inside HBONE).

[stevenctl]

HBONE Connections between zTunnels from the same source and destination Pods/IPs should be reusable across multiple underlying client connections (streams inside HBONE).

[linsun]

Speaking with Eitan, he is interested in this.

[EItanya]

I am indeed!

[SkyfireFrancisZ]

@@EItanya, please update the estimated Size for this workitem.

[EItanya]

I am off this week, but I will try and have one as soon as I'm back

[SkyfireFrancisZ]

Dev environment setup, start digging into actual connection pooling today.

[EItanya]

After quite a bit of digging I have found a number of paths forward. First some background info.

The http library that ztunnel uses is called hyper. Hyper has it's own connection pooling mechanism build into it's http client: https://github.com/hyperium/hyper/blob/0.14.x/src/client/pool.rs

However, this implementation doesn't work in our case for a number of reasons:

1. We cannot use the client impl they have due to the issue @howardjohn found hyperium/hyper#2863.
2. The pooling mechanism only uses protocol/authority, but we need the client id as well. https://github.com/hyperium/hyper/blob/d32beb3843179acdaf0e64f3e708b9dbcd0c29e4/src/client/pool.rs#LL58

The three paths forward are as follows:

1. Write an entirely new pooling mechanism from scratch. I do not recommend this as the code is quite complex, and hyper has presumably been fixing and enhancing this for quite a long time.
2. Copy/Paste their pooling implementation. At first I thought this would be the best path forward, but unfortunately a lot of the impl is private, so copying the pool impl would require copying their connection impl as well.
3. Work with upstream to fix the few aspects of their impl that don't work for us. This is my favorite option because the number of changes are low, and the maintainer seems very willing to work with us. See this issue for reference. Just to test this hypothesis, I put the generic key impl up for review: hyperium/hyper-util#12

TLDR: I don't think it's worth rushing forward on our own implementation, or completely copy pasting unless the timeline on this requires it being done in the next 1-2 weeks. Doing either of those would in my opinion create more tech debt than is worth it.

[linsun]

Risk areas - 1) may not work with tokio, 2) dependency on upstream being responsive

Going w/ option 3 for now.

[stevenctl]

Waiting on updates from hyper dev

[stevenctl]

May need to handoff.

[SkyfireFrancisZ]

John working on a PR, ETA 1wk.

[SkyfireFrancisZ]

Waiting on Steven to review PR

[SkyfireFrancisZ]

waiting on upstream

[SkyfireFrancisZ]

Blocker on Hyper team has resolved, just waiting for their feedback before we commit. This task is not a blocker for anything else.