At home and in Go, I have a number of backends that are a pain to route to because they're buried behind NAT or other firewalls.
In the Go build system, we solve this using an old package I wrote (http://godoc.org/golang.org/x/build/revdial) that lets a backend connect to the server, and then turn the single TCP connection around (after authenticating) and let the server open up and multiplex many TCP connections as needed over that single TCP connection.
That code (first added Sep 2015 in golang/build@1f0d8f2) replaced an earlier "reverse roundtripper" that did the same thing, but only allowed a single HTTP request at a time per connection.
In any case, this is a model I keep returning to and finding super convenient.
To minimize this pain managing backends, I'd like some "revdial" or "backpain" mode in tcpproxy that lets backends register themselves.
The revdial package has been in production for a long time and might be a good start, but it doesn't do back pressure, so it wouldn't be good as a general solution. It only works for us because I know we won't have streams starving each other.
@danderson was suggesting re-using Go's existing HTTP/2 server is probably a better idea, even if it's a bit more work.
The http2 code already has unit tests to verify that full duplex CONNECT requests work over it.
We can just do an HTTP/1.x protocol upgrade over HTTPS with auth to the server proxy which can then Hijack the conn, turn it around, and be an HTTP/2 client to the HTTP/2 server running on the backends. The backend would then handle the incoming CONNECT requests from the server, do the proxying where needed, and let the io.Copy
to the http.ResponseWriter
handle all the flow control automatically via the http2 package.
The code on the backend (which could be an embeddable Go package + a binary for non-Go users) would look at lot this code from Go's build system:
https://github.com/golang/build/blob/d925a7bd2b3ee25956efb3dd98a87c8d3fee7ea6/cmd/buildlet/reverse.go#L72
(but using http2+CONNECT instead of revdial)
Note that it just writes an HTTP/1.x request over HTTPS with a token in it, expects a "101 Switching Protocols", and then switches into becoming a server itself.
The token it sends as auth can also register it as a new Target (https://godoc.org/github.com/google/tcpproxy#Target) implementation on the server side, so we can say "anything matching SNI foo.com should go to backend with token XYZFOOBAR".
@danderson, thoughts?