metadata validation disallows UTF-8/ISO-8859-1 characters in headers about grpc-go

[arvindbr8]

The receive side does not go through metadata.Validate codepath. This happens only on the send side. Which means we do not error if the server receives a header with characters outside the ASCII range.

I double checked this ^^ using this test https://github.com/arvindbr8/grpc-go/blob/7cb635a70fcfc8dc6cf71c71554ff8cac4326568/test/end2end_test.go#L922

a workaround for grpc users has been pushed

Curious to know what the workaround here is

[CosmicToast]

Hmm, in that case that sounds like an argo bug indeed.
That means that the bug is fundamentally caused by the argo cd devs copying the http request they receive verbatim and retransmetting it via grpc.

I'm terribly sorry about the misunderstanding, their sources were a little hard to read, but I did manage to follow the trail all the way to the abovementioned line.

Given this, I'll bring it back up with them.

Curious to know what the workaround here is.

Tailscale will mime-qencode non-us-ascii header values.
This means that even services like argo that blindly copy over the http request will now work with it.
You can find the details in the mentioned issue: tailscale/tailscale#11603

[CosmicToast]

A very common triggering scenario are ingress nodes adding additional metadata for downstream services.
For example, X-GeoIP-City and tailscale-user-name.
An example downstream bug caused by this is argoproj/argo-workflows#12721.

[arvindbr8]

@CosmicToast It would help us if you let us know where is the call failing, is this on the send or the receive side?

Also per our spec, gRPC restrictions on the header values are more strict than of the HTTP spec. gRPC-go right now throws an error, while we should be ignoring and throwing away the header key-value.

However, there isnt a possibility of transmitting the value through the gRPC service.

[CosmicToast]

It would help us if you let us know where is the call failing, is this on the send or the receive side?

It's the receive side.
The pipeline looks something like this:

1. The sender makes a gRPC request over HTTP (either 1.1 and 2).
2. This call is intercepted by a reverse proxy. This is for authentication and common reverse proxy use-cases. The vast majority of internet traffic goes through one, but enterprise environments tend to add headers...
3. The reverse-proxy adds some headers for downstream consumers to optionally use. In my case, it's tailscale adding tailscale-user-name. In the linked bug case, it's nginx adding X-GeoIP-City.
4. The message continues on to the receiver (in my case, argo workflows server), at which point it is rejected since a header contains a character from Latin Extended 1.

Also per our spec, gRPC restrictions on the header values are more strict than of the HTTP spec. gRPC-go right now throws an error, while we should be ignoring and throwing away the header key-value.

I agree!
That's my desired resolution.
Specifically, https://github.com/Patrick0308/grpc-go/blob/master/internal/metadata/metadata.go#L104 should not return an error but drop the header in question.

However, there isnt a possibility of transmitting the value through the gRPC service.

I'm not entirely sure what you mean in this respect.
Could you clarify your comment a little?
It seems to imply that the abovementioned scenario cannot possibly happen, but it quite clearly does.

[CosmicToast]

As an update, tailscale should no longer be affected (a workaround for grpc users has been pushed).
This will continue to affect nginx users with x-geoip-city and similar headers, so I'm leaving this open for now.
If there's a definitive fix on that end as well then this can probably just be closed.