hpack frame decoding fault poisons entire connection about grpc-go

[iamqizhao]

can you share your private reproduction?

On Sun, Apr 12, 2015 at 6:44 PM, David Symonds [email protected]
wrote:

If I have a single gRPC connection and flood it such that a bunch of RPCs
end up timing out, the connection gets into an unrecoverable state. Any
future RPCs that should otherwise work end up failing with errors like
rpc error: code = 13 desc = "transport: HPACK header decode error:
decoding error: invalid indexed representation index 82"
(and the index increments over time).

I traced it to the http2 hpack decoder, which has d.parseHeaderFieldRepr
returning an error. Somewhere after that, though, I'd expect the connection
to recognise that it is stuck and automatically reset. But it doesn't.

I have a private reproduction that appears to be able to reproduce this
100% of the time. I haven't tried reproducing it with a smaller test case
yet.

I don't know who is at fault, grpc-go or http2. But it shouldn't be
possible to send a bunch of RPCs on a gRPC connection such that the
connection ends up poisoned.

/cc @bradfitz https://github.com/bradfitz in case he has thoughts.

—
Reply to this email directly or view it on GitHub
#164.

[dsymonds]

Not publicly, no. I can try to create a self-contained reproduction case though.

[iamqizhao]

grpc should treat it as ConnectionError so that reconnect is triggered (now
it is a StreamError). I am generating a pull request now. But this error is
not expected and we should figure out the root cause. It seems the hpack
state is messed up either due to a miss use in grpc or a bug in hpack impl.

On Sun, Apr 12, 2015 at 7:20 PM, Qi Zhao [email protected] wrote:

can you share your private reproduction?

On Sun, Apr 12, 2015 at 6:44 PM, David Symonds [email protected]
wrote:

If I have a single gRPC connection and flood it such that a bunch of RPCs
end up timing out, the connection gets into an unrecoverable state. Any
future RPCs that should otherwise work end up failing with errors like
rpc error: code = 13 desc = "transport: HPACK header decode error:
decoding error: invalid indexed representation index 82"
(and the index increments over time).

I traced it to the http2 hpack decoder, which has d.parseHeaderFieldRepr
returning an error. Somewhere after that, though, I'd expect the connection
to recognise that it is stuck and automatically reset. But it doesn't.

I have a private reproduction that appears to be able to reproduce this
100% of the time. I haven't tried reproducing it with a smaller test case
yet.

I don't know who is at fault, grpc-go or http2. But it shouldn't be
possible to send a bunch of RPCs on a gRPC connection such that the
connection ends up poisoned.

/cc @bradfitz https://github.com/bradfitz in case he has thoughts.

—
Reply to this email directly or view it on GitHub
#164.

[dsymonds]

Okay. I'll wait to see your pull request and can test it easily enough.

[iamqizhao]

Hi David,

not in good condition to work today. The pull request will be made later.
Sorry about that. It would be great to have a self-contained reproduction
from you so that I can investigate what is going on here. Thanks.

On Sun, Apr 12, 2015 at 7:50 PM, David Symonds [email protected]
wrote:

Okay. I'll wait to see your pull request and can test it easily enough.

—
Reply to this email directly or view it on GitHub
#164 (comment).

[iamqizhao]

okay, it seems I can reproduce it by running TestRPCTimeout test cases a few thousand times. will look into it.

[dsymonds]

Thanks. I haven't had time to create a reproduction from my code, so it's good that you've been able to reproduce it yourself. I can verify a fix that you come up with.

[iamqizhao]

should be fixed via #165. will have another pull request to make it connection though.

[iamqizhao]

s/connection/connection error/

[dsymonds]

Yep, I can no longer reproduce the problem. Thanks!