Comments (17)
Thanks! That should be plenty for now.
from quinn.
Thanks for the quick fix, I've tested it locally and stateless resets from the server now work perfectly!
from quinn.
Your understanding is correct. However, stateless resets are cryptographically secured via EndpointConfig::reset_key
, which ensures that an attacker can't kill your connection by injecting packets. Endpoints drop stateless resets that use an incorrect key, so if you want to send useful resets after a restart, you need to persist your reset key somehow.
from quinn.
Got it, so the fact that a new, random reset key is being created each time an endpoint is created (because of using Endpoint::default) is leading to this behavior but it can be fixed by configuring an explicit reset key. I'll investigate doing that and report back if I still run into any snags. Thanks for the clarification!
from quinn.
Ok I'm back! Even with a consistent reset key i seem to be running into some issues. Let me explain the setup:
- Endpoint
A
andB
both are using a fixed reset key of[0; 64]
A
establishes a connection withB
(so A acts as the client and B as the server)- If
A
restarts it'll end up seeing packets fromB
for the old connection and send a stateless reset.B
properly processes the stateless reset and immediately closes the old connection.
2022-11-15T22:17:39.299060Z DEBUG quinn_proto::endpoint: sending stateless reset for c5adab5a34cb7941 to [::1]:8080
2022-11-15T22:17:39.299225Z DEBUG quinn_proto::connection: got stateless reset
- If
B
restarts it'll end up seeing packets fromA
for the old connection and send a stateless reset.A
then seems to reject the stateless reset and doesn't close the old connection till the idle timeout period has elapsed.
2022-11-15T22:20:41.833882Z DEBUG quinn_proto::endpoint: sending stateless reset for 7fcaf33e19482a04 to [::1]:8081
2022-11-15T22:20:41.834624Z TRACE quinn_proto::connection: unable to complete packet decoding: invalid header: packet too short to extract header protection sample
2022-11-15T22:20:41.834752Z TRACE quinn_proto::connection: unable to complete packet decoding: invalid header: packet too short to extract header protection sample
Is there something else I'm missing or is there some issue with a restarted "server" side of a connection sending a stateless reset to the client side of a connection?
from quinn.
Stateless resets should work in both directions. If anything, server-sent ones should be more robust. This is tested here:
quinn/quinn-proto/src/tests/mod.rs
Lines 166 to 193 in 1d390af
Do you have a packet capture and/or reproducible test case? It's interesting that you're seeing too-short packet errors, as stateless resets shouldn't trigger that, so we may have a bug there.
from quinn.
Yes i do have a reproducible case which you can find here: https://github.com/bmwill/anemo/tree/stateless-reset
Specifically you can run the following:
# Start the server side
$ RUST_LOG=quinn=debug cargo run --bin stateless-reset -- server
# In another terminal start the client side
$ RUST_LOG=quinn=debug cargo run --bin stateless-reset
You can then Ctrl-C either side of the connection, and then restart it, in order to see the different behavior.
The reproducible case is written in terms of a p2p rpc framework which uses quinn internally. I can try to spend some time distilling this into a lower level repro case if needed (although that's going to take me a bit more time).
from quinn.
I'm unable to reproduce using that example. Stateless resets are recognized in both directions.
from quinn.
Hmm I'm able to reproduce the issue locally very consistently (and seeing the too-short packet errors), although i have seen it succeed once or twice. For what its worth I'm doing this on an M1 mac.
Is there any other information that I could capture locally that would help with the debugging?
from quinn.
A packet capture could be revealing.
from quinn.
FWIW, I can also reproduce this on my M1 Mac -- for me it seemed to work okay when I restarted the client, but when I restarted the server it got stuck resending stateless reset
s.
from quinn.
@djc Glad you're able to reproduce this issue as well. I was worried for a second this was going to be a case of "this only doesn't work on my machine" :D
@Ralith I'll work on getting a packet capture
from quinn.
It's definitely plausible for there to be quirks on macOS, as we have slightly different low level I/O code paths there. Still, hard to imagine how this behavior would result. Looking forward to seeing details.
from quinn.
Ok here is a packet capture for the scenario where the server is restarted, tries to send a stateless reset, but the client side is unable to process the reset. https://gist.github.com/bmwill/bfa7a2b64cd5e05c69f40837434a227b
https://gist.github.com/bmwill/bfa7a2b64cd5e05c69f40837434a227b#file-gistfile1-txt-L174-L178 this is the last packet sent from the server-side before it was killed.
https://gist.github.com/bmwill/bfa7a2b64cd5e05c69f40837434a227b#file-gistfile1-txt-L270-L279 is the first time that the server starts sending packets again once its been restarted, and this should be a capture of the reset packets being sent to the client. https://gist.github.com/bmwill/bfa7a2b64cd5e05c69f40837434a227b#file-gistfile1-txt-L290-L299 is the second set of reset packets sent from the server to the client.
Finally after the idle-timeout period the client kills the old connection and then begins re-establishing a new connection starting at https://gist.github.com/bmwill/bfa7a2b64cd5e05c69f40837434a227b#file-gistfile1-txt-L300.
from quinn.
Any chance you can provide that in pcap format, with accompanying Quinn trace logs?
from quinn.
Yes of course, sorry. I've uploaded a new set of files here: bmwill/anemo@7ea31e2.
- stateless-reset.pcap is the packet capture
- client contains the client logs with RUST_LOG=quinn=trace
- server contains the client logs with RUST_LOG=quinn=trace
There's also versions of client/server logs that have color codes if that's useful.
from quinn.
This was due to PING
-only packets generating very small stateless resets, combined with us not checking for stateless resets before bailing out if a packet is too small to be decoded. Fix for both issues up in #1446.
from quinn.
Related Issues (20)
- Rotation of Connection IDs HOT 5
- How to run insecure connection example? HOT 1
- Weird issue when transferring large file HOT 3
- `ACKs are delivered in order` panic when packets are reordered
- Allow a client to specify Initial Connection ID HOT 2
- Bundle flow control and ACK frames opportunistically
- RTT calculation of connection is pretty unreliable when _just_ using the library as is HOT 8
- `SendStream::stopped` suffers excessive delay in terminating with `ZeroRttRejected`
- Dropping `Endpoint` can cause unnecessary "Incoming dropped without passing to..." warnings
- Allow certificate access on Connecting HOT 5
- `SendStream::finish()` no longer errors when connection is closed HOT 2
- Allow disabling backtraces in panics HOT 3
- Consider adding an expect message to unwrap in packet_builder.rs HOT 6
- quinn is failing `zerortt` interop HOT 1
- Panic with tokio 1.39 HOT 8
- `handshakecorruption` crash HOT 5
- `recvmmsg` is disallowed by seccomp on Android x86 HOT 4
- how to use quinn to create an echo server and client? HOT 7
- Add echo server & client for examples HOT 1
- How to use empty cert connect my quic server using quinn? HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from quinn.