clarification on stateless reset about quinn

[Ralith]

Thanks! That should be plenty for now.

[bmwill]

Thanks for the quick fix, I've tested it locally and stateless resets from the server now work perfectly!

[Ralith]

Your understanding is correct. However, stateless resets are cryptographically secured via EndpointConfig::reset_key, which ensures that an attacker can't kill your connection by injecting packets. Endpoints drop stateless resets that use an incorrect key, so if you want to send useful resets after a restart, you need to persist your reset key somehow.

[bmwill]

Got it, so the fact that a new, random reset key is being created each time an endpoint is created (because of using Endpoint::default) is leading to this behavior but it can be fixed by configuring an explicit reset key. I'll investigate doing that and report back if I still run into any snags. Thanks for the clarification!

[bmwill]

Ok I'm back! Even with a consistent reset key i seem to be running into some issues. Let me explain the setup:

• Endpoint A and B both are using a fixed reset key of [0; 64]
• A establishes a connection with B (so A acts as the client and B as the server)
• If A restarts it'll end up seeing packets from B for the old connection and send a stateless reset. B properly processes the stateless reset and immediately closes the old connection.

2022-11-15T22:17:39.299060Z DEBUG quinn_proto::endpoint: sending stateless reset for c5adab5a34cb7941 to [::1]:8080
2022-11-15T22:17:39.299225Z DEBUG quinn_proto::connection: got stateless reset

• If B restarts it'll end up seeing packets from A for the old connection and send a stateless reset. A then seems to reject the stateless reset and doesn't close the old connection till the idle timeout period has elapsed.

2022-11-15T22:20:41.833882Z DEBUG quinn_proto::endpoint: sending stateless reset for 7fcaf33e19482a04 to [::1]:8081
2022-11-15T22:20:41.834624Z TRACE quinn_proto::connection: unable to complete packet decoding: invalid header: packet too short to extract header protection sample
2022-11-15T22:20:41.834752Z TRACE quinn_proto::connection: unable to complete packet decoding: invalid header: packet too short to extract header protection sample

Is there something else I'm missing or is there some issue with a restarted "server" side of a connection sending a stateless reset to the client side of a connection?

[Ralith]

Stateless resets should work in both directions. If anything, server-sent ones should be more robust. This is tested here:

quinn/quinn-proto/src/tests/mod.rs

Lines 166 to 193 in 1d390af

	#[test]
	fn server_stateless_reset() {
	let _guard = subscribe();
	let mut reset_key = vec![0; 64];
	let mut rng = rand::thread_rng();
	rng.fill_bytes(&mut reset_key);
	let reset_key = hmac::Key::new(hmac::HMAC_SHA256, &reset_key);
	let endpoint_config = Arc::new(EndpointConfig::new(Arc::new(reset_key)));
	let mut pair = Pair::new(endpoint_config.clone(), server_config());
	let (client_ch, _) = pair.connect();
	pair.server.endpoint = Endpoint::new(endpoint_config, Some(Arc::new(server_config())));
	// Send something big enough to allow room for a smaller stateless reset.
	pair.client.connections.get_mut(&client_ch).unwrap().close(
	pair.time,
	VarInt(42),
	(&[0xab; 128][..]).into(),
	);
	info!("resetting");
	pair.drive();
	assert_matches!(
	pair.client_conn_mut(client_ch).poll(),
	Some(Event::ConnectionLost {
	reason: ConnectionError::Reset
	})
	);
	}

Do you have a packet capture and/or reproducible test case? It's interesting that you're seeing too-short packet errors, as stateless resets shouldn't trigger that, so we may have a bug there.

[bmwill]

Yes i do have a reproducible case which you can find here: https://github.com/bmwill/anemo/tree/stateless-reset

Specifically you can run the following:

# Start the server side
$ RUST_LOG=quinn=debug cargo run --bin stateless-reset -- server

# In another terminal start the client side
$ RUST_LOG=quinn=debug cargo run --bin stateless-reset

You can then Ctrl-C either side of the connection, and then restart it, in order to see the different behavior.

The reproducible case is written in terms of a p2p rpc framework which uses quinn internally. I can try to spend some time distilling this into a lower level repro case if needed (although that's going to take me a bit more time).

[Ralith]

I'm unable to reproduce using that example. Stateless resets are recognized in both directions.

[bmwill]

Hmm I'm able to reproduce the issue locally very consistently (and seeing the too-short packet errors), although i have seen it succeed once or twice. For what its worth I'm doing this on an M1 mac.

Is there any other information that I could capture locally that would help with the debugging?

[Ralith]

A packet capture could be revealing.

[djc]

FWIW, I can also reproduce this on my M1 Mac -- for me it seemed to work okay when I restarted the client, but when I restarted the server it got stuck resending stateless resets.

[bmwill]

@djc Glad you're able to reproduce this issue as well. I was worried for a second this was going to be a case of "this only doesn't work on my machine" :D

@Ralith I'll work on getting a packet capture

[Ralith]

It's definitely plausible for there to be quirks on macOS, as we have slightly different low level I/O code paths there. Still, hard to imagine how this behavior would result. Looking forward to seeing details.

[bmwill]

Ok here is a packet capture for the scenario where the server is restarted, tries to send a stateless reset, but the client side is unable to process the reset. https://gist.github.com/bmwill/bfa7a2b64cd5e05c69f40837434a227b

https://gist.github.com/bmwill/bfa7a2b64cd5e05c69f40837434a227b#file-gistfile1-txt-L174-L178 this is the last packet sent from the server-side before it was killed.

https://gist.github.com/bmwill/bfa7a2b64cd5e05c69f40837434a227b#file-gistfile1-txt-L270-L279 is the first time that the server starts sending packets again once its been restarted, and this should be a capture of the reset packets being sent to the client. https://gist.github.com/bmwill/bfa7a2b64cd5e05c69f40837434a227b#file-gistfile1-txt-L290-L299 is the second set of reset packets sent from the server to the client.

Finally after the idle-timeout period the client kills the old connection and then begins re-establishing a new connection starting at https://gist.github.com/bmwill/bfa7a2b64cd5e05c69f40837434a227b#file-gistfile1-txt-L300.

[Ralith]

Any chance you can provide that in pcap format, with accompanying Quinn trace logs?

[bmwill]

Yes of course, sorry. I've uploaded a new set of files here: bmwill/anemo@7ea31e2.

• stateless-reset.pcap is the packet capture
• client contains the client logs with RUST_LOG=quinn=trace
• server contains the client logs with RUST_LOG=quinn=trace

There's also versions of client/server logs that have color codes if that's useful.

[Ralith]

This was due to PING-only packets generating very small stateless resets, combined with us not checking for stateless resets before bailing out if a packet is too small to be decoded. Fix for both issues up in #1446.