Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hang during Connecting.await for incoming connections #650

Closed
alecmocatta opened this issue Feb 29, 2020 · 17 comments · Fixed by #676
Closed

Hang during Connecting.await for incoming connections #650

alecmocatta opened this issue Feb 29, 2020 · 17 comments · Fixed by #676

Comments

@alecmocatta
Copy link
Contributor

Running my previous test case further surfaces two more issues:

  • ConnectionError::Reset on line 82. This seems possibly a bug but like the ApplicationClosed error in my previous example it doesn't block progress so is ignored;
  • Hanging in Connecting.await on line 103. strace shows packets are being sent and received, but this .await never returns.

I used the following to simulate an unreliable network on Linux:

tc qdisc add dev lo root netem delay 5ms 10ms 25% distribution normal loss 5% duplicate 5% reorder 40% 50%

Note this seems to be buggy on kernels < 4.18.
@Ralith
Copy link
Collaborator

Ralith commented Feb 29, 2020

ConnectionError::Reset on line 82. This seems possibly a bug but like the ApplicationClosed error in my previous example it doesn't block progress so is ignored

This is expected behavior if the receiver's CONNECTION_CLOSE frames are all lost and its close timeout expires, at which point the connection's state is discarded on the receiver. On the sender's next retransmit, the receiver will be unable to associate the incoming packet with a connection, and will respond by resetting the connection. If this happens rarely and only when simulating packet loss, this is probably working as intended.

Hanging in Connecting.await on line 103. strace shows packets are being sent and received, but this .await never returns.

This sounds like a bug. If you add tracing-subscriber = "0.2.2" to your dependencies, call tracing_subscriber::fmt::init, and set RUST_LOG=quinn=trace in your environment, you'll get diagnostics that will describe the packets being sent/received, which may give us some useful insight into the failure to make progress.

@alecmocatta
Copy link
Contributor Author

alecmocatta commented Mar 1, 2020 •
edited
Loading

I reworked the test case to better isolate the hang. Now it's clear no packets are being sent/received once hung. (the packets I was observing were from connection attempts from other connections.) Here is the tail of the trace; the full trace is here.

Show trace 
...
6224 sender open
Mar 01 19:15:17.636 TRACE quinn_proto::endpoint: initial_dcid=a2d776693aa3127b2dd4bac6797b359cf89f1d9d
Mar 01 19:15:17.636 TRACE quinn_proto::connection: wrote 344 Initial TLS bytes
Mar 01 19:15:17.637 TRACE quinn_proto::connection: 0-RTT enabled
6224 receiver open
6224 receiver open a
Mar 01 19:15:17.637 TRACE drive{id=7}:send{space=Initial pn=0}: quinn_proto::connection: CRYPTO: off 0 len 344
Mar 01 19:15:17.637 TRACE drive{id=7}: quinn_proto::connection: sending 1200 byte datagram
Mar 01 19:15:17.637 TRACE drive{id=5}: quinn_proto::connection: got Data packet (30 bytes) from [::1]:44363 using id c99b2971a20e0250
Mar 01 19:15:17.637 TRACE drive{id=5}:recv{space=Data pn=8}: quinn_proto::connection: draining
Mar 01 19:15:17.637 DEBUG quinn_proto::endpoint: packet for unrecognized connection 91ce75d0c04d1c0b
Mar 01 19:15:17.637 DEBUG quinn_proto::endpoint: sending stateless reset for 91ce75d0c04d1c0b to [::1]:55187
Mar 01 19:15:17.637 DEBUG quinn_proto::endpoint: sending stateless reset for a006c9605d7128c4 to [::1]:44363
Mar 01 19:15:17.638 DEBUG quinn_proto::endpoint: sending stateless reset for 4177b72c77791d57 to [::1]:44363
Mar 01 19:15:17.638 DEBUG quinn_proto::endpoint: packet for unrecognized connection fb7527a3bfa3f52b
Mar 01 19:15:17.638 DEBUG quinn_proto::endpoint: sending stateless reset for fb7527a3bfa3f52b to [::1]:55187
Mar 01 19:15:17.638 DEBUG quinn_proto::endpoint: sending stateless reset for b78222ad1aecfc3c to [::1]:44363
Mar 01 19:15:17.639 TRACE drive{id=5}: quinn_proto::connection: got Data packet (230 bytes) from [::1]:44363 using id c99b2971a20e0250
Mar 01 19:15:17.642 DEBUG quinn_proto::endpoint: packet for unrecognized connection 0179a2796afcec95
Mar 01 19:15:17.642 DEBUG quinn_proto::endpoint: sending stateless reset for 0179a2796afcec95 to [::1]:55187
Mar 01 19:15:17.643 DEBUG quinn_proto::endpoint: sending stateless reset for a4b2f795f231acaa to [::1]:44363
Mar 01 19:15:17.643 TRACE drive{id=0}: quinn_proto::connection: got Data packet (34 bytes) from [::1]:55187 using id d7c8d12c051f7777
Mar 01 19:15:17.646 TRACE first recv: quinn_proto::connection: authenticated
Mar 01 19:15:17.647 TRACE first recv:frame{ty=CRYPTO}: quinn_proto::connection: read 344 TLS bytes
Mar 01 19:15:17.647 TRACE first recv: quinn_proto::connection: Handshake keys ready
Mar 01 19:15:17.647 TRACE first recv: quinn_proto::connection: wrote 96 Initial TLS bytes
Mar 01 19:15:17.647 TRACE first recv: quinn_proto::connection: Data keys ready
Mar 01 19:15:17.647 TRACE first recv: quinn_proto::connection: wrote 136 Handshake TLS bytes
Mar 01 19:15:17.647 TRACE first recv: quinn_proto::connection: 0-RTT enabled
Mar 01 19:15:17.647 TRACE quinn_proto::endpoint: connection incoming id=4 icid=a2d776693aa3127b2dd4bac6797b359cf89f1d9d
Mar 01 19:15:17.647 TRACE drive{id=4}:send{space=Initial pn=0}: quinn_proto::connection: ACK
Mar 01 19:15:17.647 TRACE drive{id=4}:send{space=Initial pn=0}: quinn_proto::connection: CRYPTO: off 0 len 96
Mar 01 19:15:17.647 TRACE drive{id=4}:send{space=Handshake pn=0}: quinn_proto::connection: CRYPTO: off 0 len 136
Mar 01 19:15:17.647 TRACE drive{id=4}: quinn_proto::connection: sending 333 byte datagram
Mar 01 19:15:17.647 TRACE drive{id=4}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=7 id=08e1a59146832066
Mar 01 19:15:17.647 TRACE drive{id=4}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=6 id=cfdca5e26116dd2d
Mar 01 19:15:17.647 TRACE drive{id=4}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=5 id=f9a0e5e7998aa66b
Mar 01 19:15:17.647 TRACE drive{id=4}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=4 id=7559d9ed131f31b3
Mar 01 19:15:17.647 TRACE drive{id=4}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=3 id=ddaf3077e884d580
Mar 01 19:15:17.647 TRACE drive{id=4}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=2 id=29651cc7051ebb64
Mar 01 19:15:17.647 TRACE drive{id=4}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=1 id=3be41d96d9074709
Mar 01 19:15:17.647 TRACE drive{id=4}: quinn_proto::connection: sending 222 byte datagram
Mar 01 19:15:17.647 DEBUG quinn_proto::endpoint: packet for unrecognized connection 52e9a7260dbccf9e
Mar 01 19:15:17.647 DEBUG quinn_proto::endpoint: sending stateless reset for 52e9a7260dbccf9e to [::1]:55187
Mar 01 19:15:17.647 DEBUG drive{id=7}: quinn_proto::connection: discarding unexpected Data packet (222 bytes)
Mar 01 19:15:17.648 TRACE drive{id=7}: quinn_proto::connection: got Initial packet (151 bytes) from [::1]:55187 using id 77dc638991d890af
Mar 01 19:15:17.648 TRACE drive{id=7}:recv{space=Initial pn=0}: quinn_proto::connection: authenticated
Mar 01 19:15:17.648 TRACE drive{id=7}:recv{space=Initial pn=0}: quinn_proto::connection: switching remote CID to ba020a3652fd22b3
Mar 01 19:15:17.648 TRACE drive{id=7}:recv{space=Initial pn=0}:frame{ty=CRYPTO}: quinn_proto::connection: read 96 TLS bytes
Mar 01 19:15:17.648 TRACE drive{id=7}:recv{space=Initial pn=0}: quinn_proto::connection: Handshake keys ready
Mar 01 19:15:17.648 TRACE drive{id=7}: quinn_proto::connection: got Handshake packet (182 bytes) from [::1]:55187 using id 77dc638991d890af
Mar 01 19:15:17.648 TRACE drive{id=7}:recv{space=Handshake pn=0}: quinn_proto::connection: authenticated
Mar 01 19:15:17.648 TRACE drive{id=7}:recv{space=Handshake pn=0}:frame{ty=CRYPTO}: quinn_proto::connection: read 136 TLS bytes
Mar 01 19:15:17.648 TRACE drive{id=7}:recv{space=Handshake pn=0}: quinn_proto::connection: Data keys ready
Mar 01 19:15:17.648 TRACE drive{id=7}:recv{space=Handshake pn=0}: quinn_proto::connection: wrote 36 Handshake TLS bytes
Mar 01 19:15:17.648 TRACE drive{id=7}:recv{space=Handshake pn=0}: quinn_proto::connection: established
Mar 01 19:15:17.649 TRACE drive{id=7}:send{space=Initial pn=1}: quinn_proto::connection: ACK
Mar 01 19:15:17.649 TRACE drive{id=7}: quinn_proto::connection: discarding Initial keys
Mar 01 19:15:17.649 TRACE drive{id=7}:send{space=Handshake pn=0}: quinn_proto::connection: ACK
Mar 01 19:15:17.649 TRACE drive{id=7}:send{space=Handshake pn=0}: quinn_proto::connection: CRYPTO: off 0 len 36
Mar 01 19:15:17.649 TRACE drive{id=7}: quinn_proto::connection: sending 1200 byte datagram
Mar 01 19:15:17.649 TRACE drive{id=7}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=7 id=2d075217f55f3e51
Mar 01 19:15:17.649 TRACE drive{id=7}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=6 id=28953d31598bbc67
Mar 01 19:15:17.649 TRACE drive{id=7}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=5 id=21626571ca2c9749
Mar 01 19:15:17.649 TRACE drive{id=7}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=4 id=dafd68ac7fcdc7b3
Mar 01 19:15:17.649 TRACE drive{id=7}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=3 id=3b74745f26bbc5d7
Mar 01 19:15:17.649 TRACE drive{id=7}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=2 id=423c670a52a41ed5
Mar 01 19:15:17.649 TRACE drive{id=7}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=1 id=0cb47bdb7710a26f
Mar 01 19:15:17.649 TRACE drive{id=7}: quinn_proto::connection: sending 222 byte datagram
Mar 01 19:15:17.649 TRACE quinn_proto::connection: wrote 2 bytes stream=client unidirectional stream 0
6224 sender /open
Mar 01 19:15:17.650 TRACE drive{id=7}:send{space=Data pn=1}: quinn_proto::connection: STREAM id=client unidirectional stream 0 off=0 len=2 fin=false
Mar 01 19:15:17.650 TRACE drive{id=7}: quinn_proto::connection: sending 31 byte datagram
Mar 01 19:15:17.653 TRACE drive{id=4}: quinn_proto::connection: got Initial packet (51 bytes) from [::1]:44363 using id ba020a3652fd22b3
Mar 01 19:15:17.654 TRACE drive{id=4}:recv{space=Initial pn=1}: quinn_proto::connection: authenticated
Mar 01 19:15:17.654 TRACE drive{id=4}: quinn_proto::connection: got Handshake packet (1149 bytes) from [::1]:44363 using id ba020a3652fd22b3
Mar 01 19:15:17.654 TRACE drive{id=4}:recv{space=Handshake pn=0}: quinn_proto::connection: authenticated
Mar 01 19:15:17.654 TRACE drive{id=4}:recv{space=Handshake pn=0}: quinn_proto::connection: discarding Initial keys
Mar 01 19:15:17.654 TRACE drive{id=4}:recv{space=Handshake pn=0}:frame{ty=CRYPTO}: quinn_proto::connection: read 36 TLS bytes
Mar 01 19:15:17.654 TRACE drive{id=4}:recv{space=Handshake pn=0}: quinn_proto::connection: wrote 89 Data TLS bytes
Mar 01 19:15:17.654 TRACE drive{id=4}:recv{space=Handshake pn=0}: quinn_proto::connection: established
Mar 01 19:15:17.654 TRACE drive{id=4}:send{space=Handshake pn=1}: quinn_proto::connection: ACK
Mar 01 19:15:17.654 TRACE drive{id=4}:send{space=Data pn=1}: quinn_proto::connection: CRYPTO: off 0 len 89
Mar 01 19:15:17.654 TRACE drive{id=4}: quinn_proto::connection: sending 169 byte datagram
Mar 01 19:15:17.655 DEBUG quinn_proto::endpoint: packet for unrecognized connection eab7c4690c97dfef
Mar 01 19:15:17.655 DEBUG quinn_proto::endpoint: sending stateless reset for eab7c4690c97dfef to [::1]:55187
Mar 01 19:15:17.655 DEBUG quinn_proto::endpoint: sending stateless reset for bb74b62b8795db17 to [::1]:44363
Mar 01 19:15:17.658 DEBUG quinn_proto::endpoint: packet for unrecognized connection 1b10980359755274
Mar 01 19:15:17.658 DEBUG quinn_proto::endpoint: sending stateless reset for 1b10980359755274 to [::1]:55187
Mar 01 19:15:17.658 DEBUG quinn_proto::endpoint: sending stateless reset for 7868393e0843a962 to [::1]:44363
Mar 01 19:15:17.658 TRACE drive{id=4}: quinn_proto::connection: got Data packet (31 bytes) from [::1]:44363 using id ba020a3652fd22b3
Mar 01 19:15:17.658 TRACE drive{id=4}:recv{space=Data pn=1}: quinn_proto::connection: authenticated
Mar 01 19:15:17.658 TRACE drive{id=4}:recv{space=Data pn=1}:frame{ty=STREAM}: quinn_proto::connection: got stream id=client unidirectional stream 0 offset=0 len=2 fin=false
Mar 01 19:15:17.658 TRACE drive{id=4}:send{space=Data pn=2}: quinn_proto::connection: ACK
Mar 01 19:15:17.658 TRACE drive{id=4}: quinn_proto::connection: sending 34 byte datagram
Mar 01 19:15:17.663 DEBUG quinn_proto::endpoint: packet for unrecognized connection 9be0072e967e3964
Mar 01 19:15:17.664 DEBUG quinn_proto::endpoint: sending stateless reset for 9be0072e967e3964 to [::1]:55187
Mar 01 19:15:17.664 TRACE drive{id=7}: quinn_proto::connection: got Handshake packet (50 bytes) from [::1]:55187 using id 77dc638991d890af
Mar 01 19:15:17.664 TRACE drive{id=7}:recv{space=Handshake pn=1}: quinn_proto::connection: authenticated
Mar 01 19:15:17.664 TRACE drive{id=7}: quinn_proto::connection: got Data packet (119 bytes) from [::1]:55187 using id 77dc638991d890af
Mar 01 19:15:17.664 TRACE drive{id=7}:recv{space=Data pn=1}: quinn_proto::connection: authenticated
Mar 01 19:15:17.664 TRACE drive{id=7}:recv{space=Data pn=1}:frame{ty=CRYPTO}: quinn_proto::connection: read 89 TLS bytes
Mar 01 19:15:17.665 TRACE drive{id=7}:send{space=Data pn=2}: quinn_proto::connection: ACK
Mar 01 19:15:17.666 TRACE drive{id=7}: quinn_proto::connection: sending 34 byte datagram
Mar 01 19:15:17.666 TRACE drive{id=4}: quinn_proto::connection: timeout timer=LossDetection
Mar 01 19:15:17.666 TRACE drive{id=4}: quinn_proto::connection: PTO fired in_flight=341 count=0 space=Handshake
Mar 01 19:15:17.666 TRACE drive{id=4}:send{space=Handshake pn=2}: quinn_proto::connection: PING
Mar 01 19:15:17.666 TRACE drive{id=4}:send{space=Handshake pn=2}: quinn_proto::connection: ACK
Mar 01 19:15:17.666 TRACE drive{id=4}: quinn_proto::connection: sending 51 byte datagram
Mar 01 19:15:17.666 TRACE drive{id=4}:send{space=Handshake pn=3}: quinn_proto::connection: PING
Mar 01 19:15:17.666 TRACE drive{id=4}:send{space=Handshake pn=3}: quinn_proto::connection: ACK
Mar 01 19:15:17.666 TRACE drive{id=4}: quinn_proto::connection: sending 51 byte datagram
Mar 01 19:15:17.666 TRACE drive{id=7}: quinn_proto::connection: got Handshake packet (51 bytes) from [::1]:55187 using id 77dc638991d890af
Mar 01 19:15:17.666 TRACE drive{id=7}:recv{space=Handshake pn=3}: quinn_proto::connection: authenticated
Mar 01 19:15:17.667 TRACE drive{id=7}:send{space=Handshake pn=1}: quinn_proto::connection: ACK
Mar 01 19:15:17.667 TRACE drive{id=7}: quinn_proto::connection: sending 52 byte datagram
Mar 01 19:15:17.669 TRACE drive{id=7}: quinn_proto::connection: got Data packet (34 bytes) from [::1]:55187 using id 77dc638991d890af
Mar 01 19:15:17.669 TRACE drive{id=7}:recv{space=Data pn=2}: quinn_proto::connection: authenticated
Mar 01 19:15:17.669 TRACE drive{id=7}:recv{space=Data pn=2}:frame{ty=ACK}: quinn_proto::connection: discarding Handshake keys
Mar 01 19:15:17.672 TRACE drive{id=7}: quinn_proto::connection: timeout timer=LossDetection
Mar 01 19:15:17.672 TRACE drive{id=7}: quinn_proto::connection: packets lost: [0]
Mar 01 19:15:17.672 TRACE drive{id=7}:send{space=Data pn=3}: quinn_proto::connection: ACK
Mar 01 19:15:17.673 TRACE drive{id=7}:send{space=Data pn=3}: quinn_proto::connection: NEW_CONNECTION_ID sequence=1 id=0cb47bdb7710a26f
Mar 01 19:15:17.673 TRACE drive{id=7}:send{space=Data pn=3}: quinn_proto::connection: NEW_CONNECTION_ID sequence=2 id=423c670a52a41ed5
Mar 01 19:15:17.673 TRACE drive{id=7}:send{space=Data pn=3}: quinn_proto::connection: NEW_CONNECTION_ID sequence=3 id=3b74745f26bbc5d7
Mar 01 19:15:17.673 TRACE drive{id=7}:send{space=Data pn=3}: quinn_proto::connection: NEW_CONNECTION_ID sequence=4 id=dafd68ac7fcdc7b3
Mar 01 19:15:17.673 TRACE drive{id=7}:send{space=Data pn=3}: quinn_proto::connection: NEW_CONNECTION_ID sequence=5 id=21626571ca2c9749
Mar 01 19:15:17.673 TRACE drive{id=7}:send{space=Data pn=3}: quinn_proto::connection: NEW_CONNECTION_ID sequence=6 id=28953d31598bbc67
Mar 01 19:15:17.673 TRACE drive{id=7}:send{space=Data pn=3}: quinn_proto::connection: NEW_CONNECTION_ID sequence=7 id=2d075217f55f3e51
Mar 01 19:15:17.673 TRACE drive{id=7}: quinn_proto::connection: sending 230 byte datagram
Mar 01 19:15:17.676 TRACE drive{id=4}: quinn_proto::connection: got Data packet (34 bytes) from [::1]:44363 using id ba020a3652fd22b3
Mar 01 19:15:17.676 TRACE drive{id=4}:recv{space=Data pn=2}: quinn_proto::connection: authenticated
Mar 01 19:15:17.676 TRACE drive{id=4}:recv{space=Data pn=2}:frame{ty=ACK}: quinn_proto::connection: discarding Handshake keys
Mar 01 19:15:17.676 TRACE drive{id=4}:recv{space=Data pn=2}:frame{ty=ACK}: quinn_proto::connection: packets lost: [0]
Mar 01 19:15:17.676 TRACE drive{id=4}:send{space=Data pn=3}: quinn_proto::connection: ACK
Mar 01 19:15:17.676 TRACE drive{id=4}:send{space=Data pn=3}: quinn_proto::connection: NEW_CONNECTION_ID sequence=1 id=3be41d96d9074709
Mar 01 19:15:17.676 TRACE drive{id=4}:send{space=Data pn=3}: quinn_proto::connection: NEW_CONNECTION_ID sequence=2 id=29651cc7051ebb64
Mar 01 19:15:17.676 TRACE drive{id=4}:send{space=Data pn=3}: quinn_proto::connection: NEW_CONNECTION_ID sequence=3 id=ddaf3077e884d580
Mar 01 19:15:17.676 TRACE drive{id=4}:send{space=Data pn=3}: quinn_proto::connection: NEW_CONNECTION_ID sequence=4 id=7559d9ed131f31b3
Mar 01 19:15:17.676 TRACE drive{id=4}:send{space=Data pn=3}: quinn_proto::connection: NEW_CONNECTION_ID sequence=5 id=f9a0e5e7998aa66b
Mar 01 19:15:17.676 TRACE drive{id=4}:send{space=Data pn=3}: quinn_proto::connection: NEW_CONNECTION_ID sequence=6 id=cfdca5e26116dd2d
Mar 01 19:15:17.676 TRACE drive{id=4}:send{space=Data pn=3}: quinn_proto::connection: NEW_CONNECTION_ID sequence=7 id=08e1a59146832066
Mar 01 19:15:17.676 TRACE drive{id=4}: quinn_proto::connection: sending 230 byte datagram
Mar 01 19:15:17.676 TRACE drive{id=7}: quinn_proto::connection: got Data packet (34 bytes) from [::1]:55187 using id 77dc638991d890af
Mar 01 19:15:17.676  WARN drive{id=7}:recv{space=Data pn=2}: quinn_proto::connection: discarding possible duplicate packet 2
Mar 01 19:15:17.676 TRACE drive{id=7}: quinn_proto::connection: got Data packet (230 bytes) from [::1]:55187 using id 77dc638991d890af
Mar 01 19:15:17.677 TRACE drive{id=7}:recv{space=Data pn=3}: quinn_proto::connection: authenticated
Mar 01 19:15:17.677 TRACE drive{id=7}:recv{space=Data pn=3}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=1 id=3be41d96d9074709
Mar 01 19:15:17.677 TRACE drive{id=7}:recv{space=Data pn=3}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=2 id=29651cc7051ebb64
Mar 01 19:15:17.677 TRACE drive{id=7}:recv{space=Data pn=3}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=3 id=ddaf3077e884d580
Mar 01 19:15:17.677 TRACE drive{id=7}:recv{space=Data pn=3}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=4 id=7559d9ed131f31b3
Mar 01 19:15:17.677 TRACE drive{id=7}:recv{space=Data pn=3}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=5 id=f9a0e5e7998aa66b
Mar 01 19:15:17.677 TRACE drive{id=7}:recv{space=Data pn=3}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=6 id=cfdca5e26116dd2d
Mar 01 19:15:17.677 TRACE drive{id=7}:recv{space=Data pn=3}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=7 id=08e1a59146832066
Mar 01 19:15:17.678 TRACE drive{id=7}:send{space=Data pn=4}: quinn_proto::connection: ACK
Mar 01 19:15:17.678 TRACE drive{id=7}: quinn_proto::connection: sending 34 byte datagram
Mar 01 19:15:17.678 DEBUG quinn_proto::endpoint: sending stateless reset for 5a224220d7d424d1 to [::1]:44363
Mar 01 19:15:17.678 TRACE drive{id=4}: quinn_proto::connection: got Data packet (34 bytes) from [::1]:44363 using id ba020a3652fd22b3
Mar 01 19:15:17.678 TRACE drive{id=4}:recv{space=Data pn=4}: quinn_proto::connection: authenticated
Mar 01 19:15:17.679 DEBUG quinn_proto::endpoint: packet for unrecognized connection 09dad4d9149f5d67
Mar 01 19:15:17.679 DEBUG quinn_proto::endpoint: sending stateless reset for 09dad4d9149f5d67 to [::1]:55187
Mar 01 19:15:17.679 TRACE drive{id=0}: quinn_proto::connection: timeout timer=Close
Mar 01 19:15:17.683 DEBUG quinn_proto::endpoint: packet for unrecognized connection 39494e3e9fa37126
Mar 01 19:15:17.683 DEBUG quinn_proto::endpoint: sending stateless reset for 39494e3e9fa37126 to [::1]:55187
Mar 01 19:15:17.684 DEBUG drive{id=7}: quinn_proto::connection: discarding unexpected Handshake packet (51 bytes)
Mar 01 19:15:17.695 DEBUG quinn_proto::endpoint: sending stateless reset for 4a78077aa7a3f0bf to [::1]:44363
Mar 01 19:15:17.696 DEBUG quinn_proto::endpoint: packet for unrecognized connection 636918961bbc2aca
Mar 01 19:15:17.696 DEBUG quinn_proto::endpoint: sending stateless reset for 636918961bbc2aca to [::1]:55187
Mar 01 19:15:17.697 DEBUG quinn_proto::endpoint: sending stateless reset for 1ed06ab2dfacaac9 to [::1]:44363
Mar 01 19:15:17.697 DEBUG quinn_proto::endpoint: packet for unrecognized connection e8d1618731d626c4
Mar 01 19:15:17.697 DEBUG quinn_proto::endpoint: sending stateless reset for e8d1618731d626c4 to [::1]:55187
Mar 01 19:15:17.697 DEBUG quinn_proto::endpoint: sending stateless reset for b5151cc682dbf6c0 to [::1]:44363
Mar 01 19:15:17.698 DEBUG quinn_proto::endpoint: sending stateless reset for e5696af3177494e2 to [::1]:44363
Mar 01 19:15:17.698 DEBUG quinn_proto::endpoint: packet for unrecognized connection ea0ed18559ac923f
Mar 01 19:15:17.699 DEBUG quinn_proto::endpoint: ignoring unexpected 21 byte packet: not larger than minimum stateless reset size
Mar 01 19:15:17.699 DEBUG quinn_proto::endpoint: packet for unrecognized connection 8039cf89148d78c1
Mar 01 19:15:17.699 DEBUG quinn_proto::endpoint: sending stateless reset for 8039cf89148d78c1 to [::1]:55187
Mar 01 19:15:17.703 DEBUG quinn_proto::endpoint: sending stateless reset for 11cc882e50cee74a to [::1]:44363
Mar 01 19:15:17.710 TRACE drive{id=7}: quinn_proto::connection: timeout timer=LossDetection
Mar 01 19:15:17.711 TRACE drive{id=7}: quinn_proto::connection: PTO fired in_flight=230 count=0 space=Data
Mar 01 19:15:17.711 TRACE drive{id=7}:send{space=Data pn=5}: quinn_proto::connection: ACK
Mar 01 19:15:17.711 TRACE drive{id=7}:send{space=Data pn=5}: quinn_proto::connection: NEW_CONNECTION_ID sequence=7 id=2d075217f55f3e51
Mar 01 19:15:17.711 TRACE drive{id=7}:send{space=Data pn=5}: quinn_proto::connection: NEW_CONNECTION_ID sequence=6 id=28953d31598bbc67
Mar 01 19:15:17.711 TRACE drive{id=7}:send{space=Data pn=5}: quinn_proto::connection: NEW_CONNECTION_ID sequence=5 id=21626571ca2c9749
Mar 01 19:15:17.711 TRACE drive{id=7}:send{space=Data pn=5}: quinn_proto::connection: NEW_CONNECTION_ID sequence=4 id=dafd68ac7fcdc7b3
Mar 01 19:15:17.711 TRACE drive{id=7}:send{space=Data pn=5}: quinn_proto::connection: NEW_CONNECTION_ID sequence=3 id=3b74745f26bbc5d7
Mar 01 19:15:17.711 TRACE drive{id=7}:send{space=Data pn=5}: quinn_proto::connection: NEW_CONNECTION_ID sequence=2 id=423c670a52a41ed5
Mar 01 19:15:17.711 TRACE drive{id=7}:send{space=Data pn=5}: quinn_proto::connection: NEW_CONNECTION_ID sequence=1 id=0cb47bdb7710a26f
Mar 01 19:15:17.711 TRACE drive{id=7}: quinn_proto::connection: sending 230 byte datagram
Mar 01 19:15:17.711 TRACE drive{id=7}:send{space=Data pn=6}: quinn_proto::connection: ACK
Mar 01 19:15:17.711 TRACE drive{id=7}:send{space=Data pn=6}: quinn_proto::connection: NEW_CONNECTION_ID sequence=1 id=0cb47bdb7710a26f
Mar 01 19:15:17.711 TRACE drive{id=7}:send{space=Data pn=6}: quinn_proto::connection: NEW_CONNECTION_ID sequence=2 id=423c670a52a41ed5
Mar 01 19:15:17.711 TRACE drive{id=7}:send{space=Data pn=6}: quinn_proto::connection: NEW_CONNECTION_ID sequence=3 id=3b74745f26bbc5d7
Mar 01 19:15:17.711 TRACE drive{id=7}:send{space=Data pn=6}: quinn_proto::connection: NEW_CONNECTION_ID sequence=4 id=dafd68ac7fcdc7b3
Mar 01 19:15:17.711 TRACE drive{id=7}:send{space=Data pn=6}: quinn_proto::connection: NEW_CONNECTION_ID sequence=5 id=21626571ca2c9749
Mar 01 19:15:17.711 TRACE drive{id=7}:send{space=Data pn=6}: quinn_proto::connection: NEW_CONNECTION_ID sequence=6 id=28953d31598bbc67
Mar 01 19:15:17.711 TRACE drive{id=7}:send{space=Data pn=6}: quinn_proto::connection: NEW_CONNECTION_ID sequence=7 id=2d075217f55f3e51
Mar 01 19:15:17.711 TRACE drive{id=7}: quinn_proto::connection: sending 230 byte datagram
Mar 01 19:15:17.711 TRACE drive{id=4}: quinn_proto::connection: got Data packet (230 bytes) from [::1]:44363 using id ba020a3652fd22b3
Mar 01 19:15:17.711 TRACE drive{id=4}:recv{space=Data pn=6}: quinn_proto::connection: authenticated
Mar 01 19:15:17.711 TRACE drive{id=4}:recv{space=Data pn=6}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=1 id=0cb47bdb7710a26f
Mar 01 19:15:17.711 TRACE drive{id=4}:recv{space=Data pn=6}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: switching to remote CID 1: 0cb47bdb7710a26f
Mar 01 19:15:17.711 TRACE drive{id=4}:recv{space=Data pn=6}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=2 id=423c670a52a41ed5
Mar 01 19:15:17.711 TRACE drive{id=4}:recv{space=Data pn=6}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=3 id=3b74745f26bbc5d7
Mar 01 19:15:17.711 TRACE drive{id=4}:recv{space=Data pn=6}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=4 id=dafd68ac7fcdc7b3
Mar 01 19:15:17.711 TRACE drive{id=4}:recv{space=Data pn=6}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=5 id=21626571ca2c9749
Mar 01 19:15:17.711 TRACE drive{id=4}:recv{space=Data pn=6}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=6 id=28953d31598bbc67
Mar 01 19:15:17.711 TRACE drive{id=4}:recv{space=Data pn=6}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=7 id=2d075217f55f3e51
Mar 01 19:15:17.711 TRACE drive{id=4}:send{space=Data pn=4}: quinn_proto::connection: ACK
Mar 01 19:15:17.711 TRACE drive{id=4}:send{space=Data pn=4}: quinn_proto::connection: RETIRE_CONNECTION_ID sequence=0
Mar 01 19:15:17.711 TRACE drive{id=4}: quinn_proto::connection: sending 38 byte datagram
Mar 01 19:15:17.712 TRACE drive{id=7}: quinn_proto::connection: got Data packet (38 bytes) from [::1]:55187 using id 0cb47bdb7710a26f
Mar 01 19:15:17.712 TRACE drive{id=7}:recv{space=Data pn=4}: quinn_proto::connection: authenticated
Mar 01 19:15:17.712 TRACE drive{id=7}:recv{space=Data pn=4}:frame{ty=ACK}: quinn_proto::connection: packets lost: [3]
Mar 01 19:15:17.712 TRACE drive{id=7}:send{space=Data pn=7}: quinn_proto::connection: ACK
Mar 01 19:15:17.712 TRACE drive{id=7}: quinn_proto::connection: sending 34 byte datagram
Mar 01 19:15:17.712 TRACE quinn_proto::endpoint: peer retired CID 0: 77dc638991d890af
Mar 01 19:15:17.712 TRACE drive{id=7}:send{space=Data pn=8}: quinn_proto::connection: ACK
Mar 01 19:15:17.712 TRACE drive{id=7}:send{space=Data pn=8}: quinn_proto::connection: NEW_CONNECTION_ID sequence=8 id=04233e25e4623ee3
Mar 01 19:15:17.712 TRACE drive{id=7}: quinn_proto::connection: sending 62 byte datagram
Mar 01 19:15:17.715 TRACE drive{id=4}: quinn_proto::connection: got Data packet (62 bytes) from [::1]:44363 using id ba020a3652fd22b3
Mar 01 19:15:17.715 TRACE drive{id=4}:recv{space=Data pn=8}: quinn_proto::connection: authenticated
Mar 01 19:15:17.715 TRACE drive{id=4}:recv{space=Data pn=8}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=8 id=04233e25e4623ee3
Mar 01 19:15:17.715 TRACE drive{id=4}:send{space=Data pn=5}: quinn_proto::connection: ACK
Mar 01 19:15:17.715 TRACE drive{id=4}: quinn_proto::connection: sending 34 byte datagram
Mar 01 19:15:17.719 TRACE drive{id=5}: quinn_proto::connection: timeout timer=Close
Mar 01 19:15:17.725 TRACE drive{id=7}: quinn_proto::connection: timeout timer=LossDetection
Mar 01 19:15:17.725 TRACE drive{id=7}: quinn_proto::connection: packets lost: [5]
Mar 01 19:15:17.727 TRACE drive{id=4}: quinn_proto::connection: got Data packet (230 bytes) from [::1]:44363 using id ba020a3652fd22b3
Mar 01 19:15:17.727 TRACE drive{id=4}:recv{space=Data pn=5}: quinn_proto::connection: authenticated
Mar 01 19:15:17.727 TRACE drive{id=4}:recv{space=Data pn=5}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=7 id=2d075217f55f3e51
Mar 01 19:15:17.727 TRACE drive{id=4}:recv{space=Data pn=5}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=6 id=28953d31598bbc67
Mar 01 19:15:17.727 TRACE drive{id=4}:recv{space=Data pn=5}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=5 id=21626571ca2c9749
Mar 01 19:15:17.727 TRACE drive{id=4}:recv{space=Data pn=5}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=4 id=dafd68ac7fcdc7b3
Mar 01 19:15:17.727 TRACE drive{id=4}:recv{space=Data pn=5}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=3 id=3b74745f26bbc5d7
Mar 01 19:15:17.727 TRACE drive{id=4}:recv{space=Data pn=5}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=2 id=423c670a52a41ed5
Mar 01 19:15:17.727 TRACE drive{id=4}:recv{space=Data pn=5}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=1 id=0cb47bdb7710a26f
Mar 01 19:15:17.727 TRACE drive{id=4}:send{space=Data pn=6}: quinn_proto::connection: ACK
Mar 01 19:15:17.727 TRACE drive{id=4}: quinn_proto::connection: sending 36 byte datagram
Mar 01 19:15:17.727 TRACE drive{id=7}: quinn_proto::connection: got Data packet (36 bytes) from [::1]:55187 using id 0cb47bdb7710a26f
Mar 01 19:15:17.727 TRACE drive{id=7}:recv{space=Data pn=6}: quinn_proto::connection: authenticated
Mar 01 19:15:17.730 TRACE drive{id=7}: quinn_proto::connection: timeout timer=LossDetection
Mar 01 19:15:17.730 TRACE drive{id=7}: quinn_proto::connection: packets lost: [7]
Mar 01 19:15:17.735 TRACE drive{id=7}: quinn_proto::connection: got Data packet (34 bytes) from [::1]:55187 using id 0cb47bdb7710a26f
Mar 01 19:15:17.735 TRACE drive{id=7}:recv{space=Data pn=5}: quinn_proto::connection: authenticated
Mar 01 19:15:17.738 TRACE drive{id=4}: quinn_proto::connection: got Data packet (34 bytes) from [::1]:44363 using id ba020a3652fd22b3
Mar 01 19:15:17.738 TRACE drive{id=4}:recv{space=Data pn=7}: quinn_proto::connection: authenticated
Mar 01 19:15:17.823 TRACE drive{id=4}: quinn_proto::connection: timeout timer=KeyDiscard
Mar 01 19:15:18.635 TRACE drive{id=6}: quinn_proto::connection: timeout timer=LossDetection
Mar 01 19:15:18.635 TRACE drive{id=6}: quinn_proto::connection: PTO fired in_flight=977 count=0 space=Initial
Mar 01 19:15:18.635 TRACE drive{id=6}:send{space=Initial pn=1}: quinn_proto::connection: ACK
Mar 01 19:15:18.635 TRACE drive{id=6}:send{space=Initial pn=1}: quinn_proto::connection: CRYPTO: off 0 len 90
Mar 01 19:15:18.635 TRACE drive{id=6}: quinn_proto::connection: sending 145 byte datagram
Mar 01 19:15:18.635 TRACE drive{id=6}:send{space=Initial pn=2}: quinn_proto::connection: ACK
Mar 01 19:15:18.635 TRACE drive{id=6}:send{space=Initial pn=2}: quinn_proto::connection: CRYPTO: off 0 len 90
Mar 01 19:15:18.635 TRACE drive{id=6}: quinn_proto::connection: sending 145 byte datagram
Mar 01 19:15:18.635 DEBUG quinn_proto::endpoint: packet for unrecognized connection fb7527a3bfa3f52b
Mar 01 19:15:18.635 DEBUG quinn_proto::endpoint: sending stateless reset for fb7527a3bfa3f52b to [::1]:55187
Mar 01 19:15:18.636 DEBUG quinn_proto::endpoint: packet for unrecognized connection fb7527a3bfa3f52b
Mar 01 19:15:18.636 DEBUG quinn_proto::endpoint: sending stateless reset for fb7527a3bfa3f52b to [::1]:55187
Mar 01 19:15:18.638 DEBUG quinn_proto::endpoint: sending stateless reset for 49833e3792c09e6c to [::1]:44363
Mar 01 19:15:18.645 DEBUG quinn_proto::endpoint: sending stateless reset for 6df202f94363d5db to [::1]:44363
Mar 01 19:15:18.645 DEBUG quinn_proto::endpoint: packet for unrecognized connection bc084fbdbd469437
Mar 01 19:15:18.645 DEBUG quinn_proto::endpoint: sending stateless reset for bc084fbdbd469437 to [::1]:55187
Mar 01 19:15:18.648 DEBUG quinn_proto::endpoint: packet for unrecognized connection 3ca5995675bed027
Mar 01 19:15:18.648 DEBUG quinn_proto::endpoint: sending stateless reset for 3ca5995675bed027 to [::1]:55187
Mar 01 19:15:18.649 DEBUG quinn_proto::endpoint: sending stateless reset for ef9374659ac54631 to [::1]:44363
Mar 01 19:15:18.674 DEBUG quinn_proto::endpoint: sending stateless reset for a2568858c0ea4654 to [::1]:44363
Mar 01 19:15:18.689 DEBUG quinn_proto::endpoint: packet for unrecognized connection 0068d341d407620b
Mar 01 19:15:18.689 DEBUG quinn_proto::endpoint: sending stateless reset for 0068d341d407620b to [::1]:55187
Mar 01 19:15:18.695 DEBUG quinn_proto::endpoint: sending stateless reset for a39f0863aaa3bdf5 to [::1]:44363
Mar 01 19:15:18.705 DEBUG quinn_proto::endpoint: packet for unrecognized connection 4a363dcf40662f6f
Mar 01 19:15:18.705 DEBUG quinn_proto::endpoint: sending stateless reset for 4a363dcf40662f6f to [::1]:55187
Mar 01 19:15:18.722 DEBUG quinn_proto::endpoint: sending stateless reset for 585cf3646df0ac32 to [::1]:44363
Mar 01 19:15:18.723 DEBUG quinn_proto::endpoint: packet for unrecognized connection 4d504f51dee03c12
Mar 01 19:15:18.723 DEBUG quinn_proto::endpoint: sending stateless reset for 4d504f51dee03c12 to [::1]:55187
Mar 01 19:15:18.723 DEBUG quinn_proto::endpoint: sending stateless reset for 3e4935d6a09d4dc9 to [::1]:44363
Mar 01 19:15:19.634 TRACE drive{id=6}: quinn_proto::connection: timeout timer=LossDetection
Mar 01 19:15:19.634 TRACE drive{id=6}: quinn_proto::connection: PTO fired in_flight=1267 count=1 space=Handshake
Mar 01 19:15:19.635 TRACE drive{id=6}:send{space=Handshake pn=1}: quinn_proto::connection: CRYPTO: off 0 len 564
Mar 01 19:15:19.635 TRACE drive{id=6}: quinn_proto::connection: sending 610 byte datagram
Mar 01 19:15:19.635 TRACE drive{id=6}:send{space=Handshake pn=2}: quinn_proto::connection: CRYPTO: off 0 len 564
Mar 01 19:15:19.635 TRACE drive{id=6}: quinn_proto::connection: sending 610 byte datagram
Mar 01 19:15:19.635 TRACE drive{id=6}: quinn_proto::connection: blocked by anti-amplification
Mar 01 19:15:19.635 DEBUG quinn_proto::endpoint: packet for unrecognized connection fb7527a3bfa3f52b
Mar 01 19:15:19.635 DEBUG quinn_proto::endpoint: sending stateless reset for fb7527a3bfa3f52b to [::1]:55187
Mar 01 19:15:19.639 DEBUG quinn_proto::endpoint: sending stateless reset for 34d459ace598cd1e to [::1]:44363
Mar 01 19:15:19.642 DEBUG quinn_proto::endpoint: packet for unrecognized connection 6f9b4c1a28101d4a
Mar 01 19:15:19.642 DEBUG quinn_proto::endpoint: sending stateless reset for 6f9b4c1a28101d4a to [::1]:55187
Mar 01 19:15:19.642 DEBUG quinn_proto::endpoint: sending stateless reset for 8db86fdb06424591 to [::1]:44363
Mar 01 19:15:19.642 DEBUG quinn_proto::endpoint: packet for unrecognized connection 196fc13db149f3e5
Mar 01 19:15:19.642 DEBUG quinn_proto::endpoint: sending stateless reset for 196fc13db149f3e5 to [::1]:55187
Mar 01 19:15:19.643 DEBUG quinn_proto::endpoint: packet for unrecognized connection 196fc13db149f3e5
Mar 01 19:15:19.643 DEBUG quinn_proto::endpoint: sending stateless reset for 196fc13db149f3e5 to [::1]:55187
Mar 01 19:15:19.649 DEBUG quinn_proto::endpoint: packet for unrecognized connection fb7527a3bfa3f52b
Mar 01 19:15:19.649 DEBUG quinn_proto::endpoint: sending stateless reset for fb7527a3bfa3f52b to [::1]:55187
Mar 01 19:15:19.657 DEBUG quinn_proto::endpoint: sending stateless reset for a58168ad339cdd35 to [::1]:44363
Mar 01 19:15:19.662 DEBUG quinn_proto::endpoint: sending stateless reset for eef5e31584742dfd to [::1]:44363
Mar 01 19:15:19.663 DEBUG quinn_proto::endpoint: packet for unrecognized connection 1a1a83fc28ef576a
Mar 01 19:15:19.663 DEBUG quinn_proto::endpoint: sending stateless reset for 1a1a83fc28ef576a to [::1]:55187
Mar 01 19:15:19.665 DEBUG quinn_proto::endpoint: sending stateless reset for d876424cd1f149af to [::1]:44363
Mar 01 19:15:19.666 DEBUG quinn_proto::endpoint: packet for unrecognized connection 420949d7f6741931
Mar 01 19:15:19.666 DEBUG quinn_proto::endpoint: sending stateless reset for 420949d7f6741931 to [::1]:55187
Mar 01 19:15:19.666 DEBUG quinn_proto::endpoint: sending stateless reset for 8c40d449a45e5e81 to [::1]:44363
Mar 01 19:15:19.666 DEBUG quinn_proto::endpoint: packet for unrecognized connection 701181ae77937886
Mar 01 19:15:19.666 DEBUG quinn_proto::endpoint: sending stateless reset for 701181ae77937886 to [::1]:55187
Mar 01 19:15:19.668 DEBUG quinn_proto::endpoint: packet for unrecognized connection 45a44f6c06acfcf6
Mar 01 19:15:19.668 DEBUG quinn_proto::endpoint: sending stateless reset for 45a44f6c06acfcf6 to [::1]:55187
Mar 01 19:15:19.678 DEBUG quinn_proto::endpoint: sending stateless reset for c48ead2b3be852ea to [::1]:44363
Mar 01 19:15:19.679 DEBUG quinn_proto::endpoint: packet for unrecognized connection 6e033fdc5278fd01
Mar 01 19:15:19.679 DEBUG quinn_proto::endpoint: sending stateless reset for 6e033fdc5278fd01 to [::1]:55187
Mar 01 19:15:19.687 DEBUG quinn_proto::endpoint: sending stateless reset for dad768036083ff6f to [::1]:44363
Mar 01 19:15:19.687 DEBUG quinn_proto::endpoint: packet for unrecognized connection 71424986f2c0bc40
Mar 01 19:15:19.687 DEBUG quinn_proto::endpoint: sending stateless reset for 71424986f2c0bc40 to [::1]:55187
Mar 01 19:15:19.690 DEBUG quinn_proto::endpoint: sending stateless reset for 9abbb077af42db24 to [::1]:44363
Mar 01 19:15:19.703 DEBUG quinn_proto::endpoint: sending stateless reset for 9b76c7f2bb37b067 to [::1]:44363
Mar 01 19:15:19.703 DEBUG quinn_proto::endpoint: packet for unrecognized connection a5ce76de07ee6fd2
Mar 01 19:15:19.703 DEBUG quinn_proto::endpoint: sending stateless reset for a5ce76de07ee6fd2 to [::1]:55187
Mar 01 19:15:19.722 DEBUG quinn_proto::endpoint: sending stateless reset for 7b7aaafe2d2ee36d to [::1]:44363
Mar 01 19:15:19.722 DEBUG quinn_proto::endpoint: packet for unrecognized connection 27188221066c4098
Mar 01 19:15:19.722 DEBUG quinn_proto::endpoint: sending stateless reset for 27188221066c4098 to [::1]:55187
Mar 01 19:15:19.726 DEBUG quinn_proto::endpoint: sending stateless reset for 763527c678ee8f35 to [::1]:44363
Mar 01 19:15:19.726 DEBUG quinn_proto::endpoint: packet for unrecognized connection 1a9743b14813fca5
Mar 01 19:15:19.726 DEBUG quinn_proto::endpoint: sending stateless reset for 1a9743b14813fca5 to [::1]:55187
Mar 01 19:15:19.726 DEBUG quinn_proto::endpoint: sending stateless reset for f00fe8609650e952 to [::1]:44363
Mar 01 19:15:19.729 DEBUG quinn_proto::endpoint: sending stateless reset for 763527c678ee8f35 to [::1]:44363
Mar 01 19:15:19.729 DEBUG quinn_proto::endpoint: packet for unrecognized connection a697e5cd0d95f3e0
Mar 01 19:15:19.729 DEBUG quinn_proto::endpoint: sending stateless reset for a697e5cd0d95f3e0 to [::1]:55187
Mar 01 19:15:19.742 DEBUG quinn_proto::endpoint: sending stateless reset for 06ca02077e90c234 to [::1]:44363
Mar 01 19:15:19.747 DEBUG quinn_proto::endpoint: packet for unrecognized connection 2d06d61b66d1b6ba
Mar 01 19:15:19.747 DEBUG quinn_proto::endpoint: sending stateless reset for 2d06d61b66d1b6ba to [::1]:55187
Mar 01 19:15:19.751 DEBUG quinn_proto::endpoint: packet for unrecognized connection 1bfe10f3cf643722
Mar 01 19:15:19.751 DEBUG quinn_proto::endpoint: sending stateless reset for 1bfe10f3cf643722 to [::1]:55187
Mar 01 19:15:19.751 DEBUG quinn_proto::endpoint: sending stateless reset for 241aa274953a6ec7 to [::1]:44363
Mar 01 19:15:19.755 DEBUG quinn_proto::endpoint: packet for unrecognized connection feb2902de5dc2d92
Mar 01 19:15:19.755 DEBUG quinn_proto::endpoint: sending stateless reset for feb2902de5dc2d92 to [::1]:55187
Mar 01 19:15:19.756 DEBUG quinn_proto::endpoint: sending stateless reset for 052f3eb169501519 to [::1]:44363
Mar 01 19:15:19.756 DEBUG quinn_proto::endpoint: packet for unrecognized connection 9e858653930795c5
Mar 01 19:15:19.756 DEBUG quinn_proto::endpoint: sending stateless reset for 9e858653930795c5 to [::1]:55187
Mar 01 19:15:19.762 DEBUG quinn_proto::endpoint: sending stateless reset for c133c32d12f3beb8 to [::1]:44363
Mar 01 19:15:19.762 DEBUG quinn_proto::endpoint: packet for unrecognized connection c61bf0c4bb21db5b
Mar 01 19:15:19.762 DEBUG quinn_proto::endpoint: sending stateless reset for c61bf0c4bb21db5b to [::1]:55187
Mar 01 19:15:19.779 DEBUG quinn_proto::endpoint: sending stateless reset for dcc764afe973ec11 to [::1]:44363
Mar 01 19:15:19.779 DEBUG quinn_proto::endpoint: packet for unrecognized connection 171d61440a228d50
Mar 01 19:15:19.779 DEBUG quinn_proto::endpoint: sending stateless reset for 171d61440a228d50 to [::1]:55187
Mar 01 19:15:22.637 TRACE drive{id=6}: quinn_proto::connection: blocked by anti-amplification
Mar 01 19:15:22.637 TRACE drive{id=6}: quinn_proto::connection: timeout timer=LossDetection
Mar 01 19:15:22.637 TRACE drive{id=6}: quinn_proto::connection: PTO fired in_flight=2487 count=2 space=Initial
Mar 01 19:15:22.637 TRACE drive{id=6}: quinn_proto::connection: blocked by anti-amplification
Mar 01 19:15:26.636 TRACE drive{id=6}: quinn_proto::connection: blocked by anti-amplification
Mar 01 19:15:26.637 TRACE drive{id=6}: quinn_proto::connection: timeout timer=LossDetection
Mar 01 19:15:26.637 TRACE drive{id=6}: quinn_proto::connection: PTO fired in_flight=2487 count=3 space=Initial
Mar 01 19:15:26.637 TRACE drive{id=6}: quinn_proto::connection: blocked by anti-amplification
Mar 01 19:15:34.637 TRACE drive{id=6}: quinn_proto::connection: blocked by anti-amplification
Mar 01 19:15:34.637 TRACE drive{id=6}: quinn_proto::connection: timeout timer=LossDetection
Mar 01 19:15:34.637 TRACE drive{id=6}: quinn_proto::connection: PTO fired in_flight=2487 count=4 space=Initial
Mar 01 19:15:34.637 TRACE drive{id=6}: quinn_proto::connection: blocked by anti-amplification
Mar 01 19:15:50.636 TRACE drive{id=6}: quinn_proto::connection: blocked by anti-amplification
Mar 01 19:15:50.636 TRACE drive{id=6}: quinn_proto::connection: timeout timer=LossDetection
Mar 01 19:15:50.636 TRACE drive{id=6}: quinn_proto::connection: PTO fired in_flight=2487 count=5 space=Initial
Mar 01 19:15:50.636 TRACE drive{id=6}: quinn_proto::connection: blocked by anti-amplification
Mar 01 19:16:22.637 TRACE drive{id=6}: quinn_proto::connection: blocked by anti-amplification
Mar 01 19:16:22.637 TRACE drive{id=6}: quinn_proto::connection: timeout timer=LossDetection
Mar 01 19:16:22.637 TRACE drive{id=6}: quinn_proto::connection: PTO fired in_flight=2487 count=6 space=Initial
Mar 01 19:16:22.637 TRACE drive{id=6}: quinn_proto::connection: blocked by anti-amplification
Mar 01 19:17:26.636 TRACE drive{id=6}: quinn_proto::connection: blocked by anti-amplification
Mar 01 19:17:26.636 TRACE drive{id=6}: quinn_proto::connection: timeout timer=LossDetection
Mar 01 19:17:26.636 TRACE drive{id=6}: quinn_proto::connection: PTO fired in_flight=2487 count=7 space=Initial
Mar 01 19:17:26.636 TRACE drive{id=6}: quinn_proto::connection: blocked by anti-amplification
Mar 01 19:19:34.636 TRACE drive{id=6}: quinn_proto::connection: blocked by anti-amplification
Mar 01 19:19:34.637 TRACE drive{id=6}: quinn_proto::connection: timeout timer=LossDetection
Mar 01 19:19:34.637 TRACE drive{id=6}: quinn_proto::connection: PTO fired in_flight=2487 count=8 space=Initial
Mar 01 19:19:34.637 TRACE drive{id=6}: quinn_proto::connection: blocked by anti-amplification
Mar 01 19:23:50.636 TRACE drive{id=6}: quinn_proto::connection: blocked by anti-amplification
Mar 01 19:23:50.636 TRACE drive{id=6}: quinn_proto::connection: timeout timer=LossDetection
Mar 01 19:23:50.636 TRACE drive{id=6}: quinn_proto::connection: PTO fired in_flight=2487 count=9 space=Initial
Mar 01 19:23:50.636 TRACE drive{id=6}: quinn_proto::connection: blocked by anti-amplification

This is expected behavior

This makes sense, thanks for clarifying.

@alecmocatta
Copy link
Contributor Author

I've just re-run it with some of quinn's anti-amplification logic (which shows up in the above trace) disabled. Unfortunately it did not resolve it, but here's the differing trace in case it's useful:

Show trace 
...
6806 sender open
Mar 02 13:13:45.646 TRACE quinn_proto::endpoint: initial_dcid=1f57b40cac72e3c8a3d1019309c0cefc80be480b
Mar 02 13:13:45.646 TRACE quinn_proto::connection: wrote 344 Initial TLS bytes
Mar 02 13:13:45.646 TRACE quinn_proto::connection: 0-RTT enabled
6806 receiver open
Mar 02 13:13:45.647 TRACE first recv: quinn_proto::connection: authenticated
Mar 02 13:13:45.647 TRACE first recv:frame{ty=CRYPTO}: quinn_proto::connection: read 344 TLS bytes
Mar 02 13:13:45.647 TRACE first recv: quinn_proto::connection: Handshake keys ready
Mar 02 13:13:45.647 TRACE first recv: quinn_proto::connection: wrote 90 Initial TLS bytes
Mar 02 13:13:45.647 TRACE first recv: quinn_proto::connection: Data keys ready
Mar 02 13:13:45.647 TRACE first recv: quinn_proto::connection: wrote 564 Handshake TLS bytes
Mar 02 13:13:45.647 TRACE quinn_proto::endpoint: connection incoming id=4 icid=8cf9c3eb8df0b22676195ec431b2cab06c42cd0d
Mar 02 13:13:45.647 DEBUG quinn_proto::endpoint: sending stateless reset for 75a31ffaf173a7f6 to [::1]:47079
Mar 02 13:13:45.647 TRACE drive{id=4}:send{space=Data pn=11}: quinn_proto::connection: sending CONNECTION_CLOSE
Mar 02 13:13:45.647 TRACE drive{id=4}: quinn_proto::connection: sending 29 byte datagram
Mar 02 13:13:45.647 TRACE drive{id=5}:send{space=Initial pn=0}: quinn_proto::connection: CRYPTO: off 0 len 344
Mar 02 13:13:45.647 TRACE drive{id=5}: quinn_proto::connection: sending 1200 byte datagram
Mar 02 13:13:45.647 TRACE drive{id=4}:send{space=Initial pn=0}: quinn_proto::connection: ACK
Mar 02 13:13:45.647 TRACE drive{id=4}:send{space=Initial pn=0}: quinn_proto::connection: CRYPTO: off 0 len 90
Mar 02 13:13:45.647 TRACE drive{id=4}:send{space=Handshake pn=0}: quinn_proto::connection: CRYPTO: off 0 len 564
Mar 02 13:13:45.648 TRACE drive{id=4}: quinn_proto::connection: sending 755 byte datagram
Mar 02 13:13:45.648 DEBUG drive{id=7}: quinn_proto::connection: discarding unexpected Handshake packet (89 bytes)
Mar 02 13:13:45.648 TRACE drive{id=7}: quinn_proto::connection: got Data packet (36 bytes) from [::1]:47079 using id c652e63814c4502e
Mar 02 13:13:45.648 TRACE drive{id=7}:send{space=Data pn=8}: quinn_proto::connection: sending CONNECTION_CLOSE
Mar 02 13:13:45.648 TRACE drive{id=7}: quinn_proto::connection: sending 29 byte datagram
Mar 02 13:13:45.648 TRACE drive{id=4}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=7 id=2fc45a95b45c99a7
Mar 02 13:13:45.648 TRACE drive{id=4}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=6 id=cae84738e3590fa6
Mar 02 13:13:45.648 TRACE drive{id=4}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=5 id=dee6b91b1afde943
Mar 02 13:13:45.648 TRACE drive{id=4}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=4 id=19ec74500c4e4ef4
Mar 02 13:13:45.648 TRACE drive{id=4}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=3 id=ec3d665714ff0806
Mar 02 13:13:45.648 TRACE drive{id=4}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=2 id=52ea4b390a0e6395
Mar 02 13:13:45.648 TRACE drive{id=4}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=1 id=871521fdf7ba5abe
Mar 02 13:13:45.648 TRACE drive{id=4}: quinn_proto::connection: sending 222 byte datagram
6806 receiver open a
Mar 02 13:13:45.648 TRACE drive{id=7}: quinn_proto::connection: got Data packet (29 bytes) from [::1]:47079 using id c652e63814c4502e
Mar 02 13:13:45.648 TRACE drive{id=7}:recv{space=Data pn=11}: quinn_proto::connection: draining
Mar 02 13:13:45.648 TRACE drive{id=3}: quinn_proto::connection: got Data packet (222 bytes) from [::1]:48273 using id 654a46bae0b51cd9
Mar 02 13:13:45.648 TRACE drive{id=3}: quinn_proto::connection: decryption failed with packet number 8
Mar 02 13:13:45.648 DEBUG drive{id=3}: quinn_proto::connection: failed to authenticate packet
Mar 02 13:13:45.649 TRACE first recv: quinn_proto::connection: authenticated
Mar 02 13:13:45.649 TRACE first recv:frame{ty=CRYPTO}: quinn_proto::connection: read 344 TLS bytes
Mar 02 13:13:45.649 TRACE first recv: quinn_proto::connection: Handshake keys ready
Mar 02 13:13:45.649 TRACE first recv: quinn_proto::connection: wrote 96 Initial TLS bytes
Mar 02 13:13:45.649 TRACE first recv: quinn_proto::connection: Data keys ready
Mar 02 13:13:45.649 TRACE first recv: quinn_proto::connection: wrote 136 Handshake TLS bytes
Mar 02 13:13:45.649 TRACE first recv: quinn_proto::connection: 0-RTT enabled
Mar 02 13:13:45.649 TRACE quinn_proto::endpoint: connection incoming id=3 icid=1f57b40cac72e3c8a3d1019309c0cefc80be480b
Mar 02 13:13:45.649 TRACE drive{id=4}: quinn_proto::connection: got Data packet (36 bytes) from [::1]:48273 using id 583782527b0edcb6
Mar 02 13:13:45.649 TRACE drive{id=4}:send{space=Data pn=12}: quinn_proto::connection: sending CONNECTION_CLOSE
Mar 02 13:13:45.649 TRACE drive{id=4}: quinn_proto::connection: sending 29 byte datagram
Mar 02 13:13:45.649 TRACE drive{id=3}:send{space=Initial pn=0}: quinn_proto::connection: ACK
Mar 02 13:13:45.649 TRACE drive{id=3}:send{space=Initial pn=0}: quinn_proto::connection: CRYPTO: off 0 len 96
Mar 02 13:13:45.649 TRACE drive{id=3}:send{space=Handshake pn=0}: quinn_proto::connection: CRYPTO: off 0 len 136
Mar 02 13:13:45.649 TRACE drive{id=3}: quinn_proto::connection: sending 333 byte datagram
Mar 02 13:13:45.649 TRACE drive{id=3}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=7 id=5dcd1521ab0c27bd
Mar 02 13:13:45.649 TRACE drive{id=3}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=6 id=83fadd89f23ab002
Mar 02 13:13:45.649 TRACE drive{id=3}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=5 id=06ba6a39dcbbac4c
Mar 02 13:13:45.649 TRACE drive{id=3}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=4 id=006472d2dd47b6d7
Mar 02 13:13:45.649 TRACE drive{id=3}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=3 id=0bb14ee668f1f91c
Mar 02 13:13:45.650 TRACE drive{id=3}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=2 id=2479171f139e3623
Mar 02 13:13:45.650 TRACE drive{id=3}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=1 id=3af61ed9c09f9cec
Mar 02 13:13:45.650 TRACE drive{id=3}: quinn_proto::connection: sending 222 byte datagram
Mar 02 13:13:45.650 TRACE drive{id=4}: quinn_proto::connection: got Data packet (29 bytes) from [::1]:48273 using id 583782527b0edcb6
Mar 02 13:13:45.650 TRACE drive{id=4}:recv{space=Data pn=7}: quinn_proto::connection: draining
Mar 02 13:13:45.650 TRACE drive{id=4}: quinn_proto::connection: got Data packet (47 bytes) from [::1]:48273 using id 583782527b0edcb6
Mar 02 13:13:45.650 TRACE drive{id=3}: quinn_proto::connection: got Data packet (41 bytes) from [::1]:48273 using id 654a46bae0b51cd9
Mar 02 13:13:45.650 TRACE drive{id=5}: quinn_proto::connection: got Initial packet (151 bytes) from [::1]:48273 using id e4b26d8e7706bd6d
Mar 02 13:13:45.650 TRACE drive{id=5}:recv{space=Initial pn=0}: quinn_proto::connection: authenticated
Mar 02 13:13:45.650 TRACE drive{id=5}:recv{space=Initial pn=0}: quinn_proto::connection: switching remote CID to 691912417654046b
Mar 02 13:13:45.650 TRACE drive{id=5}:recv{space=Initial pn=0}:frame{ty=CRYPTO}: quinn_proto::connection: read 96 TLS bytes
Mar 02 13:13:45.650 TRACE drive{id=5}:recv{space=Initial pn=0}: quinn_proto::connection: Handshake keys ready
Mar 02 13:13:45.650 TRACE drive{id=5}: quinn_proto::connection: got Handshake packet (182 bytes) from [::1]:48273 using id e4b26d8e7706bd6d
Mar 02 13:13:45.650 TRACE drive{id=5}:recv{space=Handshake pn=0}: quinn_proto::connection: authenticated
Mar 02 13:13:45.650 TRACE drive{id=5}:recv{space=Handshake pn=0}:frame{ty=CRYPTO}: quinn_proto::connection: read 136 TLS bytes
Mar 02 13:13:45.650 TRACE drive{id=5}:recv{space=Handshake pn=0}: quinn_proto::connection: Data keys ready
Mar 02 13:13:45.650 TRACE drive{id=5}:recv{space=Handshake pn=0}: quinn_proto::connection: wrote 36 Handshake TLS bytes
Mar 02 13:13:45.650 TRACE drive{id=5}:recv{space=Handshake pn=0}: quinn_proto::connection: established
Mar 02 13:13:45.650 TRACE drive{id=5}:send{space=Initial pn=1}: quinn_proto::connection: ACK
Mar 02 13:13:45.650 TRACE drive{id=5}: quinn_proto::connection: discarding Initial keys
Mar 02 13:13:45.650 TRACE drive{id=5}:send{space=Handshake pn=0}: quinn_proto::connection: ACK
Mar 02 13:13:45.650 TRACE drive{id=5}:send{space=Handshake pn=0}: quinn_proto::connection: CRYPTO: off 0 len 36
Mar 02 13:13:45.650 TRACE drive{id=5}: quinn_proto::connection: sending 1200 byte datagram
Mar 02 13:13:45.650 TRACE drive{id=7}: quinn_proto::connection: got Data packet (31 bytes) from [::1]:47079 using id c652e63814c4502e
Mar 02 13:13:45.650 TRACE drive{id=5}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=7 id=4765fd16d158cfe6
Mar 02 13:13:45.650 TRACE drive{id=5}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=6 id=826441022294cebf
Mar 02 13:13:45.651 TRACE drive{id=5}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=5 id=8d05d27e2cc32430
Mar 02 13:13:45.651 TRACE drive{id=5}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=4 id=fdbee680a10be57f
Mar 02 13:13:45.651 TRACE drive{id=5}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=3 id=72be82502320e32c
Mar 02 13:13:45.651 TRACE drive{id=5}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=2 id=c4f005b35814e81e
Mar 02 13:13:45.651 TRACE drive{id=5}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=1 id=5cb61128923e4113
Mar 02 13:13:45.651 TRACE drive{id=5}: quinn_proto::connection: sending 222 byte datagram
Mar 02 13:13:45.651 TRACE quinn_proto::connection: wrote 2 bytes stream=client unidirectional stream 0
6806 sender /open
Mar 02 13:13:45.651 TRACE drive{id=5}:send{space=Data pn=1}: quinn_proto::connection: STREAM id=client unidirectional stream 0 off=0 len=2 fin=false
Mar 02 13:13:45.651 TRACE drive{id=5}: quinn_proto::connection: sending 31 byte datagram
Mar 02 13:13:45.651 TRACE drive{id=3}: quinn_proto::connection: got Initial packet (51 bytes) from [::1]:47079 using id 691912417654046b
Mar 02 13:13:45.651 TRACE drive{id=3}:recv{space=Initial pn=1}: quinn_proto::connection: authenticated
Mar 02 13:13:45.651 TRACE drive{id=3}: quinn_proto::connection: got Handshake packet (1149 bytes) from [::1]:47079 using id 691912417654046b
Mar 02 13:13:45.651 TRACE drive{id=3}:recv{space=Handshake pn=0}: quinn_proto::connection: authenticated
Mar 02 13:13:45.651 TRACE drive{id=3}:recv{space=Handshake pn=0}: quinn_proto::connection: discarding Initial keys
Mar 02 13:13:45.651 TRACE drive{id=3}:recv{space=Handshake pn=0}:frame{ty=CRYPTO}: quinn_proto::connection: read 36 TLS bytes
Mar 02 13:13:45.652 TRACE drive{id=3}:recv{space=Handshake pn=0}: quinn_proto::connection: wrote 89 Data TLS bytes
Mar 02 13:13:45.652 TRACE drive{id=3}:recv{space=Handshake pn=0}: quinn_proto::connection: established
Mar 02 13:13:45.652 TRACE drive{id=3}: quinn_proto::connection: got Data packet (222 bytes) from [::1]:47079 using id 691912417654046b
Mar 02 13:13:45.652 TRACE drive{id=3}:recv{space=Data pn=0}: quinn_proto::connection: authenticated
Mar 02 13:13:45.652 TRACE drive{id=3}:recv{space=Data pn=0}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=7 id=4765fd16d158cfe6
Mar 02 13:13:45.652 TRACE drive{id=3}:recv{space=Data pn=0}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: switching to remote CID 7: 4765fd16d158cfe6
Mar 02 13:13:45.652 TRACE drive{id=3}:recv{space=Data pn=0}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=6 id=826441022294cebf
Mar 02 13:13:45.652 TRACE drive{id=3}:recv{space=Data pn=0}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=5 id=8d05d27e2cc32430
Mar 02 13:13:45.652 TRACE drive{id=3}:recv{space=Data pn=0}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=4 id=fdbee680a10be57f
Mar 02 13:13:45.652 TRACE drive{id=3}:recv{space=Data pn=0}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=3 id=72be82502320e32c
Mar 02 13:13:45.652 TRACE drive{id=3}:recv{space=Data pn=0}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=2 id=c4f005b35814e81e
Mar 02 13:13:45.652 TRACE drive{id=3}:recv{space=Data pn=0}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=1 id=5cb61128923e4113
Mar 02 13:13:45.652 TRACE drive{id=3}:send{space=Handshake pn=1}: quinn_proto::connection: ACK
Mar 02 13:13:45.652 TRACE drive{id=3}:send{space=Data pn=1}: quinn_proto::connection: ACK
Mar 02 13:13:45.652 TRACE drive{id=3}:send{space=Data pn=1}: quinn_proto::connection: CRYPTO: off 0 len 89
Mar 02 13:13:45.652 TRACE drive{id=3}:send{space=Data pn=1}: quinn_proto::connection: RETIRE_CONNECTION_ID sequence=0
Mar 02 13:13:45.652 TRACE drive{id=3}: quinn_proto::connection: sending 179 byte datagram
Mar 02 13:13:45.652 DEBUG drive{id=3}: quinn_proto::connection: discarding unexpected Initial packet (145 bytes)
Mar 02 13:13:45.652 DEBUG drive{id=3}: quinn_proto::connection: discarding unexpected Handshake packet (610 bytes)
Mar 02 13:13:45.652 TRACE drive{id=4}: quinn_proto::connection: got Data packet (29 bytes) from [::1]:48273 using id 583782527b0edcb6
Mar 02 13:13:45.652 TRACE drive{id=4}: quinn_proto::connection: timeout timer=Close
Mar 02 13:13:45.652 TRACE drive{id=3}: quinn_proto::connection: timeout timer=Close
Mar 02 13:13:45.653 DEBUG quinn_proto::endpoint: sending stateless reset for 75a31ffaf173a7f6 to [::1]:47079
Mar 02 13:13:45.653 DEBUG quinn_proto::endpoint: packet for unrecognized connection 583782527b0edcb6
Mar 02 13:13:45.653 DEBUG quinn_proto::endpoint: sending stateless reset for 583782527b0edcb6 to [::1]:48273
Mar 02 13:13:45.654 TRACE drive{id=7}: quinn_proto::connection: got Data packet (230 bytes) from [::1]:47079 using id c652e63814c4502e
Mar 02 13:13:45.654 TRACE drive{id=3}: quinn_proto::connection: got Data packet (31 bytes) from [::1]:47079 using id 691912417654046b
Mar 02 13:13:45.654 TRACE drive{id=3}:recv{space=Data pn=1}: quinn_proto::connection: authenticated
Mar 02 13:13:45.654 TRACE drive{id=3}:recv{space=Data pn=1}:frame{ty=STREAM}: quinn_proto::connection: got stream id=client unidirectional stream 0 offset=0 len=2 fin=false
Mar 02 13:13:45.654 TRACE drive{id=3}:send{space=Data pn=2}: quinn_proto::connection: ACK
Mar 02 13:13:45.654 TRACE drive{id=3}: quinn_proto::connection: sending 34 byte datagram
Mar 02 13:13:45.654 TRACE drive{id=3}: quinn_proto::connection: timeout timer=LossDetection
Mar 02 13:13:45.654 TRACE drive{id=3}: quinn_proto::connection: PTO fired in_flight=351 count=0 space=Handshake
Mar 02 13:13:45.654 TRACE drive{id=3}:send{space=Handshake pn=2}: quinn_proto::connection: PING
Mar 02 13:13:45.654 TRACE drive{id=3}:send{space=Handshake pn=2}: quinn_proto::connection: ACK
Mar 02 13:13:45.654 TRACE drive{id=3}: quinn_proto::connection: sending 51 byte datagram
Mar 02 13:13:45.654 TRACE drive{id=3}:send{space=Handshake pn=3}: quinn_proto::connection: PING
Mar 02 13:13:45.654 TRACE drive{id=3}:send{space=Handshake pn=3}: quinn_proto::connection: ACK
Mar 02 13:13:45.654 TRACE drive{id=3}: quinn_proto::connection: sending 51 byte datagram
Mar 02 13:13:45.654 DEBUG quinn_proto::endpoint: packet for unrecognized connection 583782527b0edcb6
Mar 02 13:13:45.655 DEBUG quinn_proto::endpoint: sending stateless reset for 583782527b0edcb6 to [::1]:48273
Mar 02 13:13:45.655 TRACE drive{id=5}: quinn_proto::connection: got Handshake packet (51 bytes) from [::1]:48273 using id 4765fd16d158cfe6
Mar 02 13:13:45.655 TRACE drive{id=5}:recv{space=Handshake pn=2}: quinn_proto::connection: authenticated
Mar 02 13:13:45.655 TRACE drive{id=5}: quinn_proto::connection: got Handshake packet (51 bytes) from [::1]:48273 using id 4765fd16d158cfe6
Mar 02 13:13:45.655 TRACE drive{id=5}:recv{space=Handshake pn=3}: quinn_proto::connection: authenticated
Mar 02 13:13:45.655 TRACE drive{id=5}:send{space=Handshake pn=1}: quinn_proto::connection: ACK
Mar 02 13:13:45.655 TRACE drive{id=5}: quinn_proto::connection: sending 50 byte datagram
Mar 02 13:13:45.655 TRACE drive{id=3}: quinn_proto::connection: got Handshake packet (50 bytes) from [::1]:47079 using id 691912417654046b
Mar 02 13:13:45.655 TRACE drive{id=3}:recv{space=Handshake pn=1}: quinn_proto::connection: authenticated
Mar 02 13:13:45.655 TRACE drive{id=3}:recv{space=Handshake pn=1}:frame{ty=ACK}: quinn_proto::connection: packets lost: [1]
Mar 02 13:13:45.656 TRACE drive{id=7}: quinn_proto::connection: got Data packet (39 bytes) from [::1]:47079 using id c652e63814c4502e
Mar 02 13:13:45.657 TRACE drive{id=7}: quinn_proto::connection: got Data packet (29 bytes) from [::1]:47079 using id c652e63814c4502e
Mar 02 13:13:45.657 DEBUG quinn_proto::endpoint: packet for unrecognized connection 1a1a45284242d431
Mar 02 13:13:45.657 DEBUG quinn_proto::endpoint: sending stateless reset for 1a1a45284242d431 to [::1]:48273
Mar 02 13:13:45.657 TRACE drive{id=3}: quinn_proto::connection: timeout timer=LossDetection
Mar 02 13:13:45.657 TRACE drive{id=3}: quinn_proto::connection: PTO fired in_flight=351 count=0 space=Data
Mar 02 13:13:45.657 TRACE drive{id=3}:send{space=Data pn=3}: quinn_proto::connection: ACK
Mar 02 13:13:45.657 TRACE drive{id=3}:send{space=Data pn=3}: quinn_proto::connection: NEW_CONNECTION_ID sequence=1 id=3af61ed9c09f9cec
Mar 02 13:13:45.657 TRACE drive{id=3}:send{space=Data pn=3}: quinn_proto::connection: NEW_CONNECTION_ID sequence=2 id=2479171f139e3623
Mar 02 13:13:45.657 TRACE drive{id=3}:send{space=Data pn=3}: quinn_proto::connection: NEW_CONNECTION_ID sequence=3 id=0bb14ee668f1f91c
Mar 02 13:13:45.657 TRACE drive{id=3}:send{space=Data pn=3}: quinn_proto::connection: NEW_CONNECTION_ID sequence=4 id=006472d2dd47b6d7
Mar 02 13:13:45.657 TRACE drive{id=3}:send{space=Data pn=3}: quinn_proto::connection: NEW_CONNECTION_ID sequence=5 id=06ba6a39dcbbac4c
Mar 02 13:13:45.658 TRACE drive{id=3}:send{space=Data pn=3}: quinn_proto::connection: NEW_CONNECTION_ID sequence=6 id=83fadd89f23ab002
Mar 02 13:13:45.658 TRACE drive{id=3}:send{space=Data pn=3}: quinn_proto::connection: NEW_CONNECTION_ID sequence=7 id=5dcd1521ab0c27bd
Mar 02 13:13:45.658 TRACE drive{id=3}: quinn_proto::connection: sending 230 byte datagram
Mar 02 13:13:45.658 TRACE drive{id=3}:send{space=Data pn=4}: quinn_proto::connection: ACK
Mar 02 13:13:45.658 TRACE drive{id=3}:send{space=Data pn=4}: quinn_proto::connection: CRYPTO: off 0 len 89
Mar 02 13:13:45.658 TRACE drive{id=3}:send{space=Data pn=4}: quinn_proto::connection: RETIRE_CONNECTION_ID sequence=0
Mar 02 13:13:45.658 TRACE drive{id=3}: quinn_proto::connection: sending 129 byte datagram
Mar 02 13:13:45.658 TRACE drive{id=5}: quinn_proto::connection: got Handshake packet (50 bytes) from [::1]:48273 using id 4765fd16d158cfe6
Mar 02 13:13:45.658 TRACE drive{id=5}:recv{space=Handshake pn=1}: quinn_proto::connection: authenticated
Mar 02 13:13:45.658 TRACE drive{id=5}: quinn_proto::connection: got Data packet (129 bytes) from [::1]:48273 using id 4765fd16d158cfe6
Mar 02 13:13:45.658 TRACE drive{id=5}:recv{space=Data pn=1}: quinn_proto::connection: authenticated
Mar 02 13:13:45.658 TRACE drive{id=5}:recv{space=Data pn=1}:frame{ty=ACK}: quinn_proto::connection: discarding Handshake keys
Mar 02 13:13:45.658 TRACE drive{id=5}:recv{space=Data pn=1}:frame{ty=CRYPTO}: quinn_proto::connection: read 89 TLS bytes
Mar 02 13:13:45.658 TRACE drive{id=5}:send{space=Data pn=2}: quinn_proto::connection: ACK
Mar 02 13:13:45.658 TRACE drive{id=5}: quinn_proto::connection: sending 34 byte datagram
Mar 02 13:13:45.658 TRACE quinn_proto::endpoint: peer retired CID 0: e4b26d8e7706bd6d
Mar 02 13:13:45.658 TRACE drive{id=5}:send{space=Data pn=3}: quinn_proto::connection: ACK
Mar 02 13:13:45.658 TRACE drive{id=5}:send{space=Data pn=3}: quinn_proto::connection: NEW_CONNECTION_ID sequence=8 id=b2e93e3e2f0c1f4e
Mar 02 13:13:45.658 TRACE drive{id=5}: quinn_proto::connection: sending 62 byte datagram
Mar 02 13:13:45.659 DEBUG drive{id=7}: quinn_proto::connection: discarding unexpected Initial packet (51 bytes)
Mar 02 13:13:45.659 DEBUG drive{id=7}: quinn_proto::connection: discarding unexpected Handshake packet (1149 bytes)
Mar 02 13:13:45.659 TRACE drive{id=3}: quinn_proto::connection: got Data packet (34 bytes) from [::1]:47079 using id 691912417654046b
Mar 02 13:13:45.659 TRACE drive{id=3}:recv{space=Data pn=2}: quinn_proto::connection: authenticated
Mar 02 13:13:45.659 TRACE drive{id=3}:recv{space=Data pn=2}:frame{ty=ACK}: quinn_proto::connection: discarding Handshake keys
Mar 02 13:13:45.659 TRACE drive{id=3}:recv{space=Data pn=2}:frame{ty=ACK}: quinn_proto::connection: packets lost: [0]
Mar 02 13:13:45.659 TRACE drive{id=5}: quinn_proto::connection: got Data packet (230 bytes) from [::1]:48273 using id 4765fd16d158cfe6
Mar 02 13:13:45.659 TRACE drive{id=5}:recv{space=Data pn=3}: quinn_proto::connection: authenticated
Mar 02 13:13:45.660 TRACE drive{id=5}:recv{space=Data pn=3}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=1 id=3af61ed9c09f9cec
Mar 02 13:13:45.660 TRACE drive{id=5}:recv{space=Data pn=3}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=2 id=2479171f139e3623
Mar 02 13:13:45.660 TRACE drive{id=5}:recv{space=Data pn=3}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=3 id=0bb14ee668f1f91c
Mar 02 13:13:45.660 TRACE drive{id=5}:recv{space=Data pn=3}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=4 id=006472d2dd47b6d7
Mar 02 13:13:45.660 TRACE drive{id=5}:recv{space=Data pn=3}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=5 id=06ba6a39dcbbac4c
Mar 02 13:13:45.660 TRACE drive{id=5}:recv{space=Data pn=3}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=6 id=83fadd89f23ab002
Mar 02 13:13:45.660 TRACE drive{id=5}:recv{space=Data pn=3}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=7 id=5dcd1521ab0c27bd
Mar 02 13:13:45.660 TRACE drive{id=5}:send{space=Data pn=4}: quinn_proto::connection: ACK
Mar 02 13:13:45.660 TRACE drive{id=5}: quinn_proto::connection: sending 36 byte datagram
Mar 02 13:13:45.660 DEBUG quinn_proto::endpoint: packet for unrecognized connection abcea92aae5e0d17
Mar 02 13:13:45.660 DEBUG quinn_proto::endpoint: sending stateless reset for abcea92aae5e0d17 to [::1]:48273
Mar 02 13:13:45.660 DEBUG quinn_proto::endpoint: packet for unrecognized connection e4b26d8e7706bd6d
Mar 02 13:13:45.660 DEBUG quinn_proto::endpoint: sending stateless reset for e4b26d8e7706bd6d to [::1]:48273
Mar 02 13:13:45.660 TRACE drive{id=3}: quinn_proto::connection: got Data packet (36 bytes) from [::1]:47079 using id 691912417654046b
Mar 02 13:13:45.660 TRACE drive{id=3}:recv{space=Data pn=4}: quinn_proto::connection: authenticated
Mar 02 13:13:45.660 TRACE drive{id=3}:recv{space=Data pn=4}:frame{ty=ACK}: quinn_proto::connection: packets lost: [2]
Mar 02 13:13:45.662 TRACE drive{id=3}: quinn_proto::connection: got Data packet (62 bytes) from [::1]:47079 using id 691912417654046b
Mar 02 13:13:45.662 TRACE drive{id=3}:recv{space=Data pn=3}: quinn_proto::connection: authenticated
Mar 02 13:13:45.662 TRACE drive{id=3}:recv{space=Data pn=3}:frame{ty=NEW_CONNECTION_ID}: quinn_proto::connection: sequence=8 id=b2e93e3e2f0c1f4e
Mar 02 13:13:45.662 TRACE drive{id=3}:send{space=Data pn=5}: quinn_proto::connection: ACK
Mar 02 13:13:45.662 TRACE drive{id=3}: quinn_proto::connection: sending 34 byte datagram
Mar 02 13:13:45.662 TRACE drive{id=5}: quinn_proto::connection: got Data packet (34 bytes) from [::1]:48273 using id 4765fd16d158cfe6
Mar 02 13:13:45.662 TRACE drive{id=5}:recv{space=Data pn=5}: quinn_proto::connection: authenticated
Mar 02 13:13:45.667 DEBUG quinn_proto::endpoint: sending stateless reset for 0f8286980d26ad8b to [::1]:47079
Mar 02 13:13:45.668 TRACE drive{id=3}: quinn_proto::connection: timeout timer=LossDetection
Mar 02 13:13:45.668 TRACE drive{id=3}: quinn_proto::connection: PTO fired in_flight=129 count=0 space=Data
Mar 02 13:13:45.668 TRACE drive{id=3}:send{space=Data pn=6}: quinn_proto::connection: ACK
Mar 02 13:13:45.668 TRACE drive{id=3}:send{space=Data pn=6}: quinn_proto::connection: CRYPTO: off 0 len 89
Mar 02 13:13:45.668 TRACE drive{id=3}:send{space=Data pn=6}: quinn_proto::connection: RETIRE_CONNECTION_ID sequence=0
Mar 02 13:13:45.668 TRACE drive{id=3}: quinn_proto::connection: sending 129 byte datagram
Mar 02 13:13:45.668 TRACE drive{id=3}:send{space=Data pn=7}: quinn_proto::connection: ACK
Mar 02 13:13:45.668 TRACE drive{id=3}:send{space=Data pn=7}: quinn_proto::connection: CRYPTO: off 0 len 89
Mar 02 13:13:45.668 TRACE drive{id=3}:send{space=Data pn=7}: quinn_proto::connection: RETIRE_CONNECTION_ID sequence=0
Mar 02 13:13:45.668 TRACE drive{id=3}: quinn_proto::connection: sending 129 byte datagram
Mar 02 13:13:45.668 TRACE drive{id=7}: quinn_proto::connection: timeout timer=Close
Mar 02 13:13:45.668 DEBUG quinn_proto::endpoint: packet for unrecognized connection b11ed1a82369fac9
Mar 02 13:13:45.668 DEBUG quinn_proto::endpoint: sending stateless reset for b11ed1a82369fac9 to [::1]:48273
Mar 02 13:13:45.668 TRACE drive{id=5}: quinn_proto::connection: got Data packet (129 bytes) from [::1]:48273 using id 4765fd16d158cfe6
Mar 02 13:13:45.668 TRACE drive{id=5}:recv{space=Data pn=7}: quinn_proto::connection: authenticated
Mar 02 13:13:45.668 TRACE drive{id=5}: quinn_proto::connection: got Data packet (129 bytes) from [::1]:48273 using id 4765fd16d158cfe6
Mar 02 13:13:45.668 TRACE drive{id=5}:recv{space=Data pn=4}: quinn_proto::connection: authenticated
Mar 02 13:13:45.669 TRACE drive{id=5}:send{space=Data pn=5}: quinn_proto::connection: ACK
Mar 02 13:13:45.669 TRACE drive{id=5}: quinn_proto::connection: sending 36 byte datagram
Mar 02 13:13:45.669 DEBUG quinn_proto::endpoint: sending stateless reset for b268b13a707c8eae to [::1]:47079
Mar 02 13:13:45.669 DEBUG quinn_proto::endpoint: sending stateless reset for b268b13a707c8eae to [::1]:47079
Mar 02 13:13:45.669 DEBUG quinn_proto::endpoint: sending stateless reset for d1f187c4a3929efa to [::1]:47079
Mar 02 13:13:45.669 DEBUG quinn_proto::endpoint: sending stateless reset for 7c0728d20cabdad1 to [::1]:47079
Mar 02 13:13:45.669 DEBUG quinn_proto::endpoint: sending stateless reset for c8d94d5dec26845d to [::1]:47079
Mar 02 13:13:45.669 TRACE drive{id=3}: quinn_proto::connection: got Data packet (36 bytes) from [::1]:47079 using id 691912417654046b
Mar 02 13:13:45.669 TRACE drive{id=3}:recv{space=Data pn=5}: quinn_proto::connection: authenticated
Mar 02 13:13:45.669 DEBUG quinn_proto::endpoint: packet for unrecognized connection 64b492e380b9fd49
Mar 02 13:13:45.669 DEBUG quinn_proto::endpoint: sending stateless reset for 64b492e380b9fd49 to [::1]:48273
Mar 02 13:13:45.669 DEBUG quinn_proto::endpoint: packet for unrecognized connection 1add77cc48121582
Mar 02 13:13:45.669 DEBUG quinn_proto::endpoint: sending stateless reset for 1add77cc48121582 to [::1]:48273
Mar 02 13:13:45.669 TRACE drive{id=5}: quinn_proto::connection: got Data packet (34 bytes) from [::1]:48273 using id 4765fd16d158cfe6
Mar 02 13:13:45.669 TRACE drive{id=5}:recv{space=Data pn=2}: quinn_proto::connection: authenticated
Mar 02 13:13:45.671 DEBUG quinn_proto::endpoint: packet for unrecognized connection 9c32d2ef9e052018
Mar 02 13:13:45.671 DEBUG quinn_proto::endpoint: sending stateless reset for 9c32d2ef9e052018 to [::1]:48273
Mar 02 13:13:45.671 TRACE drive{id=3}: quinn_proto::connection: timeout timer=LossDetection
Mar 02 13:13:45.671 TRACE drive{id=3}: quinn_proto::connection: packets lost: [6]
Mar 02 13:13:45.677 DEBUG quinn_proto::endpoint: sending stateless reset for b43ccfa7a89a0dda to [::1]:47079
Mar 02 13:13:45.677 DEBUG quinn_proto::endpoint: sending stateless reset for 7a4171be2fd4928e to [::1]:47079
Mar 02 13:13:45.677 DEBUG quinn_proto::endpoint: packet for unrecognized connection dc17e26c1282832d
Mar 02 13:13:45.677 DEBUG quinn_proto::endpoint: sending stateless reset for dc17e26c1282832d to [::1]:48273
Mar 02 13:13:45.677 DEBUG quinn_proto::endpoint: sending stateless reset for 6d2d2c750be02dc3 to [::1]:47079
Mar 02 13:13:45.679 TRACE drive{id=5}: quinn_proto::connection: got Data packet (129 bytes) from [::1]:48273 using id 4765fd16d158cfe6
Mar 02 13:13:45.679 TRACE drive{id=5}:recv{space=Data pn=6}: quinn_proto::connection: authenticated
Mar 02 13:13:45.679 TRACE drive{id=5}:send{space=Data pn=6}: quinn_proto::connection: ACK
Mar 02 13:13:45.679 TRACE drive{id=5}: quinn_proto::connection: sending 36 byte datagram
Mar 02 13:13:45.680 DEBUG quinn_proto::endpoint: sending stateless reset for 751e01f97c3e2c93 to [::1]:47079
Mar 02 13:13:45.682 DEBUG quinn_proto::endpoint: packet for unrecognized connection adb55c3d13c15d53
Mar 02 13:13:45.682 DEBUG quinn_proto::endpoint: sending stateless reset for adb55c3d13c15d53 to [::1]:48273
Mar 02 13:13:45.682 DEBUG quinn_proto::endpoint: packet for unrecognized connection 2465e96e6ef26819
Mar 02 13:13:45.682 DEBUG quinn_proto::endpoint: sending stateless reset for 2465e96e6ef26819 to [::1]:48273
Mar 02 13:13:45.683 DEBUG quinn_proto::endpoint: sending stateless reset for 8f299852a33e8261 to [::1]:47079
Mar 02 13:13:45.686 DEBUG quinn_proto::endpoint: sending stateless reset for 72f81ef8910585f7 to [::1]:47079
Mar 02 13:13:45.686 DEBUG quinn_proto::endpoint: packet for unrecognized connection c94e85d32f47dfdd
Mar 02 13:13:45.687 DEBUG quinn_proto::endpoint: sending stateless reset for c94e85d32f47dfdd to [::1]:48273
Mar 02 13:13:45.687 DEBUG quinn_proto::endpoint: packet for unrecognized connection cec94a7e7cecbaf9
Mar 02 13:13:45.687 DEBUG quinn_proto::endpoint: sending stateless reset for cec94a7e7cecbaf9 to [::1]:48273
Mar 02 13:13:45.688 DEBUG quinn_proto::endpoint: packet for unrecognized connection 57ca4332d73bd860
Mar 02 13:13:45.688 DEBUG quinn_proto::endpoint: sending stateless reset for 57ca4332d73bd860 to [::1]:48273
Mar 02 13:13:45.688 DEBUG quinn_proto::endpoint: sending stateless reset for 35dc91231f030a9e to [::1]:47079
Mar 02 13:13:45.688 DEBUG quinn_proto::endpoint: sending stateless reset for 5a1939139799f277 to [::1]:47079
Mar 02 13:13:45.690 DEBUG quinn_proto::endpoint: packet for unrecognized connection ce38c28a6ab3d824
Mar 02 13:13:45.690 DEBUG quinn_proto::endpoint: sending stateless reset for ce38c28a6ab3d824 to [::1]:48273
Mar 02 13:13:45.690 TRACE drive{id=3}: quinn_proto::connection: got Data packet (36 bytes) from [::1]:47079 using id 691912417654046b
Mar 02 13:13:45.690 TRACE drive{id=3}:recv{space=Data pn=6}: quinn_proto::connection: authenticated
Mar 02 13:13:45.691 DEBUG quinn_proto::endpoint: sending stateless reset for 09de26190cb0190f to [::1]:47079
Mar 02 13:13:45.692 DEBUG quinn_proto::endpoint: packet for unrecognized connection 8052a7a5ccdd55c1
Mar 02 13:13:45.692 DEBUG quinn_proto::endpoint: sending stateless reset for 8052a7a5ccdd55c1 to [::1]:48273
Mar 02 13:13:45.692 DEBUG quinn_proto::endpoint: sending stateless reset for b766bf987ca8eb41 to [::1]:47079
Mar 02 13:13:45.692 DEBUG quinn_proto::endpoint: packet for unrecognized connection e911fb8ea8a43753
Mar 02 13:13:45.692 DEBUG quinn_proto::endpoint: sending stateless reset for e911fb8ea8a43753 to [::1]:48273
Mar 02 13:13:45.692 DEBUG quinn_proto::endpoint: sending stateless reset for abc866ab5b02a057 to [::1]:47079
Mar 02 13:13:45.692 DEBUG quinn_proto::endpoint: packet for unrecognized connection f438634a1cfa7ef2
Mar 02 13:13:45.692 DEBUG quinn_proto::endpoint: sending stateless reset for f438634a1cfa7ef2 to [::1]:48273
Mar 02 13:13:45.692 DEBUG quinn_proto::endpoint: sending stateless reset for 4bda24b5f39054a9 to [::1]:47079
Mar 02 13:13:45.696 DEBUG quinn_proto::endpoint: packet for unrecognized connection 16b8d336f4a0eee1
Mar 02 13:13:45.697 DEBUG quinn_proto::endpoint: sending stateless reset for 16b8d336f4a0eee1 to [::1]:48273
Mar 02 13:13:45.697 DEBUG quinn_proto::endpoint: sending stateless reset for cfc8b3feb962f0b2 to [::1]:47079
Mar 02 13:13:45.697 DEBUG quinn_proto::endpoint: packet for unrecognized connection e539de846b6cd948
Mar 02 13:13:45.697 DEBUG quinn_proto::endpoint: sending stateless reset for e539de846b6cd948 to [::1]:48273
Mar 02 13:13:45.698 DEBUG quinn_proto::endpoint: sending stateless reset for 0e37ba31dc59124b to [::1]:47079
Mar 02 13:13:45.698 DEBUG quinn_proto::endpoint: packet for unrecognized connection e2e7097e5959052f
Mar 02 13:13:45.699 DEBUG quinn_proto::endpoint: sending stateless reset for e2e7097e5959052f to [::1]:48273
Mar 02 13:13:45.700 DEBUG quinn_proto::endpoint: packet for unrecognized connection 12b61bdd056a7868
Mar 02 13:13:45.700 DEBUG quinn_proto::endpoint: sending stateless reset for 12b61bdd056a7868 to [::1]:48273
Mar 02 13:13:45.701 DEBUG quinn_proto::endpoint: packet for unrecognized connection 648b1bbd32a4825b
Mar 02 13:13:45.701 DEBUG quinn_proto::endpoint: sending stateless reset for 648b1bbd32a4825b to [::1]:48273
Mar 02 13:13:45.701 DEBUG quinn_proto::endpoint: sending stateless reset for 907fd98abeb70f59 to [::1]:47079
Mar 02 13:13:45.701 DEBUG quinn_proto::endpoint: packet for unrecognized connection 4c22d64d9e9fb56e
Mar 02 13:13:45.701 DEBUG quinn_proto::endpoint: sending stateless reset for 4c22d64d9e9fb56e to [::1]:48273
Mar 02 13:13:45.701 DEBUG quinn_proto::endpoint: sending stateless reset for bd25ad643e2f48a4 to [::1]:47079
Mar 02 13:13:45.706 DEBUG quinn_proto::endpoint: packet for unrecognized connection b9eea6d039f6a728
Mar 02 13:13:45.706 DEBUG quinn_proto::endpoint: sending stateless reset for b9eea6d039f6a728 to [::1]:48273
Mar 02 13:13:45.709 DEBUG quinn_proto::endpoint: sending stateless reset for 098503028321fb5d to [::1]:47079
Mar 02 13:13:45.710 DEBUG quinn_proto::endpoint: packet for unrecognized connection 1679954645ce81a1
Mar 02 13:13:45.710 DEBUG quinn_proto::endpoint: sending stateless reset for 1679954645ce81a1 to [::1]:48273
Mar 02 13:13:45.710 DEBUG quinn_proto::endpoint: sending stateless reset for 9644798d79b20605 to [::1]:47079
Mar 02 13:13:45.712 DEBUG quinn_proto::endpoint: sending stateless reset for 9c05dae436d11b75 to [::1]:47079
Mar 02 13:13:45.712 DEBUG quinn_proto::endpoint: packet for unrecognized connection 11cf9cf388a48a67
Mar 02 13:13:45.712 DEBUG quinn_proto::endpoint: sending stateless reset for 11cf9cf388a48a67 to [::1]:48273
Mar 02 13:13:45.713 DEBUG quinn_proto::endpoint: packet for unrecognized connection d7cac6cad9acf9ce
Mar 02 13:13:45.713 DEBUG quinn_proto::endpoint: sending stateless reset for d7cac6cad9acf9ce to [::1]:48273
Mar 02 13:13:45.713 DEBUG quinn_proto::endpoint: sending stateless reset for 76612d35bde05e66 to [::1]:47079
Mar 02 13:13:45.713 DEBUG quinn_proto::endpoint: sending stateless reset for 8beb8f80a2eb98b1 to [::1]:47079
Mar 02 13:13:45.713 DEBUG quinn_proto::endpoint: packet for unrecognized connection b0201ce8df22d99c
Mar 02 13:13:45.713 DEBUG quinn_proto::endpoint: sending stateless reset for b0201ce8df22d99c to [::1]:48273
Mar 02 13:13:45.713 DEBUG quinn_proto::endpoint: packet for unrecognized connection ee75a43b390e4f17
Mar 02 13:13:45.713 DEBUG quinn_proto::endpoint: sending stateless reset for ee75a43b390e4f17 to [::1]:48273
Mar 02 13:13:45.714 DEBUG quinn_proto::endpoint: packet for unrecognized connection d353ddf76538fa59
Mar 02 13:13:45.714 DEBUG quinn_proto::endpoint: sending stateless reset for d353ddf76538fa59 to [::1]:48273
Mar 02 13:13:45.715 DEBUG quinn_proto::endpoint: sending stateless reset for 8250fbfb9c27af89 to [::1]:47079
Mar 02 13:13:45.715 DEBUG quinn_proto::endpoint: packet for unrecognized connection 1a3426bb0ecad940
Mar 02 13:13:45.715 DEBUG quinn_proto::endpoint: sending stateless reset for 1a3426bb0ecad940 to [::1]:48273
Mar 02 13:13:45.715 DEBUG quinn_proto::endpoint: sending stateless reset for c3f73bdc9011d100 to [::1]:47079
Mar 02 13:13:45.715 TRACE drive{id=3}: quinn_proto::connection: timeout timer=KeyDiscard
Mar 02 13:13:45.720 DEBUG quinn_proto::endpoint: packet for unrecognized connection 50d8fd786650c20b
Mar 02 13:13:45.720 DEBUG quinn_proto::endpoint: sending stateless reset for 50d8fd786650c20b to [::1]:48273
Mar 02 13:13:45.720 DEBUG quinn_proto::endpoint: sending stateless reset for f8c0a9f66e020744 to [::1]:47079
Mar 02 13:13:45.723 DEBUG quinn_proto::endpoint: sending stateless reset for ff9d108aa132a485 to [::1]:47079
Mar 02 13:13:45.723 DEBUG quinn_proto::endpoint: packet for unrecognized connection 5851bb4e703d72d1
Mar 02 13:13:45.723 DEBUG quinn_proto::endpoint: sending stateless reset for 5851bb4e703d72d1 to [::1]:48273
Mar 02 13:13:45.723 DEBUG quinn_proto::endpoint: sending stateless reset for 5709ee63b3ca6f6e to [::1]:47079
Mar 02 13:13:45.723 DEBUG quinn_proto::endpoint: packet for unrecognized connection 799934f049a469cd
Mar 02 13:13:45.723 DEBUG quinn_proto::endpoint: sending stateless reset for 799934f049a469cd to [::1]:48273
Mar 02 13:13:45.723 DEBUG quinn_proto::endpoint: sending stateless reset for da36e1103b860ffb to [::1]:47079
Mar 02 13:13:45.723 DEBUG quinn_proto::endpoint: packet for unrecognized connection cd248b0b5a59e86b
Mar 02 13:13:45.723 DEBUG quinn_proto::endpoint: sending stateless reset for cd248b0b5a59e86b to [::1]:48273
Mar 02 13:13:45.727 DEBUG quinn_proto::endpoint: sending stateless reset for c489c91fcae39658 to [::1]:47079
Mar 02 13:13:45.728 DEBUG quinn_proto::endpoint: sending stateless reset for f8101f48474c49e9 to [::1]:47079
Mar 02 13:13:45.728 DEBUG quinn_proto::endpoint: packet for unrecognized connection 825ae070c967e61b
Mar 02 13:13:45.728 DEBUG quinn_proto::endpoint: sending stateless reset for 825ae070c967e61b to [::1]:48273
Mar 02 13:13:45.735 DEBUG quinn_proto::endpoint: sending stateless reset for c76013f9fe07297b to [::1]:47079
Mar 02 13:13:45.735 DEBUG quinn_proto::endpoint: packet for unrecognized connection 39a0f3ae8eacbb0e
Mar 02 13:13:45.735 DEBUG quinn_proto::endpoint: sending stateless reset for 39a0f3ae8eacbb0e to [::1]:48273
Mar 02 13:13:45.736 DEBUG quinn_proto::endpoint: packet for unrecognized connection e3a79b2cd278b05f
Mar 02 13:13:45.736 DEBUG quinn_proto::endpoint: sending stateless reset for e3a79b2cd278b05f to [::1]:48273
Mar 02 13:13:45.736 DEBUG quinn_proto::endpoint: sending stateless reset for 5199cfefeac6c126 to [::1]:47079
Mar 02 13:13:45.738 DEBUG quinn_proto::endpoint: packet for unrecognized connection 44abc4bee20666aa
Mar 02 13:13:45.738 DEBUG quinn_proto::endpoint: sending stateless reset for 44abc4bee20666aa to [::1]:48273
Mar 02 13:13:45.738 DEBUG quinn_proto::endpoint: sending stateless reset for 7cd1b611f503e1eb to [::1]:47079
Mar 02 13:13:45.738 DEBUG quinn_proto::endpoint: packet for unrecognized connection 1762a3da94016831
Mar 02 13:13:45.738 DEBUG quinn_proto::endpoint: sending stateless reset for 1762a3da94016831 to [::1]:48273
Mar 02 13:13:45.738 DEBUG quinn_proto::endpoint: sending stateless reset for 974ae0241815e580 to [::1]:47079
Mar 02 13:13:45.738 DEBUG quinn_proto::endpoint: packet for unrecognized connection df7c6ea4f4a870b7
Mar 02 13:13:45.738 DEBUG quinn_proto::endpoint: sending stateless reset for df7c6ea4f4a870b7 to [::1]:48273
Mar 02 13:13:45.738 DEBUG quinn_proto::endpoint: sending stateless reset for d8a00f2f240403c8 to [::1]:47079
Mar 02 13:13:45.738 DEBUG quinn_proto::endpoint: packet for unrecognized connection 962e9a99cbf85255
Mar 02 13:13:45.738 DEBUG quinn_proto::endpoint: sending stateless reset for 962e9a99cbf85255 to [::1]:48273
Mar 02 13:13:45.739 DEBUG quinn_proto::endpoint: packet for unrecognized connection c4165f79eb5bacb0
Mar 02 13:13:45.739 DEBUG quinn_proto::endpoint: sending stateless reset for c4165f79eb5bacb0 to [::1]:48273
Mar 02 13:13:45.739 DEBUG quinn_proto::endpoint: sending stateless reset for abf63ea015bc97d3 to [::1]:47079
Mar 02 13:13:45.739 DEBUG quinn_proto::endpoint: sending stateless reset for 570276b98342b24a to [::1]:47079
Mar 02 13:13:45.739 DEBUG quinn_proto::endpoint: packet for unrecognized connection 7170c3716bb4a760
Mar 02 13:13:45.739 DEBUG quinn_proto::endpoint: sending stateless reset for 7170c3716bb4a760 to [::1]:48273
Mar 02 13:13:45.740 DEBUG quinn_proto::endpoint: packet for unrecognized connection 78e80473b7e7a0b7
Mar 02 13:13:45.740 DEBUG quinn_proto::endpoint: sending stateless reset for 78e80473b7e7a0b7 to [::1]:48273
Mar 02 13:13:45.740 DEBUG quinn_proto::endpoint: sending stateless reset for 7a6e64066547e233 to [::1]:47079
Mar 02 13:13:45.740 DEBUG quinn_proto::endpoint: packet for unrecognized connection dc7bd2a78c95ae23
Mar 02 13:13:45.741 DEBUG quinn_proto::endpoint: sending stateless reset for dc7bd2a78c95ae23 to [::1]:48273
Mar 02 13:13:45.742 DEBUG quinn_proto::endpoint: sending stateless reset for 15472580d2e7f6ea to [::1]:47079
Mar 02 13:13:45.742 DEBUG quinn_proto::endpoint: sending stateless reset for b10875ef919d1267 to [::1]:47079
Mar 02 13:13:45.742 DEBUG quinn_proto::endpoint: packet for unrecognized connection e4d036f569e3d999
Mar 02 13:13:45.742 DEBUG quinn_proto::endpoint: sending stateless reset for e4d036f569e3d999 to [::1]:48273
Mar 02 13:13:45.742 DEBUG quinn_proto::endpoint: sending stateless reset for c2bd38fd161d8485 to [::1]:47079
Mar 02 13:13:45.747 DEBUG quinn_proto::endpoint: packet for unrecognized connection 8e9b9e2073582149
Mar 02 13:13:45.748 DEBUG quinn_proto::endpoint: sending stateless reset for 8e9b9e2073582149 to [::1]:48273
Mar 02 13:13:45.749 DEBUG quinn_proto::endpoint: packet for unrecognized connection 8be16220c1dc6432
Mar 02 13:13:45.749 DEBUG quinn_proto::endpoint: sending stateless reset for 8be16220c1dc6432 to [::1]:48273
Mar 02 13:13:45.752 DEBUG quinn_proto::endpoint: sending stateless reset for 6116ba2e0fafcdb6 to [::1]:47079
Mar 02 13:13:45.753 DEBUG quinn_proto::endpoint: packet for unrecognized connection 944e580f8d68504f
Mar 02 13:13:45.753 DEBUG quinn_proto::endpoint: sending stateless reset for 944e580f8d68504f to [::1]:48273
Mar 02 13:13:45.753 DEBUG quinn_proto::endpoint: sending stateless reset for b5a29b31ba7f43e2 to [::1]:47079
Mar 02 13:13:45.753 DEBUG quinn_proto::endpoint: packet for unrecognized connection 1dc432b654ced3c4
Mar 02 13:13:45.753 DEBUG quinn_proto::endpoint: sending stateless reset for 1dc432b654ced3c4 to [::1]:48273
Mar 02 13:13:45.754 DEBUG quinn_proto::endpoint: sending stateless reset for 974ae0241815e580 to [::1]:47079
Mar 02 13:13:45.757 DEBUG quinn_proto::endpoint: sending stateless reset for 7c20b1e7e6027d29 to [::1]:47079
Mar 02 13:13:45.757 DEBUG quinn_proto::endpoint: packet for unrecognized connection 2ee800d54cc7deea
Mar 02 13:13:45.757 DEBUG quinn_proto::endpoint: sending stateless reset for 2ee800d54cc7deea to [::1]:48273
Mar 02 13:13:45.758 DEBUG quinn_proto::endpoint: sending stateless reset for d31d6e82c8c81eaf to [::1]:47079
Mar 02 13:13:45.761 DEBUG quinn_proto::endpoint: sending stateless reset for 93102c666f1b36e1 to [::1]:47079
Mar 02 13:13:45.764 DEBUG quinn_proto::endpoint: sending stateless reset for b1f3405fbbb017aa to [::1]:47079
Mar 02 13:13:45.764 DEBUG quinn_proto::endpoint: packet for unrecognized connection 0d85093de35b459a
Mar 02 13:13:45.764 DEBUG quinn_proto::endpoint: sending stateless reset for 0d85093de35b459a to [::1]:48273
Mar 02 13:13:45.767 DEBUG quinn_proto::endpoint: packet for unrecognized connection e1ce098d76073fdd
Mar 02 13:13:45.767 DEBUG quinn_proto::endpoint: sending stateless reset for e1ce098d76073fdd to [::1]:48273
Mar 02 13:13:45.776 DEBUG quinn_proto::endpoint: packet for unrecognized connection dc637fb339b3c93a
Mar 02 13:13:45.776 DEBUG quinn_proto::endpoint: sending stateless reset for dc637fb339b3c93a to [::1]:48273
Mar 02 13:13:45.778 DEBUG quinn_proto::endpoint: sending stateless reset for 8575959cf09a801d to [::1]:47079
Mar 02 13:13:45.781 DEBUG quinn_proto::endpoint: sending stateless reset for 4d93e4e14dd3aaf0 to [::1]:47079
Mar 02 13:13:45.788 DEBUG quinn_proto::endpoint: packet for unrecognized connection 3e9be606e125ed11
Mar 02 13:13:45.788 DEBUG quinn_proto::endpoint: sending stateless reset for 3e9be606e125ed11 to [::1]:48273
Mar 02 13:13:45.790 DEBUG quinn_proto::endpoint: sending stateless reset for b3db419a3522faff to [::1]:47079
Mar 02 13:13:45.790 DEBUG quinn_proto::endpoint: packet for unrecognized connection a1916d5eff076f7e
Mar 02 13:13:45.790 DEBUG quinn_proto::endpoint: sending stateless reset for a1916d5eff076f7e to [::1]:48273
Mar 02 13:13:45.801 DEBUG quinn_proto::endpoint: sending stateless reset for dc17e8071e41db5c to [::1]:47079
Mar 02 13:13:45.801 DEBUG quinn_proto::endpoint: sending stateless reset for 87a49c2e288eca35 to [::1]:47079
Mar 02 13:13:45.824 DEBUG quinn_proto::endpoint: packet for unrecognized connection cb3154020ab25693
Mar 02 13:13:45.824 DEBUG quinn_proto::endpoint: sending stateless reset for cb3154020ab25693 to [::1]:48273
Mar 02 13:13:45.839 DEBUG quinn_proto::endpoint: sending stateless reset for 2b36e3cd3d79feb6 to [::1]:47079
Mar 02 13:13:45.854 DEBUG quinn_proto::endpoint: packet for unrecognized connection 7ac01d4cc7494054
Mar 02 13:13:45.854 DEBUG quinn_proto::endpoint: sending stateless reset for 7ac01d4cc7494054 to [::1]:48273
Mar 02 13:13:45.854 DEBUG quinn_proto::endpoint: sending stateless reset for e1ef2593e25d1bf3 to [::1]:47079
Mar 02 13:13:45.865 DEBUG quinn_proto::endpoint: packet for unrecognized connection a12f053292612708
Mar 02 13:13:45.865 DEBUG quinn_proto::endpoint: sending stateless reset for a12f053292612708 to [::1]:48273
Mar 02 13:13:45.866 DEBUG quinn_proto::endpoint: sending stateless reset for e3ba2370b6ccdc3f to [::1]:47079
Mar 02 13:13:45.866 DEBUG quinn_proto::endpoint: packet for unrecognized connection 3d464742f3f9ffc0
Mar 02 13:13:45.866 DEBUG quinn_proto::endpoint: sending stateless reset for 3d464742f3f9ffc0 to [::1]:48273
Mar 02 13:13:45.866 DEBUG quinn_proto::endpoint: packet for unrecognized connection 3d464742f3f9ffc0
Mar 02 13:13:45.866 DEBUG quinn_proto::endpoint: sending stateless reset for 3d464742f3f9ffc0 to [::1]:48273
Mar 02 13:13:45.866 DEBUG quinn_proto::endpoint: sending stateless reset for 7bd62b368038768a to [::1]:47079
Mar 02 13:13:45.889 DEBUG quinn_proto::endpoint: sending stateless reset for 7bd62b368038768a to [::1]:47079
Mar 02 13:13:45.894 DEBUG quinn_proto::endpoint: packet for unrecognized connection c8cd35726e03ced8
Mar 02 13:13:45.894 DEBUG quinn_proto::endpoint: sending stateless reset for c8cd35726e03ced8 to [::1]:48273
Mar 02 13:13:46.650 TRACE drive{id=4}: quinn_proto::connection: timeout timer=LossDetection
Mar 02 13:13:46.650 TRACE drive{id=4}: quinn_proto::connection: PTO fired in_flight=977 count=0 space=Initial
Mar 02 13:13:46.650 TRACE drive{id=4}:send{space=Initial pn=1}: quinn_proto::connection: ACK
Mar 02 13:13:46.650 TRACE drive{id=4}:send{space=Initial pn=1}: quinn_proto::connection: CRYPTO: off 0 len 90
Mar 02 13:13:46.650 TRACE drive{id=4}: quinn_proto::connection: sending 145 byte datagram
Mar 02 13:13:46.650 TRACE drive{id=4}:send{space=Initial pn=2}: quinn_proto::connection: ACK
Mar 02 13:13:46.650 TRACE drive{id=4}:send{space=Initial pn=2}: quinn_proto::connection: CRYPTO: off 0 len 90
Mar 02 13:13:46.650 TRACE drive{id=4}: quinn_proto::connection: sending 145 byte datagram
Mar 02 13:13:46.650 DEBUG quinn_proto::endpoint: packet for unrecognized connection 654a46bae0b51cd9
Mar 02 13:13:46.650 DEBUG quinn_proto::endpoint: sending stateless reset for 654a46bae0b51cd9 to [::1]:48273
Mar 02 13:13:46.650 DEBUG quinn_proto::endpoint: packet for unrecognized connection 654a46bae0b51cd9
Mar 02 13:13:46.650 DEBUG quinn_proto::endpoint: sending stateless reset for 654a46bae0b51cd9 to [::1]:48273
Mar 02 13:13:46.650 DEBUG quinn_proto::endpoint: sending stateless reset for 0e28b03210fae4b6 to [::1]:47079
Mar 02 13:13:46.650 DEBUG quinn_proto::endpoint: packet for unrecognized connection 297bc9f1869dd2b9
Mar 02 13:13:46.650 DEBUG quinn_proto::endpoint: sending stateless reset for 297bc9f1869dd2b9 to [::1]:48273
Mar 02 13:13:46.653 DEBUG quinn_proto::endpoint: sending stateless reset for 1d248b671732723f to [::1]:47079
Mar 02 13:13:46.660 DEBUG quinn_proto::endpoint: packet for unrecognized connection 477dc2c164ff4270
Mar 02 13:13:46.660 DEBUG quinn_proto::endpoint: sending stateless reset for 477dc2c164ff4270 to [::1]:48273
Mar 02 13:13:46.660 DEBUG quinn_proto::endpoint: sending stateless reset for fee60a624dfe4ca9 to [::1]:47079
Mar 02 13:13:46.660 DEBUG quinn_proto::endpoint: sending stateless reset for 6b259e07c8aca4a0 to [::1]:47079
Mar 02 13:13:46.672 DEBUG quinn_proto::endpoint: packet for unrecognized connection c8ab18eebfb1ad78
Mar 02 13:13:46.672 DEBUG quinn_proto::endpoint: sending stateless reset for c8ab18eebfb1ad78 to [::1]:48273
Mar 02 13:13:46.672 DEBUG quinn_proto::endpoint: sending stateless reset for 1662473369a18858 to [::1]:47079
Mar 02 13:13:46.672 DEBUG quinn_proto::endpoint: packet for unrecognized connection 1b223ad0f715da9d
Mar 02 13:13:46.672 DEBUG quinn_proto::endpoint: sending stateless reset for 1b223ad0f715da9d to [::1]:48273
Mar 02 13:13:46.674 DEBUG quinn_proto::endpoint: packet for unrecognized connection eea429273b419c66
Mar 02 13:13:46.674 DEBUG quinn_proto::endpoint: sending stateless reset for eea429273b419c66 to [::1]:48273
Mar 02 13:13:46.674 DEBUG quinn_proto::endpoint: packet for unrecognized connection eea429273b419c66
Mar 02 13:13:46.674 DEBUG quinn_proto::endpoint: sending stateless reset for eea429273b419c66 to [::1]:48273
Mar 02 13:13:46.674 DEBUG quinn_proto::endpoint: sending stateless reset for 363921137fc6fb5f to [::1]:47079
Mar 02 13:13:46.674 DEBUG quinn_proto::endpoint: packet for unrecognized connection 765a99883f2ba924
Mar 02 13:13:46.674 DEBUG quinn_proto::endpoint: sending stateless reset for 765a99883f2ba924 to [::1]:48273
Mar 02 13:13:46.679 DEBUG quinn_proto::endpoint: sending stateless reset for 6207e4bc079377ad to [::1]:47079
Mar 02 13:13:46.681 DEBUG quinn_proto::endpoint: packet for unrecognized connection 8416b4fae3fe5e4e
Mar 02 13:13:46.681 DEBUG quinn_proto::endpoint: sending stateless reset for 8416b4fae3fe5e4e to [::1]:48273
Mar 02 13:13:46.681 DEBUG quinn_proto::endpoint: sending stateless reset for f9a025acf5a3da4e to [::1]:47079
Mar 02 13:13:46.682 DEBUG quinn_proto::endpoint: sending stateless reset for 6fbe73a43b5ea8b3 to [::1]:47079
Mar 02 13:13:46.690 DEBUG quinn_proto::endpoint: sending stateless reset for 1cbb1b741a244d33 to [::1]:47079
Mar 02 13:13:46.698 DEBUG quinn_proto::endpoint: packet for unrecognized connection cfaf54fa1a6873e2
Mar 02 13:13:46.698 DEBUG quinn_proto::endpoint: sending stateless reset for cfaf54fa1a6873e2 to [::1]:48273
Mar 02 13:13:46.698 DEBUG quinn_proto::endpoint: packet for unrecognized connection bb9fcd8e2ab4e0fb
Mar 02 13:13:46.698 DEBUG quinn_proto::endpoint: sending stateless reset for bb9fcd8e2ab4e0fb to [::1]:48273
Mar 02 13:13:46.698 DEBUG quinn_proto::endpoint: sending stateless reset for c03577a83aea71da to [::1]:47079
Mar 02 13:13:46.702 DEBUG quinn_proto::endpoint: packet for unrecognized connection 5ab61f21a7c9c9e6
Mar 02 13:13:46.702 DEBUG quinn_proto::endpoint: sending stateless reset for 5ab61f21a7c9c9e6 to [::1]:48273
Mar 02 13:13:46.702 DEBUG quinn_proto::endpoint: sending stateless reset for d758119570ab051d to [::1]:47079
Mar 02 13:13:46.704 DEBUG quinn_proto::endpoint: packet for unrecognized connection 0c4d94220a16914c
Mar 02 13:13:46.704 DEBUG quinn_proto::endpoint: sending stateless reset for 0c4d94220a16914c to [::1]:48273
Mar 02 13:13:46.707 DEBUG quinn_proto::endpoint: sending stateless reset for c03577a83aea71da to [::1]:47079
Mar 02 13:13:46.707 DEBUG quinn_proto::endpoint: packet for unrecognized connection b609bd3605eb79a9
Mar 02 13:13:46.707 DEBUG quinn_proto::endpoint: sending stateless reset for b609bd3605eb79a9 to [::1]:48273

@djc djc mentioned this issue Mar 2, 2020
Closed
@Ralith
Copy link
Collaborator

Ralith commented Mar 5, 2020

I suspect this might be due to the somewhat dubious handshake state machine in draft 24. I'm going to try to get us updated to draft 27 and then revisit.

@Ralith
Copy link
Collaborator

Ralith commented Mar 8, 2020

The draft 27 update has been merged.

@alecmocatta
Copy link
Contributor Author

I'm still seeing the same hang on current master. From running the test case a few times I believe it is less frequent, though I could be wrong and I'll test more tomorrow. I'm also seeing #670.

@Ralith
Copy link
Collaborator

Ralith commented Mar 10, 2020

Thanks for checking. I haven't been able to reproduce either case using the current reduced-hang branch, having ran up to 100k iterations; updated traces could be helpful.

@Ralith
Copy link
Collaborator

Ralith commented Mar 10, 2020

Reviewing the above traces, it looks like the server is functioning normally on the -proto level (i.e. it deems the connection established and processes the client's stream frames), so there must be something wrong with the pathway that wakes up Connecting.

@Ralith
Copy link
Collaborator

Ralith commented Mar 10, 2020

In the second-to-last trace above, I believe id=4 represents the server's side of the connection in question. The line

Mar 01 19:15:17.654 TRACE drive{id=4}:recv{space=Handshake pn=0}: quinn_proto::connection: established

indicates that control has reached a point which must pass through self.events.push_back(Event::Connected);. That queue is drained in Connection::forward_app_events, which is always called after process_conn_events, in which packets are handled and the event is generated. forward_app_events always signals self.on_connected when it encounters a Connected event. That's the oneshot that Connecting is waiting on.

I'm having trouble seeing how that chain could malfunction. Do I recall correctly you mentioning you suspected a tokio issue previously?

@djc
Copy link
Member

djc commented Mar 10, 2020

@alecmocatta would be cool if you can investigate some more with current master (which includes some fixes). Please also make sure to run the latest tokio, and then maybe add some debugging based on @Ralith's pointers above? Let me know if you have more questions.

@alecmocatta
Copy link
Contributor Author

alecmocatta commented Mar 10, 2020 •
edited
Loading

The issue that I suspected was related to tokio I saw only when spawning onto a multi-threaded runtime, and it involved two threads busy-looping. I don't think it's likely to be triggering this as well but it's not impossible.

Under my setup - a 1-core linux 4.18 VM running under VirtualBox, after having run the above netem command to simulate an unreliable network - I get about 20 iterations per second, and ~99% of the time I see either the hang or #670 within a minute. Adding RUST_LOG=quinn=trace seems to make the hang less likely to occur, and extends that time to ~3 minutes. I've added my Cargo.lock to the branch, and an updated trace of the original hang here, and the #670 error here.

@djc
Copy link
Member

djc commented Mar 10, 2020

Ah, but that repository is still using the sync-defragmented branch? Should try with master.

@alecmocatta
Copy link
Contributor Author

@djc Check out the reduced-hang branch.

@Ralith
Copy link
Collaborator

Ralith commented Mar 10, 2020

Thanks for verifying! I forgot about the netem stuff, will investigate further today. Given the above investigation, I'm sure I can at least find an assumption getting invalidated somewhere, assuming I was looking at the right ID.

@Ralith Ralith mentioned this issue Mar 11, 2020
Closed
@Ralith
Copy link
Collaborator

Ralith commented Mar 11, 2020 •
edited
Loading

With netem I'm able to reproduce both issues very consistently--often in the first iteration, in fact. Thanks!

@Ralith Ralith mentioned this issue Mar 11, 2020
Open
@Ralith
Copy link
Collaborator

Ralith commented Mar 11, 2020 •
edited
Loading

I'm pretty sure I see what's happening with the hang:

  1. The client's first packet is being duplicated over a significant time period (about 22ms in one example). The first packet creates a connection which goes through the full life cycle and is ultimately closed and forgotten.
  2. The duplicate arrives. The server has no way to distinguish it from a genuine fresh connection attempt, so it proceeds with the full handshake procedure, including yielding a new Connecting future to the application. This pattern can be identified easily by looking for traces where initial_dcid=foo appears once and icid=foo appears twice.
  3. The client endpoint no longer has state for this connection, so it rejects the server's handshake packets with a stateless reset.
  4. The server doesn't have a current reset token from the client, so it cannot recognize the stateless resets, and responds with its own stateless resets. This cycle continues until the stateless reset size drops below the critical threshold and no longer prompts a response.
  5. Having received no acknowledgement, the server retransmits the handshake messages. This restarts the above cycle.
  6. The server's retransmits add up until they're halted by anti-amplification.

At this point the server cannot take any further action on the connection initiated by the duplicated packet. Because you've disabled the idle timeout, the connection is permanently hung. This is working as intended.

In summary, the idle timeout must not be disabled in environments where a client might disappear unexpectedly or packets may be duplicated and no other mechanism exists to clean up zombie connections. I'll prepare a PR to update the documentation to clarify this.

@Ralith Ralith mentioned this issue Mar 11, 2020
Merged
@djc djc closed this as completed in #676 Mar 11, 2020
@alecmocatta
Copy link
Contributor Author

Thanks for that comprehensive explanation @Ralith!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Document hazards of infinite idle timeouts quinn-rs/quinn
3 participants
@Ralith @djc @alecmocatta