Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Pairing M5 failed - PacketBuffer: pool EMPTY after receiving Sigma3 message #23258

Closed
kean-apple opened this issue Oct 19, 2022 · 7 comments
Closed

[BUG] Pairing M5 failed - PacketBuffer: pool EMPTY after receiving Sigma3 message #23258

kean-apple opened this issue Oct 19, 2022 · 7 comments
Assignees
@tcarmelveilleux
Labels
esp32 stale Stale issue or PR

Comments

@kean-apple
Copy link

kean-apple commented Oct 19, 2022 •
edited by bzbarsky-apple
Loading

Reproduction steps

  1. Reset M5 board
  2. Tried pairing with Apple Home App
  3. After receiving Sigma3 message on M5, started to see PacketBuffer: pool EMPTY messages. Pairing did fail

M5 snippet:

I (55961) chip[SC]: Received Sigma1 msg
I (55981) chip[SC]: CASE matched destination ID: fabricIndex 1, NodeID 0x00000000454F246C
I (56921) ROUTE_HOOK: Received RIO
I (56921) ROUTE_HOOK: prefix FD8B:6557:EAAF:: lifetime 1800

I (56931) chip[EM]: <<< [E:14722r M:193169753 (Ack:74941945)] (U) Msg TX to 0:0000000000000000 [0000] --- Type 0000:31 (SecureChannel:CASE_Sigma2)
I (56941) chip[IN]: (U) Sending msg 193169753 to IP address 'UDP:[FDDA:64FB:9A0A:48E5:1CFF:2DD5:827A:A8A6%st1]:49751'
I (56951) chip[SC]: Sent Sigma2 msg
E (56951) chip[DL]: Long dispatch time: 1038 ms, for event type 3
I (57121) chip[EM]: >>> [E:14722r M:74941946 (Ack:193169753)] (U) Msg RX from 0:3CD135D602565506 [0000] --- Type 0000:32 (SecureChannel:CASE_Sigma3)
I (57131) chip[EM]: <<< [E:14722r M:193169754 (Ack:74941946)] (U) Msg TX to 0:0000000000000000 [0000] --- Type 0000:10 (SecureChannel:StandaloneAck)
I (57141) chip[IN]: (U) Sending msg 193169754 to IP address 'UDP:[FDDA:64FB:9A0A:48E5:1CFF:2DD5:827A:A8A6%st1]:49751'
I (57151) chip[SC]: Received Sigma3 msg
E (57151) chip[SC]: The device does not support GetClock_RealTimeMS() API: 6c.  Falling back to Last Known Good UTC Time
E (57521) chip[CSL]: PacketBuffer: pool EMPTY.
E (57531) chip[IN]: Cannot copy received pbuf of size 55
E (57531) chip[CSL]: PacketBuffer: pool EMPTY.
E (57541) chip[IN]: Cannot copy received pbuf of size 55
E (57641) chip[CSL]: PacketBuffer: pool EMPTY.
E (57641) chip[IN]: Cannot copy received pbuf of size 480
E (57671) chip[CSL]: PacketBuffer: pool EMPTY.
E (57671) chip[IN]: Cannot copy received pbuf of size 65
E (57841) chip[CSL]: PacketBuffer: pool EMPTY.
E (57841) chip[IN]: Cannot copy received pbuf of size 165
E (57861) chip[CSL]: PacketBuffer: pool EMPTY.
E (57861) chip[IN]: Cannot copy received pbuf of size 55
E (57931) chip[CSL]: PacketBuffer: pool EMPTY.
E (57941) chip[IN]: Cannot copy received pbuf of size 142
E (58041) chip[CSL]: PacketBuffer: pool EMPTY.
E (58041) chip[IN]: Cannot copy received pbuf of size 47
E (58141) chip[CSL]: PacketBuffer: pool EMPTY.
E (58141) chip[IN]: Cannot copy received pbuf of size 125
E (58161) chip[CSL]: PacketBuffer: pool EMPTY.
E (58161) chip[IN]: Cannot copy received pbuf of size 211
E (58291) chip[CSL]: PacketBuffer: pool EMPTY.
E (58291) chip[IN]: Cannot copy received pbuf of size 373
I (58511) chip[EM]: <<< [E:14722r M:193169755 (Ack:74941946)] (U) Msg TX to 0:0000000000000000 [0000] --- Type 0000:40 (SecureChannel:StatusReport)
I (58531) chip[IN]: (U) Sending msg 193169755 to IP address 'UDP:[FDDA:64FB:9A0A:48E5:1CFF:2DD5:827A:A8A6%st1]:49751'
E (58531) chip[CSL]: PacketBuffer: pool EMPTY.

Bug prevalence

This was seen 1 in 20 pairing attempts

GitHub hash of the SDK that was being used

c279578

Platform

esp32

Platform Version(s)

ESP-IDF v4.4.2

Anything else?

M5-pairing-failed-empty-packet-buffer.txt
@mrjerryjohns
Copy link
Contributor

mrjerryjohns commented Oct 19, 2022 •
edited
Loading

This likely the same cause as #23180, specifically, my comment here. In this situation, you're not seeing event queue overflows but PBUF overflow because the event queue is likely not the bottleneck here (i.e it's sized large enough such that it can weather the number of DNS messages being processed), but the PBUF pool isn't large enough to handle the mDNS packet deluge.

@mrjerryjohns
Copy link
Contributor

Can you let me know what the value for CONFIG_MAX_EVENT_QUEUE_SIZE is in the build that ran on the esp32?

@tcarmelveilleux
Copy link
Contributor

I have also seen large numbers of IPv6 NA (Neighbour Announcement) around failure, not just mDNS. Packet capture at the Wi-Fi level on the link used by ESP-32 is rquired to debug further. There may be LWIP constants related to IPv6 that cause some of these overflows as well.

@kean-apple
Copy link
Author

I used the default: #define CHIP_DEVICE_CONFIG_MAX_EVENT_QUEUE_SIZE 100

@mrjerryjohns
Copy link
Contributor

The default is 25 no? With 100, that explains why you're seeing PBUF issues, since the event queue no longer is the limiting factor.

@tcarmelveilleux tcarmelveilleux self-assigned this Dec 7, 2022
tcarmelveilleux added a commit to tcarmelveilleux/connectedhomeip that referenced this issue Dec 7, 2022
@tcarmelveilleux
Introduce basic UDP packet filtering scheme
74fda22 
- Embedded class devices on Wi-Fi networks can see large number of
  mDNS packets from normal network. If these arrive while device
  is processing CASE/PASE or any other long running system activity,
  the amount of queuing can overrun the packet buffer pools or
  event queues, which can cause very bad outcomes.
- Filtering of packets can always be done by a product deep in its
  network stack. However, allowing some level of basic filtering
  for UDP packets (that cover most of the traffic that could be
  problematic and need queuing) for at least lwIP devices allows
  sharing code for filters that may be smarter.

Follow-up PR will show an example hook-up to ESP32

Issue project-chip#23258
Issue project-chip#23180

This PR:
- Adds generic EndpointQueueFilter.h, which is portable/testable
  on its own.
- Adds hook-ups to UDPEndPointImplLwIP to run an EndpointQueueFilter
  on incoming packets

Testing done:
- Full unit test suite for the filtering framework and basic
  filter.
@tcarmelveilleux tcarmelveilleux mentioned this issue Dec 7, 2022
Merged
andy31415 pushed a commit that referenced this issue Dec 7, 2022
@tcarmelveilleux
Introduce basic UDP packet filtering scheme (#23957)
f8f4f1e 
* Introduce basic UDP packet filtering scheme

- Embedded class devices on Wi-Fi networks can see large number of
  mDNS packets from normal network. If these arrive while device
  is processing CASE/PASE or any other long running system activity,
  the amount of queuing can overrun the packet buffer pools or
  event queues, which can cause very bad outcomes.
- Filtering of packets can always be done by a product deep in its
  network stack. However, allowing some level of basic filtering
  for UDP packets (that cover most of the traffic that could be
  problematic and need queuing) for at least lwIP devices allows
  sharing code for filters that may be smarter.

Follow-up PR will show an example hook-up to ESP32

Issue #23258
Issue #23180

This PR:
- Adds generic EndpointQueueFilter.h, which is portable/testable
  on its own.
- Adds hook-ups to UDPEndPointImplLwIP to run an EndpointQueueFilter
  on incoming packets

Testing done:
- Full unit test suite for the filtering framework and basic
  filter.

* Address review comments
@bzbarsky-apple bzbarsky-apple mentioned this issue Apr 7, 2023
Closed
@stale
Copy link

stale bot commented Jun 9, 2023

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

@stale stale bot added the stale Stale issue or PR label Jun 9, 2023
@dhrishi
Copy link
Contributor

dhrishi commented Oct 25, 2024

Closing this as the memory related issues with M5 have been solved

@dhrishi dhrishi closed this as completed Oct 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tcarmelveilleux tcarmelveilleux

Labels
esp32 stale Stale issue or PR
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@tcarmelveilleux @mrjerryjohns @dhrishi @bzbarsky-apple @kean-apple