Downlink messages are queued but not sent to class A ABP end-device

[Sheng2216]

The issue we are experiencing is that the downlink is never sent to the end-device, and this seems to occur on ChirpStack V4 only. Our setup includes a gateway running the UDP packet forwarder and the packet multiplexer. The packet multiplex configuration links two ChirpStack servers: one is V4 and the other is V3. On the V3 server, the downlink works fine, although a few downlink packets may be lost occasionally. However, on the V4 server, all the downlinks we created in the Enqueue list are completely ignored, even though the V4 server is still receiving uplinks.

we disabled the frame-counter validation since end device is ABP:

and this is the EU868 device profile:

and here is the chirpstack-multiplexer config:

[general]
# Log level
#
# debug=5, info=4, warning=3, error=2, fatal=1, panic=0
log_level=5


[packet_multiplexer]
# Bind
#
# The interface:port on which the packet-multiplexer will bind for receiving
# data from the packet-forwarder (UDP data).
bind="0.0.0.0:1700"

# Backends
#
# The backends to which the packet-multiplexer will forward the
# packet-forwarder UDP data.
#
# Example:
[[packet_multiplexer.backend]]
# Chirpstack V4 on rpi
  host="10.2.13.28:1700"
  uplink_only=false
  gateway_ids=[
    "E45F01FFFEB94911"
  ]

# Chirpstack V4 on rpi
[[packet_multiplexer.backend]]
  host="10.2.13.31:1700"
  uplink_only=false
  gateway_ids = [
    "E45F01FFFEB94911"
  ]

[Sheng2216]

and here is the log from the chirpstack container:

2023-05-25T09:42:11.635965Z ERROR chirpstack::uplink: Deduplication error error=No channel found for frequency: 868800000, dr: 7
2023-05-25T09:42:12.185195Z  INFO chirpstack::gateway::backend::mqtt: Message received from gateway region_config_id="eu868" topic="eu868/gateway/e45f01fffeb94911/event/up" qos=0 json=false
2023-05-25T09:42:12.391278Z  INFO up{deduplication_id=e51786a5-9471-496c-a6c7-a1364d27ebe9}: chirpstack::uplink: Uplink received m_type="ConfirmedDataUp"
2023-05-25T09:42:12.397507Z  WARN up{deduplication_id=e51786a5-9471-496c-a6c7-a1364d27ebe9}:data_up: chirpstack::uplink::data: No device-session exists for dev_addr dev_addr=02e43483
2023-05-25T09:42:15.292899Z  INFO chirpstack::gateway::backend::mqtt: Message received from gateway region_config_id="eu868" topic="eu868/gateway/e45f01fffeb94911/event/up" qos=0 json=false
2023-05-25T09:42:15.498763Z  INFO up{deduplication_id=304d25de-a199-493e-b39e-da08c9a73ed4}: chirpstack::uplink: Uplink received m_type="ConfirmedDataUp"
2023-05-25T09:42:15.505079Z  WARN up{deduplication_id=304d25de-a199-493e-b39e-da08c9a73ed4}:data_up: chirpstack::uplink::data: No device-session exists for dev_addr dev_addr=027b5553
2023-05-25T09:42:16.725621Z  INFO chirpstack::gateway::backend::mqtt: Message received from gateway region_config_id="eu868" topic="eu868/gateway/e45f01fffeb94911/event/up" qos=0 json=false
2023-05-25T09:42:16.932344Z  INFO up{deduplication_id=1d349916-3f38-4107-84db-89f4a0d258dc}: chirpstack::uplink: Uplink received m_type="UnconfirmedDataUp"
2023-05-25T09:42:16.942877Z  INFO up{deduplication_id=1d349916-3f38-4107-84db-89f4a0d258dc}:data_up: chirpstack::storage::device_session: Device-session saved dev_eui=2f4e255fff1b8275 dev_addr=01bbb2b2
2023-05-25T09:42:16.954307Z  INFO chirpstack::storage::device: Setting device lock dev_eui=2f4e255fff1b8275
2023-05-25T09:42:16.966608Z  INFO up{deduplication_id=1d349916-3f38-4107-84db-89f4a0d258dc}:data_up: chirpstack::storage::device: Data-rate updated dev_eui=2f4e255fff1b8275 dr=0
2023-05-25T09:42:16.970270Z  INFO up{deduplication_id=1d349916-3f38-4107-84db-89f4a0d258dc}:data_up: chirpstack::storage::device_gateway: Gateway rx-info saved dev_eui=2f4e255fff1b8275
2023-05-25T09:42:16.973147Z  INFO up{deduplication_id=1d349916-3f38-4107-84db-89f4a0d258dc}:data_up: chirpstack::storage::device_session: Device-session saved dev_eui=2f4e255fff1b8275 dev_addr=01bbb2b2
2023-05-25T09:42:16.973876Z  INFO chirpstack::integration::mqtt: Publishing event topic=application/dd93f513-0d63-4fb3-9973-bcd75cda828a/device/2f4e255fff1b8275/event/up
2023-05-25T09:42:16.976777Z  INFO up{deduplication_id=1d349916-3f38-4107-84db-89f4a0d258dc}:data_up: chirpstack::storage::metrics: Metrics saved name=device:2f4e255fff1b8275 aggregation=HOUR
2023-05-25T09:42:16.979231Z  INFO up{deduplication_id=1d349916-3f38-4107-84db-89f4a0d258dc}:data_up: chirpstack::storage::metrics: Metrics saved name=device:2f4e255fff1b8275 aggregation=DAY
2023-05-25T09:42:16.981712Z  INFO up{deduplication_id=1d349916-3f38-4107-84db-89f4a0d258dc}:data_up: chirpstack::storage::metrics: Metrics saved name=device:2f4e255fff1b8275 aggregation=MONTH
2023-05-25T09:42:17.083221Z ERROR up{deduplication_id=1d349916-3f38-4107-84db-89f4a0d258dc}: chirpstack::uplink::data: Handle uplink error error=LR-FHSS is not supported for downlink
2023-05-25T09:42:18.429187Z  INFO chirpstack::gateway::backend::mqtt: Message received from gateway region_config_id="eu868" topic="eu868/gateway/e45f01fffeb94911/event/up" qos=0 json=false
2023-05-25T09:42:18.635566Z ERROR chirpstack::uplink: Deduplication error error=No channel found for frequency: 868800000, dr: 7
2023-05-25T09:42:19.173592Z  INFO chirpstack::gateway::backend::mqtt: Message received from gateway region_config_id="eu868" topic="eu868/gateway/e45f01fffeb94911/event/up" qos=0 json=false
2023-05-25T09:42:19.380479Z  INFO up{deduplication_id=e253d31c-f68c-4c48-89d0-7f746748539f}: chirpstack::uplink: Uplink received m_type="ConfirmedDataUp"
2023-05-25T09:42:19.386897Z  WARN up{deduplication_id=e253d31c-f68c-4c48-89d0-7f746748539f}:data_up: chirpstack::uplink::data: No device-session exists for dev_addr dev_addr=02e43483
2023-05-25T09:42:22.305946Z  INFO chirpstack::gateway::backend::mqtt: Message received from gateway region_config_id="eu868" topic="eu868/gateway/e45f01fffeb94911/event/up" qos=0 json=false
2023-05-25T09:42:22.512260Z  INFO up{deduplication_id=ed01736a-d33d-47ca-94c7-eb337d5f8de2}: chirpstack::uplink: Uplink received m_type="ConfirmedDataUp"
2023-05-25T09:42:22.517786Z  WARN up{deduplication_id=ed01736a-d33d-47ca-94c7-eb337d5f8de2}:data_up: chirpstack::uplink::data: No device-session exists for dev_addr dev_addr=027b5553
2023-05-25T09:42:30.508081Z  INFO chirpstack::gateway::backend::mqtt: Message received from gateway region_config_id="eu868" topic="eu868/gateway/e45f01fffeb94911/event/up" qos=0 json=false
2023-05-25T09:42:30.714060Z  INFO up{deduplication_id=9d848f40-77db-409e-b135-e4cd0b83d52e}: chirpstack::uplink: Uplink received m_type="ConfirmedDataUp"
2023-05-25T09:42:30.720197Z  WARN up{deduplication_id=9d848f40-77db-409e-b135-e4cd0b83d52e}:data_up: chirpstack::uplink::data: No device-session exists for dev_addr dev_addr=02b74098
2023-05-25T09:42:33.186188Z  INFO chirpstack::gateway::backend::mqtt: Message received from gateway region_config_id="eu868" topic="eu868/gateway/e45f01fffeb94911/event/up" qos=0 json=false
2023-05-25T09:42:33.392215Z  INFO up{deduplication_id=d97f8bf9-07dc-4d6f-b3f2-8d9ffeb75b1a}: chirpstack::uplink: Uplink received m_type="ConfirmedDataUp"
2023-05-25T09:42:33.398261Z  WARN up{deduplication_id=d97f8bf9-07dc-4d6f-b3f2-8d9ffeb75b1a}:data_up: chirpstack::uplink::data: No device-session exists for dev_addr dev_addr=02e43483
2023-05-25T09:42:36.115404Z  INFO chirpstack::gateway::backend::mqtt: Message received from gateway region_config_id="eu868" topic="eu868/gateway/e45f01fffeb94911/event/stats" qos=0 json=false
2023-05-25T09:42:36.124039Z  INFO stats{gateway_id=e45f01fffeb94911}: chirpstack::storage::gateway: Gateway state updated gateway_id=e45f01fffeb94911
2023-05-25T09:42:36.127389Z  INFO stats{gateway_id=e45f01fffeb94911}: chirpstack::storage::metrics: Metrics saved name=gw:e45f01fffeb94911 aggregation=HOUR
2023-05-25T09:42:36.129971Z  INFO stats{gateway_id=e45f01fffeb94911}: chirpstack::storage::metrics: Metrics saved name=gw:e45f01fffeb94911 aggregation=DAY
2023-05-25T09:42:36.133183Z  INFO stats{gateway_id=e45f01fffeb94911}: chirpstack::storage::metrics: Metrics saved name=gw:e45f01fffeb94911 aggregation=MONTH
2023-05-25T09:42:36.292598Z  INFO chirpstack::gateway::backend::mqtt: Message received from gateway region_config_id="eu868" topic="eu868/gateway/e45f01fffeb94911/event/up" qos=0 json=false
2023-05-25T09:42:36.498819Z  INFO up{deduplication_id=e0e4538d-744b-457a-a172-a0392d19627e}: chirpstack::uplink: Uplink received m_type="ConfirmedDataUp"
2023-05-25T09:42:36.505516Z  WARN up{deduplication_id=e0e4538d-744b-457a-a172-a0392d19627e}:data_up: chirpstack::uplink::data: No device-session exists for dev_addr dev_addr=027b5553
2023-05-25T09:42:36.721017Z  INFO chirpstack::gateway::backend::mqtt: Message received from gateway region_config_id="eu868" topic="eu868/gateway/e45f01fffeb94911/event/up" qos=0 json=false
2023-05-25T09:42:36.927726Z  INFO up{deduplication_id=cce2fe00-0427-42cd-aa71-413a4ded166e}: chirpstack::uplink: Uplink received m_type="UnconfirmedDataUp"
2023-05-25T09:42:36.938720Z  INFO up{deduplication_id=cce2fe00-0427-42cd-aa71-413a4ded166e}:data_up: chirpstack::storage::device_session: Device-session saved dev_eui=2f4e255fff1b8275 dev_addr=01bbb2b2
2023-05-25T09:42:36.949175Z  INFO chirpstack::storage::device: Setting device lock dev_eui=2f4e255fff1b8275
2023-05-25T09:42:36.961100Z  INFO up{deduplication_id=cce2fe00-0427-42cd-aa71-413a4ded166e}:data_up: chirpstack::storage::device: Data-rate updated dev_eui=2f4e255fff1b8275 dr=0
2023-05-25T09:42:36.964325Z  INFO up{deduplication_id=cce2fe00-0427-42cd-aa71-413a4ded166e}:data_up: chirpstack::storage::device_gateway: Gateway rx-info saved dev_eui=2f4e255fff1b8275
2023-05-25T09:42:36.967074Z  INFO up{deduplication_id=cce2fe00-0427-42cd-aa71-413a4ded166e}:data_up: chirpstack::storage::device_session: Device-session saved dev_eui=2f4e255fff1b8275 dev_addr=01bbb2b2
2023-05-25T09:42:36.968957Z  INFO chirpstack::integration::mqtt: Publishing event topic=application/dd93f513-0d63-4fb3-9973-bcd75cda828a/device/2f4e255fff1b8275/event/up
2023-05-25T09:42:36.970511Z  INFO up{deduplication_id=cce2fe00-0427-42cd-aa71-413a4ded166e}:data_up: chirpstack::storage::metrics: Metrics saved name=device:2f4e255fff1b8275 aggregation=HOUR
2023-05-25T09:42:36.973635Z  INFO up{deduplication_id=cce2fe00-0427-42cd-aa71-413a4ded166e}:data_up: chirpstack::storage::metrics: Metrics saved name=device:2f4e255fff1b8275 aggregation=DAY
2023-05-25T09:42:36.976346Z  INFO up{deduplication_id=cce2fe00-0427-42cd-aa71-413a4ded166e}:data_up: chirpstack::storage::metrics: Metrics saved name=device:2f4e255fff1b8275 aggregation=MONTH

some lines that look suspicious to me:

2023-05-25T09:41:57.077974Z ERROR up{deduplication_id=05eba84b-f6b1-4f83-908b-6ba3fa128de1}: chirpstack::uplink::data: Handle uplink error error=LR-FHSS is not supported for downlink

and also

2023-05-25T09:41:57.077974Z ERROR up{deduplication_id=05eba84b-f6b1-4f83-908b-6ba3fa128de1}: chirpstack::uplink::data: Handle uplink error error=LR-FHSS is not supported for downlink
2023-05-25T09:42:04.631867Z ERROR chirpstack::uplink: Deduplication error error=No channel found for frequency: 868800000, dr: 7

[brocaar]

ERROR chirpstack::uplink: Deduplication error error=No channel found for frequency: 868800000, dr: 7

Are you sure the uplink is using a valid data-rate & channel combination? Would you mind sharing some of the UDP packets? I'm interested in the raw rxpk payload. See for some debug examples:

https://www.chirpstack.io/docs/guides/connect-gateway.html#received-uplink-1

[fdur]

I noticed a "similar" issue in my setup.
I'm also using UDP packet forwarder.
My device is able to join to chirpstack and is able to push data
When a downlink data is "scheduled" on chirpstack, my device can receive it only if a PULL_DATA and a PUSH_DATA are sent
If only a PULL_DATA is sent, the data (by PULL_RESP) is not received.
If only a PUSH_DATA is sent, the data (by PULL_RESP) is not received.
I'm using chirpstack v4 (docker git f691d6c) and I didn't tested v3

On my understanding, in such use case (a downlink message is created on chirpstack), the downlink message is sent to a device when this device sends a PULL_DATA message.