Unable to connect UE with Open5Gs after enabling TLS in the SBIs of each network function in Open5Gs

[timamghosh30]

Open5GS Release, Revision, or Tag: v2.7.1
Steps to reproduce

1. Kubernetes cluster (control plane with two worker nodes): https://github.com/theocld/rke2-vagrant
2. Deployed OPen5gs (v2.7.1) using helm chart: open5gs-5g helm.zip
   Namespace: edge
3. Deployed UERANSIM (v3.2.6) using helm chart:ueransim.zip
   Namespace: ue
   Use: open5gs-values.yaml

Logs:
Open5gs:logs-ogs.zip
UERANSIM: logs-ueransim.zip

Expected behaviour

UE is successfully registered and PDU session is created.

Observed Behaviour

AMF Log:
07/15 10:18:43.218: [amf] ERROR: [suci-0-999-70-0000-0-0-0000000001] Cannot receive SBI message (../src/amf/amf-sm.c:698)

07/15 10:18:43.218: [amf] WARNING: [suci-0-999-70-0000-0-0-0000000001] Registration reject [90] (../src/amf/nas-path.c:205)

07/15 10:18:44.219: [sbi] ERROR: Connection timer expired (../lib/sbi/client.c:605)

07/15 10:18:44.219: [sbi] WARNING: ogs_sbi_client_handler() failed [-3] (../lib/sbi/path.c:62)

07/15 10:18:53.232: [gmm] WARNING: Registration request (../src/amf/gmm-sm.c:1747)

07/15 10:18:53.232: [gmm] INFO: [suci-0-999-70-0000-0-0-0000000001] SUCI (../src/amf/gmm-handler.c:172)

NRF Log:
07/15 09:12:54.455: [sbi] ERROR: Connection timer expired (../lib/sbi/client.c:605)

07/15 09:12:54.455: [nrf] WARNING: client_notify_cb() failed [-3] (../src/nrf/sbi-path.c:138)

07/15 09:12:54.455: [sbi] ERROR: Connection timer expired (../lib/sbi/client.c:605)

07/15 09:12:54.456: [nrf] WARNING: client_notify_cb() failed [-3] (../src/nrf/sbi-path.c:138)

07/15 09:12:54.456: [sbi] ERROR: Connection timer expired (../lib/sbi/client.c:605)

07/15 09:12:54.456: [nrf] WARNING: client_notify_cb() failed [-3] (../src/nrf/sbi-path.c:138)

07/15 09:12:54.456: [sbi] ERROR: Connection timer expired (../lib/sbi/client.c:605)

07/15 09:12:54.456: [nrf] WARNING: client_notify_cb() failed [-3] (../src/nrf/sbi-path.c:138)

SCP Log:
07/15 09:12:40.368: [sbi] INFO: RST_STREAM received: stream_id=19 (../lib/sbi/nghttp2-server.c:1230)

07/15 09:13:21.963: [sbi] INFO: RST_STREAM received: stream_id=21 (../lib/sbi/nghttp2-server.c:1230)

07/15 09:13:21.963: [sbi] ERROR: on_stream_close_callback() failed (5:STREAM_CLOSED) (../lib/sbi/nghttp2-server.c:1262)

07/15 09:13:21.964: [sbi] ERROR: Connection timer expired (../lib/sbi/client.c:605)

07/15 09:13:21.964: [scp] WARNING: response_handler() failed [-3] (../src/scp/sbi-path.c:627)

07/15 09:13:21.964: [sbi] ERROR: stream has already been removed (../lib/sbi/nghttp2-server.c:585)

07/15 09:13:42.164: [sbi] INFO: RST_STREAM received: stream_id=27 (../lib/sbi/nghttp2-server.c:1230)

07/15 09:13:42.164: [sbi] ERROR: on_stream_close_callback() failed (5:STREAM_CLOSED) (../lib/sbi/nghttp2-server.c:1262)

07/15 09:13:42.165: [sbi] ERROR: Connection timer expired (../lib/sbi/client.c:605)

UE Log:
[2024-07-15 09:50:45.527] [rrc] [info] UE switches to state [RRC-IDLE]

[2024-07-15 09:50:45.527] [nas] [info] UE switches to state [CM-IDLE]

[2024-07-15 09:50:55.537] [nas] [debug] NAS timer[3511] expired [1]

[2024-07-15 09:50:55.537] [nas] [debug] Initial registration required due to [T3511-EXPIRY-IN-ATT-REG]

[2024-07-15 09:50:55.537] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig]

[2024-07-15 09:50:55.537] [nas] [debug] Sending Initial Registration

[2024-07-15 09:50:55.537] [nas] [info] UE switches to state [MM-REGISTER-INITIATED]

[2024-07-15 09:50:55.538] [rrc] [debug] Sending RRC Setup Request

[2024-07-15 09:50:55.540] [rrc] [info] RRC connection established

[2024-07-15 09:50:55.540] [rrc] [info] UE switches to state [RRC-CONNECTED]

[2024-07-15 09:50:55.540] [nas] [info] UE switches to state [CM-CONNECTED]

[2024-07-15 09:51:00.706] [nas] [error] Initial Registration failed [PAYLOAD_NOT_FORWARDED]

[2024-07-15 09:51:00.706] [nas] [debug] Handling Registration Reject abnormal case

[2024-07-15 09:51:00.706] [nas] [info] UE switches to state [MM-DEREGISTERED/ATTEMPTING-REGISTRATION]

[2024-07-15 09:51:05.548] [nas] [warning] Registration Reject ignored since the MM state is not MM_REGISTERED_INITIATED

[2024-07-15 09:51:05.548] [nas] [warning] Sending MM Status with cause [MESSAGE_NOT_COMPATIBLE_WITH_PROTOCOL_STATE]

[2024-07-15 09:51:11.554] [nas] [debug] NAS timer[3511] expired [1]

[2024-07-15 09:51:11.554] [nas] [debug] Initial registration required due to [T3511-EXPIRY-IN-ATT-REG]

[2024-07-15 09:51:11.554] [nas] [debug] Sending Initial Registration

[2024-07-15 09:51:11.554] [nas] [info] UE switches to state [MM-REGISTER-INITIATED]

[2024-07-15 09:51:21.569] [nas] [error] Initial Registration failed [PAYLOAD_NOT_FORWARDED]

[2024-07-15 09:51:21.569] [nas] [debug] Handling Registration Reject abnormal case

[2024-07-15 09:51:21.569] [nas] [info] UE switches to state [MM-DEREGISTERED/ATTEMPTING-REGISTRATION]

[2024-07-15 09:51:31.574] [nas] [debug] NAS timer[3511] expired [1]

[2024-07-15 09:51:31.574] [nas] [debug] Initial registration required due to [T3511-EXPIRY-IN-ATT-REG]

[2024-07-15 09:51:31.574] [nas] [debug] Sending Initial Registration

[2024-07-15 09:51:31.574] [nas] [info] UE switches to state [MM-REGISTER-INITIATED]

[2024-07-15 09:51:41.588] [nas] [error] Initial Registration failed [PAYLOAD_NOT_FORWARDED]

[2024-07-15 09:51:41.588] [nas] [debug] Handling Registration Reject abnormal case

[2024-07-15 09:51:41.588] [nas] [info] UE switches to state [MM-DEREGISTERED/ATTEMPTING-REGISTRATION]

[2024-07-15 09:51:51.594] [nas] [debug] NAS timer[3511] expired [1]

[2024-07-15 09:51:51.594] [nas] [debug] Initial registration required due to [T3511-EXPIRY-IN-ATT-REG]

[2024-07-15 09:51:51.594] [nas] [debug] Sending Initial Registration

[2024-07-15 09:51:51.594] [nas] [info] UE switches to state [MM-REGISTER-INITIATED]

[2024-07-15 09:52:01.609] [nas] [error] Initial Registration failed [PAYLOAD_NOT_FORWARDED]

[2024-07-15 09:52:01.609] [nas] [debug] Handling Registration Reject abnormal case

[2024-07-15 09:52:01.609] [nas] [info] UE switches to state [MM-DEREGISTERED/ATTEMPTING-REGISTRATION]

[2024-07-15 10:04:02.327] [nas] [debug] NAS timer[3502] expired [1]

[2024-07-15 10:04:02.327] [nas] [debug] Initial registration required due to [T3502-EXPIRY-IN-ATT-REG]

[2024-07-15 10:04:02.327] [nas] [debug] Sending Initial Registration

[2024-07-15 10:04:02.327] [nas] [info] UE switches to state [MM-REGISTER-INITIATED]

[2024-07-15 10:04:12.334] [nas] [error] Initial Registration failed [PAYLOAD_NOT_FORWARDED]

[2024-07-15 10:04:12.334] [nas] [debug] Handling Registration Reject abnormal case

[2024-07-15 10:04:12.334] [nas] [info] UE switches to state [MM-DEREGISTERED/ATTEMPTING-REGISTRATION]

[2024-07-15 10:04:22.347] [nas] [debug] NAS timer[3511] expired [1]

[2024-07-15 10:04:22.347] [nas] [debug] Initial registration required due to [T3511-EXPIRY-IN-ATT-REG]

[2024-07-15 10:04:22.347] [nas] [debug] Sending Initial Registration

[2024-07-15 10:04:22.347] [nas] [info] UE switches to state [MM-REGISTER-INITIATED]

[2024-07-15 10:04:32.352] [nas] [error] Initial Registration failed [PAYLOAD_NOT_FORWARDED]

Other NFs show no abnormal behavior
Problem occurs when the UE is trying to connect the core network

eNodeB/gNodeB : UERANSIM v3.2.6
UE Models and versions: UERANSIM v3.2.6