Skip to content

Troubleshooting

Jump to bottom
Larry Peterson edited this page Mar 12, 2024 · 33 revisions

This page as moved to the Aether Wiki

General troubleshooting hints are included in the Aether OnRamp Guide, but some of the issues people report are due to the specifics of either their local environment or the hardware they are using. This page summarizes such issues and how they might be addressed.

Known Issues

  1. AMP (specifically, ROC) does not cleanly uninstall/reinstall. The problem is being worked on, but until it is resolved, the only workaround is to do a fresh reinstall of Kubernetes.

  2. gNBsim will sometimes report 4 of 5 UEs successfully connecting. This is likely a race condition that has been fixed in a development version of the webui image, and is not indicative of an OnRamp config/install problem. The following edit will work around the problem until a new version of SD-Core is released:

diff --git a/roles/core/templates/sdcore-5g-values.yaml b/roles/core/templates/sdcore-5g-values.yaml
index 07ac4f7..c8cb8c0 100644
--- a/roles/core/templates/sdcore-5g-values.yaml
+++ b/roles/core/templates/sdcore-5g-values.yaml
@@ -17,8 +17,8 @@ omec-control-plane:
   enable5G: true
   images:
     repository: "registry.opennetworking.org/docker.io/"
-    #tags:
-    #  amf: <amf image tag>
+    tags:
+      webui: gabearrobo/5gc-webui:0.0.1-dev
     # refer to above Helm Chart to add other NF images
  1. When connecting multiple UEs to a physical gNB (e.g., five moto g 5G phones to MOSO CANOPY), it has been reported that only four connect at a time. You have to turn one off to connect a new one. This appears to be an issue with an old version of the gNB software that can be corrected with an update.

Frequency Bands / UEs / PLMNs

The OnRamp guide recommends the MOSO CANOPY 5G INDOOR SMALL CELL (Model SCD5164), which is also known as the SERCOMM Bridgestone gNB (SCE5164-B78/-B48/-B77). The product data sheet implies support for multiple frequencies, including n78 and n48, but individual devices may be limited to just one (e.g., n48, as shown below).

Not all UEs support all bands, and even when they do, there appear to be other limitations. For example, there is a limited set of UE’s that support the CBRS PLMN (315010). The following is the list of UEs that should work for n48, with the first being the only one confirmed to date:

  • Moto G 5G (confirmed working for n48 with PLMN=00101)

  • Oneplus 9

  • Google Pixel 7 (see below)

  • Samsung Galaxy S22 Ultra

  • iPhone 13 with IOS 17

  • Asus ROG phone 6

With another gNB model that uses n78, we have verified PLMN = 00101 works for the following two Android Phones:

  • Google Pixel 4
  • Moto G 5G (same model as above)

With Pixel phones, it is best to use PLMN 00101; SERCOMM is seeing issues with 315010.

Programming SIMs

The recommended OYEITIMES MCR3516 SIM Card Writer is for 4G, but can also be used for 5G (albeit without the full set of service options). It requires running the SIM App on a Windows machine. After inserting a blank card, click the "Read Card" button at the top. This will fill many of the fields with their initial/default values. The following describes what fields you need to modify, resulting in the set of final values shown in the following screenshot. When you are done modifying the SIM, click the "Write Card" button to write your changes to the card.

sim-card-annotated

Many of the default settings are "good enough" to get started, but will eventually need to be changed if you plan to deploy a secure/production-ready network. For example, the green shaded boxes in the Common Parameters panel (e.g., PIN1, PUK1, PIN2, PUK2, ADM) are keys used to secure the SIM. The default settings will work, but will not protect the SIM from malicious actors.

The most important edits you need to make are to the fields annotated in red. They need to match values defined in the SD-Core values override file. This includes the 15-digit IMSI, along with the KI and OPC fields. The screenshot shows the 18-digit IMSI as being set as well, but we are using 15-digit IMSIs. The screenshot also shows the GSM Parameters being set (using the same values in the LTE Parameters panel), but this is not necessary since 5G uses only the LTE parameters (but there is no harm in setting them).

The fields annotated in blue are pre-filled on the card. The Type and ATR fields should remain unchanged, but you may want to modify the ICCID field. This is a unique id assigned to each card; whatever unique value(s) you assign to a set of SIM cards should work.

Other Options

An alternative approach (not yet verified to work with Aether) is to follow the procedure outlined by the srsRAN project, which includes: (1) an open source program (pysim) to program SIMs, and (2) a set of pre-configured (but reprogrammable) SIM cards from Sysmocom.

Configuring MOSO CANOPY 5G INDOOR SMALL CELL

We recommend the following recipe to get the MosoLab gNB running the first time. Most out-of-the-box settings are fine. If you've already changed some of them and want to start over, visit:

Support > Configuration > Factory Reset

You'll need to "Apply" and reboot.

Make the following edits to the factory defaults. For each page, be sure to "Apply" and just dismiss the pop-up telling you to reboot. You only need to reboot once after all of the changes are made.

Settings > WAN

IPv4 Connection Type = Static
IP Address = Whatever DHCP assigns
Gateway = IP addr of Aether server

Settings > 5GC

PLMN = 00101
AMF Address = IP addr of Aether server

Settings > GPS

Enable GPS = 0

Settings > NTP Server

Default is ok

Settings > Sync Settings

Enable PTP = 0
Sync Mode = FREE_RUNNING

Note that the above assumes GPS and PTP are disabled, which is why Setting > Sync_Settings > Sync_Mode should be set to FREE_RUNNING. If you connect the GPS antenna and enable GPS, then Setting > Sync_Settings > Sync_Mode should be set to TIME (this is the factory default).

Note that the above also works for DHCP-assigned IP address and Gateway. See the next section for more information about of how to manage the WAN settings in this case.

You may also want to verify the following configuration parameters are all disabled (or set to 0):

  • Settings > Initial SecGW Server
  • Settings > SecGW Server
  • Settings > TR09 Management > Initial HeMS Server
  • Settings > O1 Management
  • Settings > SAS

SAS eventually needs to enabled (and configured), but for limited/indoor settings, this step may come after an initial boot-up.

When done with all the changes, select

Support > Configuration > Reboot

Verify that your configuration changes were applied once the radio comes back up.

Default GW

The above Gateway configuration assumes DHCP assigns an IP address to your gNB, but that the address is not likely to change, making it safe to treat it as static. The following elaborates on this.

When configuring the WAN (via the LAN), start with DHCP enabled. Note the IP address the gNB has been assigned, and then after disconnecting from the LAN, connect to the GUI via this address. You will be on the same L2 subnet as the Aether server, which you should be able to ping using the gNB’s diagnostic tool.

The default gateway DHCP returns does not know how to route packets to the UPF. To fix this, modify the WAN settings to use a static IP, but with the DHCP-provided IP used as the static address. Also set the default gateway to the IP address of your Aether server; it is configured to route GTP-tunneled packets to the UPF.

Your Aether server also needs to be configured to forward IP packets between the gNB and the Internet because the gNB needs to be able to reach NTP (and other) servers. For instructions on how to do that, see (for example) the following guide. Note that enabling IP forwarding is often not sufficient, due to how Kubernetes programs iptables. If you type

$ sudo iptables -L -v -n | grep FORWARD

and see the FORWARD chain is set to DROP, then you need to reset it to ACCEPT as follows:

$ sudo iptables -P FORWARD ACCEPT

You can verify that your server is successfully forwarding traffic by running the PING or TRACEROUTE diagnostic tool on the gNB.

MongoDB CrashLoopBackOff

MongoDB not successfully starting (getting stuck in CrashLoopBackOff state) may be due to an older CPU that does not support the AVX instruction required by MongoDB 5.0+. See the following post to the MongoDB Developer Community. If running on a VM (e.g., selecting host processor on Proxmox), keep in mind that you may be assigned a problematic CPU.

Source/Destination Checking on AWS

When running gNBsim and SD-Core on separate servers in AWS, if summary.log reports "UEs Passed: 0, UEs Failed: 5", then it might be that AWS settings are interfering with user plane traffic. Digging deeper, when gnbsim.log shows PDU session establishment is successful but data packets are not transferred between gNBsim and the SD-Core, then User Plane routing is the likely culprit. If you have verified the routes are installed correctly, then it may be the case that AWS has source/destination checking enabled. To disable source/destination checking:

  1. Open the Amazon EC2 console.
  2. In the navigation pane, choose Instances.
  3. Select the instance (you will need to repeat for both the Ansible master and worker nodes).
  4. Choose Actions, Networking, Change source/destination check.
  5. For source/destination checking, select Stop.
  6. Click Save.

Network Service on Ubuntu

Aether assumes Ubuntu Server, in part because Ubuntu Desktop typically disables systemd.networkd and does not have netplan installed. This means Ansible is unable to find the .network file, which results in a failed task in the Ansible playbook for the route role when installing 5gc. If Ubuntu Desktop is your only option, one workaround is to install netplan with

$ sudo apt install netplan.io

and then manually add file /etc/systemd/network/05-netplan-eno1.network with content:

[Match]
Name=eno1

[Network]
DHCP=yes
IPForward=yes

NM_CONTROLLED=no

Then execute

$ sudo systemctl restart systemd.networkd

That will help Aether find and edit network files.

$ sudo systemctl enable systemd.networkd

will enable the service if it’s not enabled by default.

But it’s not clear this specific .network file is appropriate for all scenarios; it might mess up something else. The following is alternative content for that file (taken from a working system that had systemd.networkd installed by default):

[Match]
Name=eno1

[Network]
DHCP=ipv4
LinkLocalAddressing=ipv6

[DHCP]
RouteMetric=100
UseMTU=true

You might also consider changes to the netplan config file that would have generated the .network file in the first place. From the same working system as above:

$ ls /etc/netplan
./  ../  01-netcfg.yaml
$ cat /etc/netplan/01-netcfg.yaml
# This file describes the network interfaces available on your system
# For more information, see netplan(5).
network:
  version: 2
  renderer: networkd
  ethernets:
    eno1:
      dhcp4: yes

Bottomline: You need to identify the least intrusive solution, which may be challenging if you end up mixing management directives between systemd.networkd and the Network Manager. The best approach is to start with Ubuntu Server in the first place.

Clone this wiki locally