e2term closes with error message "Error binding. Bad file descriptor "

[vaguiarc]

Hello,

I'm running the ./setup-ric-bronze.sh script. The container e2term is created but closes shortly after with the error message: Error binding. Bad file descriptor.

The output from the command docker logs e2term is:

environments service name is _SERVICE_HOST
service ip is 
sed: cannot rename /opt/e2/config/sedkApwDE: Device or resource busy
sed: cannot rename /opt/e2/config/sedFQDPyE: Device or resource busy
nano=38000
loglevel=debug
volume=log
#the key name of the environment holds the local ip address
#ip address of the E2T in the RMR
local-ip=10.0.2.10
#trace is start, stop
trace=start
external-fqdn=e2t.com
#put pointer to the key that point to pod name
pod_name=E2TERM_POD_NAME
sctp-port=36422
{"ts":1720114853775,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"line = nano=38000 "}
{"ts":1720114853775,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"entry = nano value = 38000"}
{"ts":1720114853775,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"line = loglevel=debug "}
{"ts":1720114853775,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"entry = loglevel value = debug"}
{"ts":1720114853775,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"line = volume=log "}
{"ts":1720114853775,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"entry = volume value = log"}
{"ts":1720114853775,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"line = local-ip=10.0.2.10 "}
{"ts":1720114853775,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"entry = local-ip value = 10.0.2.10"}
{"ts":1720114853775,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"line = trace=start "}
{"ts":1720114853775,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"entry = trace value = start"}
{"ts":1720114853775,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"line = external-fqdn=e2t.com "}
{"ts":1720114853775,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"entry = external-fqdn value = e2t.com"}
{"ts":1720114853775,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"line = pod_name=E2TERM_POD_NAME "}
{"ts":1720114853775,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"entry = pod_name value = E2TERM_POD_NAME"}
{"ts":1720114853775,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"line = sctp-port=36422 "}
{"ts":1720114853775,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"entry = sctp-port value = 36422"}
{"ts":1720114853775,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"Trace set to: start"}
{"ts":1720114853775,"crit":"INFO","id":"E2Terminator","mdc":{"pod name":"e2term","my ip":"10.0.2.10","tmpLogFilespec":"log/tmp/E2Term_%Y-%m-%d_%H-%M-%S.%N.tmpStr","volume":"log","LogLevel":"4","RMR Port":"38000"},"msg":"running parameters for instance : {\"address\": \"10.0.2.10:38000\",\"fqdn\": \"e2t.com\",\"pod_name\": \"e2term\"}"}
1720114853 23/RMR [INFO] ric message routing library on SI95/g mv=3 flg=00 (d762b36 4.0.2 built: Apr 27 2020)
{"ts":1720114853778,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"We are after RMR INIT wait for RMR_Ready"}
{"ts":1720114853778,"crit":"INFO","id":"E2Terminator","mdc":{},"msg":"RMR running"}
{"ts":1720114853779,"crit":"ERROR","id":"E2Terminator","mdc":{},"msg":"Error binding. Bad file descriptor "}

The other containers starts correctly. The output of the command docker ps -a:

CONTAINER ID   IMAGE               COMMAND                  CREATED         STATUS                       PORTS      NAMES
fb588d7a6deb   e2term:bronze       "sh -c ./startup.sh"     7 minutes ago   Exited (255) 7 minutes ago              e2term
689d4d73a788   e2mgr:bronze        "sh -c './main  -por…"   7 minutes ago   Up 7 minutes                 3800/tcp   e2mgr
13b4189d3b1b   e2rtmansim:bronze   "/bin/sh -c 'exec ./…"   7 minutes ago   Up 7 minutes                            e2rtmansim
5bc9f364315d   dbaas:bronze        "redis-server"           7 minutes ago   Up 7 minutes                 6379/tcp   db

Could you help me on this problem?

[leonardobonati]

Hello,

Could you please describe the setup that you are using? E.g., servers and all. This would help us to pinpoint the issue.

Thank you

[vaguiarc]

I'm following this tutorial: https://openrangym.com/tutorials/ns-o-ran

Its instructs to clone this repository, run the script ./import-wines-images.sh to import all the dockers containers and images, and then run the script ./setup-ric-bronze.sh to set up and launch. I didn't change any setup files, they remain the same as they are in the repository.

I'm not really familiar with the tool colosseum-near-rt-ric but I followed the same steps in that tutorial in another PC, and the e2term was launched.

[leonardobonati]

Hello,

By looking at your logs, there might be some disk space issues, e.g., sed: cannot rename /opt/e2/config/sedkApwDE: Device or resource busy. Could you please confirm that you have enough disk space on the PCs where you are trying to run this?

[vaguiarc]

Hi,
There are 107 GB free in the disk

When I type free it gives me this output:

               total        used        free      shared  buff/cache   available
Mem:        15756252     3795500     2289208      769672     9671544    10854924
Swap:        3999740           0     3999740

[leonardobonati]

Hello, that looks like RAM memory to me, not disk space.

[vaguiarc]

I checked the disk memory through the Disks in the partition that has the linux:

The second one, you right, it's the RAM memory. Sorry for my mistake. The right command is df -T, and the result is:

Filesystem     Type     1K-blocks      Used Available Use% Mounted on
tmpfs          tmpfs      1575628      4292   1571336   1% /run
/dev/nvme0n1p3 ext4     240605740 136293736  92017092  60% /
tmpfs          tmpfs      7878124    287556   7590568   4% /dev/shm
tmpfs          tmpfs         5120         4      5116   1% /run/lock
efivarfs       efivarfs       374       301        69  82% /sys/firmware/efi/efivars
tmpfs          tmpfs      7878124         0   7878124   0% /run/qemu
/dev/nvme0n1p1 vfat        486456     77172    409284  16% /boot/efi
tmpfs          tmpfs      1575624       172   1575452   1% /run/user/1595471630

[leonardobonati]

Is the SCTP protocol enabled on your machine? This is used by the e2term container, which might complain if not enabled.

[vaguiarc]

It appears that it is enabled.

I ran the sctp_test using IP address of brric, and it worked:
The command used was sctp_test -H 10.0.2.1 -P 5001 -l and the result was:

local:addr=10.0.2.1, port=5001, family=2
seed = 1720787735

Starting tests...
	socket(SOCK_SEQPACKET, IPPROTO_SCTP)  ->  sk=3
	bind(sk=3, [a:10.0.2.1,p:5001])  --  attempt 1/10
	listen(sk=3,backlog=100)
Server: Receiving packets.
	recvm
sg(sk=3) 

When I used a different IP that is not assigned to any interface, it didn't work, and the message was similarly to the error message from e2term docker log:

local:addr=192.168.1.10, port=5001, family=2
seed = 1720787961

Starting tests...
	socket(SOCK_SEQPACKET, IPPROTO_SCTP)  ->  sk=3
	bind(sk=3, [a:192.168.1.10,p:5001])  --  attempt 1/10


		***bind: can not bind to 192.168.1.10:5001: Cannot assign requested address ****

I believe that the IP address that the e2term is trying to connect to at moment of the error message is not IP address of brric. Could that be the issue?

[leonardobonati]

That could be the issue. The setup script of the RIC is supposed to setup some iptable rules to forward the traffic from the base station E2 to the e2term container of the RIC. This might not work if done outside Colosseum and might need to be adjusted manually (or try to point the E2 of the base station toward the e2term container directly, if supported in your setup).

[vaguiarc]

I have the rric IP address, and I believe the e2term IP address is set the IP address correctly since I haven't change anything. Do you have any suggestion of how can I make this tutorial work?

[Thecave3]

The ns-O-RAN tutorial should work with no changes required. Are you sure you are using the correct branch?
Can you share the logs of ns-O-RAN?

[vaguiarc]

I didn't get to the part of ns-O-RAN. It fails when I run ./setup-ric-bronze.sh; the e2term container exits