Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Free5gc NRF is waiting for MongoDB indefinitely #81

Closed
Zhenye-Na opened this issue Mar 20, 2023 · 2 comments
Closed

Free5gc NRF is waiting for MongoDB indefinitely #81

Zhenye-Na opened this issue Mar 20, 2023 · 2 comments

Comments

@Zhenye-Na
Copy link

Also raised a similar issue here free5gc/free5gc#436

I am using helm to install free5gc-nrf chart. The MongoDB dependency could be instantiated but nrf is waiting for MongoDB forever. But I verified that mongodb is up. So not sure how does this happen.

kubectl get pods

NAME                                   READY   STATUS     RESTARTS   AGE
mongodb-0                              1/1     Running    0          9m14s
nrf-free5gc-nrf-nrf-79d7d9b788-cmghx   0/1     Init:0/1   0          9m14s

If I checked the log of wait-mongo container of nrf-free5gc-nrf-nrf-79d7d9b788-cmghx

kubectl logs nrf-free5gc-nrf-nrf-79d7d9b788-cmghx -c wait-mongo
nc: bad address 'mongodb:27017'
waiting for the MongoDB
nc: bad address 'mongodb:27017'
waiting for the MongoDB
....

Meanwhile I checked the log of MongoDB-0 pod

kubectl logs mongodb-0
mongodb 04:34:33.79
mongodb 04:34:33.80 Welcome to the Bitnami mongodb container
mongodb 04:34:33.80 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mongodb
mongodb 04:34:33.80 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mongodb/issues
mongodb 04:34:33.80
mongodb 04:34:33.80 INFO  ==> ** Starting MongoDB setup **
mongodb 04:34:33.82 INFO  ==> Validating settings in MONGODB_* env vars...
mongodb 04:34:33.82 WARN  ==> You set the environment variable ALLOW_EMPTY_PASSWORD=yes. For safety reasons, do not use this flag in a production environment.
mongodb 04:34:33.84 INFO  ==> Initializing MongoDB...
mongodb 04:34:33.86 INFO  ==> Deploying MongoDB with persisted data...
mongodb 04:34:33.87 INFO  ==> ** MongoDB setup finished! **

mongodb 04:34:33.89 INFO  ==> ** Starting MongoDB **

{"t":{"$date":"2023-03-18T04:34:33.934+00:00"},"s":"I",  "c":"CONTROL",  "id":20698,   "ctx":"main","msg":"***** SERVER RESTARTED *****"}
{"t":{"$date":"2023-03-18T04:34:33.939+00:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2023-03-18T04:34:33.944+00:00"},"s":"W",  "c":"ASIO",     "id":22601,   "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2023-03-18T04:34:33.946+00:00"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
{"t":{"$date":"2023-03-18T04:34:33.947+00:00"},"s":"I",  "c":"STORAGE",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/bitnami/mongodb/data/db","architecture":"64-bit","host":"mongodb-0"}}
{"t":{"$date":"2023-03-18T04:34:33.947+00:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"4.4.4","gitVersion":"8db30a63db1a9d84bdcad0c83369623f708e0397","openSSLVersion":"OpenSSL 1.1.1d  10 Sep 2019","modules":[],"allocator":"tcmalloc","environment":{"distmod":"debian10","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2023-03-18T04:34:33.947+00:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"PRETTY_NAME=\"Debian GNU/Linux 10 (buster)\"","version":"Kernel 5.4.226-129.415.amzn2.x86_64"}}}
{"t":{"$date":"2023-03-18T04:34:33.947+00:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"config":"/opt/bitnami/mongodb/conf/mongodb.conf","net":{"bindIp":"*","ipv6":false,"port":27017,"unixDomainSocket":{"enabled":true,"pathPrefix":"/opt/bitnami/mongodb/tmp"}},"processManagement":{"fork":false,"pidFilePath":"/opt/bitnami/mongodb/tmp/mongodb.pid"},"security":{"authorization":"disabled"},"setParameter":{"enableLocalhostAuthBypass":"true"},"storage":{"dbPath":"/bitnami/mongodb/data/db","directoryPerDB":false,"journal":{"enabled":true}},"systemLog":{"destination":"file","logAppend":true,"logRotate":"reopen","path":"/opt/bitnami/mongodb/logs/mongodb.log","quiet":false,"verbosity":0}}}}
{"t":{"$date":"2023-03-18T04:34:33.951+00:00"},"s":"I",  "c":"STORAGE",  "id":22270,   "ctx":"initandlisten","msg":"Storage engine to use detected by data files","attr":{"dbpath":"/bitnami/mongodb/data/db","storageEngine":"wiredTiger"}}
{"t":{"$date":"2023-03-18T04:34:33.951+00:00"},"s":"I",  "c":"STORAGE",  "id":22297,   "ctx":"initandlisten","msg":"Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem","tags":["startupWarnings"]}
{"t":{"$date":"2023-03-18T04:34:33.951+00:00"},"s":"I",  "c":"STORAGE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=7399M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],"}}
{"t":{"$date":"2023-03-18T04:34:34.563+00:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1679114074:563496][1:0x7f24214f2140], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 1 through 2"}}
{"t":{"$date":"2023-03-18T04:34:34.646+00:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1679114074:645988][1:0x7f24214f2140], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 2 through 2"}}
{"t":{"$date":"2023-03-18T04:34:34.722+00:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1679114074:722877][1:0x7f24214f2140], txn-recover: [WT_VERB_RECOVERY | WT_VERB_RECOVERY_PROGRESS] Main recovery loop: starting at 1/26624 to 2/256"}}
{"t":{"$date":"2023-03-18T04:34:34.816+00:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1679114074:816017][1:0x7f24214f2140], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 1 through 2"}}
{"t":{"$date":"2023-03-18T04:34:34.871+00:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1679114074:871564][1:0x7f24214f2140], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 2 through 2"}}
{"t":{"$date":"2023-03-18T04:34:34.918+00:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1679114074:918108][1:0x7f24214f2140], txn-recover: [WT_VERB_RECOVERY | WT_VERB_RECOVERY_PROGRESS] Set global recovery timestamp: (0, 0)"}}
{"t":{"$date":"2023-03-18T04:34:34.918+00:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1679114074:918183][1:0x7f24214f2140], txn-recover: [WT_VERB_RECOVERY | WT_VERB_RECOVERY_PROGRESS] Set global oldest timestamp: (0, 0)"}}
{"t":{"$date":"2023-03-18T04:34:34.942+00:00"},"s":"I",  "c":"STORAGE",  "id":4795906, "ctx":"initandlisten","msg":"WiredTiger opened","attr":{"durationMillis":991}}
{"t":{"$date":"2023-03-18T04:34:34.942+00:00"},"s":"I",  "c":"RECOVERY", "id":23987,   "ctx":"initandlisten","msg":"WiredTiger recoveryTimestamp","attr":{"recoveryTimestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2023-03-18T04:34:34.943+00:00"},"s":"I",  "c":"STORAGE",  "id":4366408, "ctx":"initandlisten","msg":"No table logging settings modifications are required for existing WiredTiger tables","attr":{"loggingEnabled":true}}
{"t":{"$date":"2023-03-18T04:34:34.943+00:00"},"s":"I",  "c":"STORAGE",  "id":22262,   "ctx":"initandlisten","msg":"Timestamp monitor starting"}
{"t":{"$date":"2023-03-18T04:34:34.947+00:00"},"s":"I",  "c":"STORAGE",  "id":20536,   "ctx":"initandlisten","msg":"Flow Control is enabled on this deployment"}
{"t":{"$date":"2023-03-18T04:34:34.947+00:00"},"s":"I",  "c":"FTDC",     "id":20625,   "ctx":"initandlisten","msg":"Initializing full-time diagnostic data capture","attr":{"dataDirectory":"/bitnami/mongodb/data/db/diagnostic.data"}}
{"t":{"$date":"2023-03-18T04:34:34.949+00:00"},"s":"I",  "c":"NETWORK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"/opt/bitnami/mongodb/tmp/mongodb-27017.sock"}}
{"t":{"$date":"2023-03-18T04:34:34.949+00:00"},"s":"I",  "c":"NETWORK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"0.0.0.0"}}
{"t":{"$date":"2023-03-18T04:34:34.949+00:00"},"s":"I",  "c":"NETWORK",  "id":23016,   "ctx":"listener","msg":"Waiting for connections","attr":{"port":27017,"ssl":"off"}}
{"t":{"$date":"2023-03-18T04:34:42.980+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:38498","connectionId":1,"connectionCount":1}}
{"t":{"$date":"2023-03-18T04:34:42.980+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn1","msg":"client metadata","attr":{"remote":"127.0.0.1:38498","client":"conn1","doc":{"application":{"name":"MongoDB Shell"},"driver":{"name":"MongoDB Internal Client","version":"4.4.4"},"os":{"type":"Linux","name":"PRETTY_NAME=\"Debian GNU/Linux 10 (buster)\"","architecture":"x86_64","version":"Kernel 5.4.226-129.415.amzn2.x86_64"}}}}
{"t":{"$date":"2023-03-18T04:34:42.984+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn1","msg":"Connection ended","attr":{"remote":"127.0.0.1:38498","connectionId":1,"connectionCount":0}}
{"t":{"$date":"2023-03-18T04:34:52.979+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:37554","connectionId":2,"connectionCount":1}}
{"t":{"$date":"2023-03-18T04:34:52.980+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn2","msg":"client metadata","attr":{"remote":"127.0.0.1:37554","client":"conn2","doc":{"application":{"name":"MongoDB Shell"},"driver":{"name":"MongoDB Internal Client","version":"4.4.4"},"os":{"type":"Linux","name":"PRETTY_NAME=\"Debian GNU/Linux 10 (buster)\"","architecture":"x86_64","version":"Kernel 5.4.226-129.415.amzn2.x86_64"}}}}
{"t":{"$date":"2023-03-18T04:34:52.984+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn2","msg":"Connection ended","attr":{"remote":"127.0.0.1:37554","connectionId":2,"connectionCount":0}}
{"t":{"$date":"2023-03-18T04:35:03.010+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:36944","connectionId":3,"connectionCount":1}}
{"t":{"$date":"2023-03-18T04:35:03.011+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn3","msg":"client metadata","attr":{"remote":"127.0.0.1:36944","client":"conn3","doc":{"application":{"name":"MongoDB Shell"},"driver":{"name":"MongoDB Internal Client","version":"4.4.4"},"os":{"type":"Linux","name":"PRETTY_NAME=\"Debian GNU/Linux 10 (buster)\"","architecture":"x86_64","version":"Kernel 5.4.226-129.415.amzn2.x86_64"}}}}
{"t":{"$date":"2023-03-18T04:35:03.014+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn3","msg":"Connection ended","attr":{"remote":"127.0.0.1:36944","connectionId":3,"connectionCount":0}}
{"t":{"$date":"2023-03-18T04:35:13.023+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:59666","connectionId":4,"connectionCount":1}}
{"t":{"$date":"2023-03-18T04:35:13.023+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn4","msg":"client metadata","attr":{"remote":"127.0.0.1:59666","client":"conn4","doc":{"application":{"name":"MongoDB Shell"},"driver":{"name":"MongoDB Internal Client","version":"4.4.4"},"os":{"type":"Linux","name":"PRETTY_NAME=\"Debian GNU/Linux 10 (buster)\"","architecture":"x86_64","version":"Kernel 5.4.226-129.415.amzn2.x86_64"}}}}
@Zhenye-Na Zhenye-Na mentioned this issue Mar 20, 2023
Closed
@Zhenye-Na
Copy link
Author

Added more findings:

It seems like the NRF chart works when:

  1. AWS EKS Cluster has only instances coming from EKS Managed Node Group
  2. AWS EKS Cluster has only instances coming from EKS Self-managed Node Group

If there are multiple instances, and the source of them is mix-matching. The NRF helm chart is installed and MongoDB is running, the wait-mongo container of NRF could not find the the Mongo pods, with same logs found above

kubectl logs nrf-free5gc-nrf-nrf-79d7d9b788-cmghx -c wait-mongo
nc: bad address 'mongodb:27017'
waiting for the MongoDB
nc: bad address 'mongodb:27017'
waiting for the MongoDB

In AWS EKS Console, I am able to see that mongodb-0 and nrf-free5gc-nrf-nrf-79d7d9b788-cmghx are installed in the same EC2 instances, resources such as memory, total number of pods or IPs are not running up.

So I would really appreciate it if any tips or findings can be provided.

Thanks!!

@Zhenye-Na
Copy link
Author

Closed this issue since it is not related to specific helm chart implementation.

For anyone who is curious, this issue is related to DNS lookup issue that is similar to aws/aws-cdk#10884

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Zhenye-Na