geonode django container dosn't get database connection

[mwallschlaeger]

after deploying the helm chart it sometimes randomly orrures that the geonode container can not find the database or does not have access to it. You see the issue in the geonode container logs like:

MONITORING_SERVICE_NAME=geonode-monitoring
MONITORING_DATA_TTL=365
/usr/lib/python3/dist-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated
  "class": algorithms.Blowfish,
**************************databases*******************************
psql: error: connection to server at "geonode-postgresql" (10.104.197.204), port 5432 failed: FATAL:  password authentication failed for user "postgres"
connection to server at "geonode-postgresql" (10.104.197.204), port 5432 failed: FATAL:  pg_hba.conf rejects connection for host "10.244.0.254", user "postgres", database "postgres", no encryption
Postgres is unavailable - sleeping
psql: error: connection to server at "geonode-postgresql" (10.104.197.204), port 5432 failed: FATAL:  password authentication failed for user "postgres"
connection to server at "geonode-postgresql" (10.104.197.204), port 5432 failed: FATAL:  pg_hba.conf rejects connection for host "10.244.0.254", user "postgres", database "postgres", no encryption
Postgres is unavailable - sleeping
psql: error: connection to server at "geonode-postgresql" (10.104.197.204), port 5432 failed: FATAL:  password authentication failed for user "postgres"
connection to server at "geonode-postgresql" (10.104.197.204), port 5432 failed: FATAL:  pg_hba.conf rejects connection for host "10.244.0.254", user "postgres", database "postgres", no encryption
Postgres is unavailable - sleeping
psql: error: connection to server at "geonode-postgresql" (10.104.197.204), port 5432 failed: FATAL:  password authentication failed for user "postgres"
connection to server at "geonode-postgresql" (10.104.197.204), port 5432 failed: FATAL:  pg_hba.conf rejects connection for host "10.244.0.254", user "postgres", database "postgres", no encryption
Postgres is unavailable - sleeping
psql: error: connection to server at "geonode-postgresql" (10.104.197.204), port 5432 failed: FATAL:  password authentication failed for user "postgres"
connection to server at "geonode-postgresql" (10.104.197.204), port 5432 failed: FATAL:  pg_hba.conf rejects connection for host "10.244.0.254", user "postgres", database "postgres", no encryption
Postgres is unavailable - sleeping

if the database is found once it will be found forever but sometimes within the initial deploy its not working

[bradjones1]

I'm running into a similar situation with geoserver:

28 Aug 07:39:12 WARN   [util.JDBCExceptionReporter] - SQL Error: 0, SQLState: null
28 Aug 07:39:12 ERROR  [util.JDBCExceptionReporter] - Cannot create PoolableConnectionFactory (FATAL: password authentication failed for user "geonode")

[mwallschlaeger]

hi Brad, I generally fixed this issue for now by deleting and reinitializing the whole deployment. I think it's kind of a race condition while starting the pods.

[mwallschlaeger]

still can happen in 1.0.0

[mwallschlaeger]

i think this is related to the database env variable loading in the geonode pod here:

_helpers.tpl:
https://github.com/zalf-rdm/geonode-k8s/blob/f354949b0ccf5c16ae160fa1ce7972a6999d4291/charts/geonode/templates/_helpers.tpl#L36C1-L60C12

combined with the init container only checking if the database is accessable tcp layer. I guess sometimes the geonode pod gets created and load env vars from a secret which isn't final and therefore the geonode pod starts without or with wrong database credentials. I would suggest to edit the init container to actively interact with the postgres container using the postgresql-client