-
Notifications
You must be signed in to change notification settings - Fork 638
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Upgrade from 2.12.2 to 2.13.1 failed - mkdir: cannot create directory '/var/lib/pgsql/data/userdata': Permission denied #1775
Comments
Looks similar to #1770 |
In the documentation from here there is this paragraph :
is there any reason to change container image from official postgres to sclorg one ? Thank you very much for your help. |
In the Dockerfile here we can see :
So if I understand it correctly, by default in the postgres 15 container the postgres user have it's uid equals to 26. So, I don't understand why, by default during the installation of postgres 15 we have this kind of issue :
to run the installation of awx or upgrade awx I'm using this script :
and here is the
And to apply :
|
@fosterseth is your patch from #1770 (comment) can work for my issue too ? Is it possible to have an auto init for the postgres data path / config ? because it looks like UID = 26 is a prerequisites for postgres in the repository from sclorg so users shouldn't have to add custom spec to have it working in case of upgrade from 2.12.2 to 2.13.1, right ? Thank you very much for your help |
@kurokobo, @TheRealHaoLiu , @fosterseth , how can I proceed to solve the issue in my case because I have How can we solve this issue when the default storage class is : longhorn ? Thank you very much for your help. |
Hey @craph I also suffered this so I rolled back to the previous operator version. |
@mateuszdrab Thank you very much for the update. One question, if the old pvc is still there, do we have to do restore from backup because normally all the previous data should be inside the longhorn pvc, right ? |
I'd just use the old PVC, in my case it was gone so I had no choice as after the rollback it was recreated with an empty database. |
@mateuszdrab How do you proceed to have an AWX backup ? Do you use https://github.com/ansible/awx-operator/tree/devel/roles/backup ? |
@fosterseth , @TheRealHaoLiu , do you have any updates about how to upgrade safely awx from 2.12.2 to future version of awx-operator without loosing any data (postgres data migration..., documentation) ? |
No, I just had the backup of the PVCs of both awx and Postgres. I rolled back and I restored those |
Could you give this PR a try and see if it solves your issue? |
I was completely stuck with this error until I used quay.io/fosterseth/awx-operator:postgres_init & the init container commands I tried the postgres_security_context_settings first but that alone didn't do it. Haven't tried without the security context settings/postgres_data_path and just the init container. I won't be surprised if I have some superfluous settings. I had to add the crd edit and init postgres commands after using kustomize as the github ref for 2.14 didn't include them. Just figured I'd post what worked for me and I'm appreciative of everyone's work and comments that helped get this back and running for my deployments. CustomResourceDefinition
spec: |
This issue is still ongoing for me as well. Even when trying to spin up a fresh awx instance im still stuck with the database crashlooping on error 'mkdir: cannot create directory '/var/lib/pgsql/data/userdata': permission denied' I've tried the following "fixes" mentioned in other issues with no change |
Me too. Attempted to fresh install AWX 24.3.1 with AWX operator 2.16.1 |
@RaceFPV @jyanesnotariado I'm guessing you tried this out any idea why that method doesn't work in your case? |
I'm thinking its because I didn't upgrade the CRDs first before running the upgrade process for the helm chart itself, and that's why that option wasn't working for me, still testing if that's the case. |
Hi @rooftopcellist , @fosterseth , PR #1805 solved my issue but after the migration is done, the pod "migration" is still present with status completed. Thank you very much. |
I did! but even though it downloaded and ran, I kept running into the same error. What I did to fix it was chown 26:root on the mapped volume and it worked for me. |
Hi @rooftopcellist , @fosterseth , PR #1805 solved my issue but after the migration is done, the pod "migration" is still present with status completed. Can I delete the pod "migration" ? Thank you very much. Best regards, |
Just hit this and checking the source for the postgres deployment I come to this line:
This is the container security context, which does not allow fsGroup setting: Ideally, the awx CR should allow to update the postgres pod security context to allow setting fsGroup: While the implemented solution of the init container might sort out the issue, it can overlap with functionality in the storage provider (like the NFS CSI) where the provider can update permissions IF fsGroup is declared. |
Please confirm the following
Bug Summary
Unable to upgrade from 2.12.2 to 2.13.1
CrashLoopBackOff in the Postgres container.
AWX Operator version
2.13.1
AWX version
24.0.0
Kubernetes platform
kubernetes
Kubernetes/Platform version
v1.25.16+rke2r1
Modifications
no
Steps to reproduce
upgrade from 2.12.2 to 2.13.1
Expected results
migration from postgres 13 to 15 should work without any permissions issues
Actual results
Unable to upgrade from 2.12.2 to 2.13.1
CrashLoopBackOff in the Postgres container.
Additional information
No response
Operator Logs
No response
The text was updated successfully, but these errors were encountered: