Bitnami Postgres doesn't deploy in Rancher Desktop due to volume mount permissions #502

agracey · 2021-08-13T14:57:20Z

During an install of Epinio, I'm getting an error when it tries to install a bitnami postgres pod:

PS C:\Users\Andrew Gracey> kubectl logs gitea-postgresql-0 -ngitea gitea-postgresql
postgresql 14:42:42.35
postgresql 14:42:42.35 Welcome to the Bitnami postgresql container
postgresql 14:42:42.35 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql
postgresql 14:42:42.35 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql/issues
postgresql 14:42:42.35
postgresql 14:42:42.35 INFO  ==> ** Starting PostgreSQL setup **
postgresql 14:42:42.36 INFO  ==> Validating settings in POSTGRESQL_* env vars..
postgresql 14:42:42.37 INFO  ==> Loading custom pre-init scripts...
postgresql 14:42:42.37 INFO  ==> Initializing PostgreSQL database...
mkdir: cannot create directory ‘/bitnami/postgresql’: Permission denied
postgresql 14:42:42.37 INFO  ==> Stopping PostgreSQL...

I found this which is likely the same issue so I'm wondering if there's any way to support fsgroup enabled storage? (Is that a thing?)
bitnami/charts#1210

Also, should I also file an issue with https://github.com/epinio/epinio?

The text was updated successfully, but these errors were encountered:

jandubois · 2021-08-13T16:00:38Z

Are you installing the postgresql chart with --set volumePermissions.enabled=true? That should create an init container that sets the volume permissions to the expected values.

I believe the same problem can happen with nfs-volume-provisioner as well.

jandubois · 2021-08-13T18:50:18Z

I just realized that we control https://github.com/rancher/local-path-provisioner, so we should investigate if we can't fix it.

I found some hint in kubernetes/kubernetes#93802 (comment):

fsgroup is ignored for shared filesystems like nfs

However, "Local Persistent Volumes" are supposed to provide full control over ownership:

Additional benefits include support for formatting of block devices during mount, and volume ownership using fsGroup.

jandubois · 2021-08-20T18:29:59Z

Note that we want to keep this issue open even though the Epinio issue may be resolved by adding the volume permissions setting (epinio/epinio#704).

We do want to investigate if there is anything we can do in the local-path-provisioner to avoid the issue to begin with.

habibi07 · 2021-08-26T13:20:47Z

Hi, i had the same issue with bitnami postgresql helm chart. I solved it by providing extraInitContainers. Im not sure if this helps in your case but it may give you a clue how to solve it.

readReplicas:
  extraInitContainers:
    - name: init-perms
      image: ubuntu
      command: ["/bin/sh"]
      args:
        - -c
        - >-
          chown -R 1001:1001 /bitnami/postgresql
      volumeMounts:
      - name: data
        mountPath: /bitnami/postgresql
        subPath:
primary:
  extraInitContainers:
    - name: init-perms
      image: ubuntu
      command: ["/bin/sh"]
      args:
        - -c
        - >-
          chown -R 1001:1001 /bitnami/postgresql
      volumeMounts:
      - name: data
        mountPath: /bitnami/postgresql
        subPath:

jandubois · 2021-08-26T17:27:18Z

I solved it by providing extraInitContainers.

@habibi07 Did setting --set volumePermissions.enabled=true by itself not fix the problem for you?

It has kind of complex logic, but is supposed to already do the chown that your extra init container is running. If this didn't work for you, maybe file a bug against the chart?

agracey · 2021-08-26T18:27:31Z

I'm having what I think is a similar issue with deploying MinIO on Rancher Desktop.

PS C:\Users\Andrew Gracey\Projects> kubectl logs minio-5c7f78cdc7-ph475 -nminio

 You are running an older version of MinIO released 4 days ago
 Update: Run `mc admin update`


ERROR Unable to initialize backend: mkdir /export/.minio.sys: permission denied

jandubois · 2021-08-26T18:35:36Z

I'm having what I think is a similar issue with deploying MinIO on Rancher Desktop.

@agracey Assuming you are using the Bitnami MinIO chart, the fix is the same as before: --set volumePermissions.enabled=true

All the Bitnami charts that use PVCs use the same settings: Volume Permissions Parameters

agracey · 2021-08-26T19:49:58Z

Ok, things just got weird. I can no longer reproduce the error... And I don't know why not. I could easily reproduce this morning then when I went to actually measure things, I couldn't get it to fail!

agracey · 2021-08-26T19:53:18Z

@habibi07 If you reproduce, could you run kubectl describe configmap local-path-config -n kube-system and (from the host) ls -la /var/lib/rancher/k3s/storage

agracey · 2021-08-27T17:20:30Z

From what I can tell, it looks like there's a slight difference in ConfigMap provided to the local-path-provisioner form this commit in k3s: k3s-io/k3s@a1d7a62#diff-2db2a6c1c81cb2084152e746784de4a98eb9bc5ec06091ebfdfb8c52ca4fd6f3

I'm not sure this would be the issue and I can't test due to not being able to reproduce again

pramodsharma62 · 2021-10-04T17:12:06Z

@agracey - I am facing exact same issue. I enabled persistence pv claim. Is there any workaround?

persistence:
enabled: true
size: 10Gi
existingClaim: postgressql-pv-claim

Claim
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
labels:
app: postgressql
name: postgressql-pv-claim
namespace: default
spec:
storageClassName: manual
accessModes:

ReadWriteOnce
resources:
requests:
storage: 10Gi

Logs
│ 2021-10-04 17:08:49.072 GMT [1] LOG: skipping missing configuration file "/bitnami/postgresql/data/postgresql.auto.conf" ││ 2021-10-04 17:08:49.073 GMT [1] FATAL: data directory "/bitnami/postgresql/data" has wrong ownership ││ 2021-10-04 17:08:49.073 GMT [1] HINT: The server must be started by the user that owns the data directory.

jandubois · 2021-10-04T17:58:07Z

@pramodsharma62 See above: #502 (comment)

--set volumePermissions.enabled=true

pramodsharma62 · 2021-10-04T18:24:10Z

@jandubois : unfortunately still the same issue

jandubois · 2021-10-04T18:29:12Z

unfortunately still the same issue

@pramodsharma62 Please show the exact helm command you are running. If you load postgres as a subchart of a different chart, you will have to prefix the config setting with the subchart name, e.g.

--set postgresql.volumePermissions.enabled=true

[Edited; the original had an erroneous additional =true at the end]

But it is really difficult to tell without knowing what commands you are running.

pramodsharma62 · 2021-10-04T18:41:24Z

@jandubois - I am using jfrog with postgress and already enabled postgress on jfrog helm chart values.yaml.
Seee https://github.com/jfrog/charts/blob/1ebfce79e4c53204a05cd637a8e1507b1c2739a4/stable/artifactory/values.yaml#L1705

postgresql:
enabled: true
image:
registry: releases-docker.jfrog.io
repository: bitnami/postgresql
tag: 13.2.0-debian-10-r55
postgresqlUsername: artifactory
postgresqlPassword: ""
postgresqlDatabase: artifactory
postgresqlExtendedConf:

I am running helm upgrade command.
helm upgrade artifactory jfrog/artifactory -f values.yaml --set volumePermissions.enabled=true --set databaseUpgradeReady=true

pramodsharma62 · 2021-10-04T18:43:05Z

I don't think there chart has postgresql.volumePermissions.enabled=true=true

I might have to deploy postgressql separately?

jandubois · 2021-10-04T18:44:31Z

@pramodsharma62 Thank you! Sorry for the typo (well, cut&pasto), the options is just

--set postgresql.volumePermissions.enabled=true

I'm pretty confident this will work for you.

pramodsharma62 · 2021-10-04T19:03:56Z

No success still the same issue.

gresql 18:59:38.14 INFO ==> ** PostgreSQL setup finished! ** ││ postgresql 18:59:38.15 INFO ==> ** Starting PostgreSQL ** ││ 2021-10-04 18:59:38.169 GMT [1] LOG: skipping missing configuration file "/bitnami/postgresql/data/postgresql.auto.conf" ││ 2021-10-04 18:59:38.169 GMT [1] FATAL: data directory "/bitnami/postgresql/data" has wrong ownership ││ 2021-10-04 18:59:38.169 GMT [1] HINT: The server must be started by the user that owns the data directory. ││ stream closed ││ ││ ││

jandubois · 2021-10-04T20:23:08Z

I don't know; maybe you need to specify it on the initial deployment and cannot specify it during an update.

I'm still convinced that the correct application of this option will make postgresql work for you.

agracey · 2021-10-04T21:02:07Z

I think @jandubois is likely correct and the PV is likely surviving the helm upgrade.

To workaround this, you can run this incredibly insecure command inside the rancher-desktop VM:
chmod 777 /var/lib/rancher/k3s/storage/*

I'm using windows, so I can access the rancher-desktop WSL instance through the windows terminal:

I'd imagine there's a lima command to do similar on Mac?

roelensw · 2021-10-13T15:43:20Z

I had the same issue
postgresql 15:33:30.89 INFO ==> Initializing PostgreSQL database... mkdir: cannot create directory ‘/bitnami/postgresql/data’: Permission denied

The option fixed it --set postgresql.volumePermissions.enabled=true

cdprete · 2022-02-14T15:27:47Z

Hi, i had the same issue with bitnami postgresql helm chart. I solved it by providing extraInitContainers. Im not sure if this helps in your case but it may give you a clue how to solve it.

readReplicas:
  extraInitContainers:
    - name: init-perms
      image: ubuntu
      command: ["/bin/sh"]
      args:
        - -c
        - >-
          chown -R 1001:1001 /bitnami/postgresql
      volumeMounts:
      - name: data
        mountPath: /bitnami/postgresql
        subPath:
primary:
  extraInitContainers:
    - name: init-perms
      image: ubuntu
      command: ["/bin/sh"]
      args:
        - -c
        - >-
          chown -R 1001:1001 /bitnami/postgresql
      volumeMounts:
      - name: data
        mountPath: /bitnami/postgresql
        subPath:

Why not implementing something like bitnami/charts#1210 (comment) instead?
I think it's way cleaner than having to have an init container just to fix something that should work out of the box...

agracey mentioned this issue Aug 13, 2021

Set flag to fix postgres volume permissions epinio/epinio#684

Closed

mook-as added kind/bug Something isn't working area/volume Access to host volumes from inside the VM or containers parity/project Feature is available from other projects labels Aug 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bitnami Postgres doesn't deploy in Rancher Desktop due to volume mount permissions #502

Bitnami Postgres doesn't deploy in Rancher Desktop due to volume mount permissions #502

agracey commented Aug 13, 2021

jandubois commented Aug 13, 2021

jandubois commented Aug 13, 2021

jandubois commented Aug 20, 2021

habibi07 commented Aug 26, 2021

jandubois commented Aug 26, 2021

agracey commented Aug 26, 2021 •

edited

Loading

jandubois commented Aug 26, 2021

agracey commented Aug 26, 2021

agracey commented Aug 26, 2021 •

edited

Loading

agracey commented Aug 27, 2021

pramodsharma62 commented Oct 4, 2021 •

edited

Loading

jandubois commented Oct 4, 2021

pramodsharma62 commented Oct 4, 2021

jandubois commented Oct 4, 2021 •

edited

Loading

pramodsharma62 commented Oct 4, 2021

pramodsharma62 commented Oct 4, 2021

jandubois commented Oct 4, 2021

pramodsharma62 commented Oct 4, 2021

jandubois commented Oct 4, 2021

agracey commented Oct 4, 2021

roelensw commented Oct 13, 2021

cdprete commented Feb 14, 2022

Bitnami Postgres doesn't deploy in Rancher Desktop due to volume mount permissions #502

Bitnami Postgres doesn't deploy in Rancher Desktop due to volume mount permissions #502

Comments

agracey commented Aug 13, 2021

jandubois commented Aug 13, 2021

jandubois commented Aug 13, 2021

jandubois commented Aug 20, 2021

habibi07 commented Aug 26, 2021

jandubois commented Aug 26, 2021

agracey commented Aug 26, 2021 • edited Loading

jandubois commented Aug 26, 2021

agracey commented Aug 26, 2021

agracey commented Aug 26, 2021 • edited Loading

agracey commented Aug 27, 2021

pramodsharma62 commented Oct 4, 2021 • edited Loading

persistence: enabled: true size: 10Gi existingClaim: postgressql-pv-claim

jandubois commented Oct 4, 2021

pramodsharma62 commented Oct 4, 2021

jandubois commented Oct 4, 2021 • edited Loading

pramodsharma62 commented Oct 4, 2021

pramodsharma62 commented Oct 4, 2021

jandubois commented Oct 4, 2021

pramodsharma62 commented Oct 4, 2021

jandubois commented Oct 4, 2021

agracey commented Oct 4, 2021

roelensw commented Oct 13, 2021

cdprete commented Feb 14, 2022

agracey commented Aug 26, 2021 •

edited

Loading

agracey commented Aug 26, 2021 •

edited

Loading

pramodsharma62 commented Oct 4, 2021 •

edited

Loading

persistence:
enabled: true
size: 10Gi
existingClaim: postgressql-pv-claim

jandubois commented Oct 4, 2021 •

edited

Loading