Permission issues with docker-sync, php-dev-macos in macos environment

[perarg]

Describe your issue
I have followed the guide word by word. I use MacOS so i enabled docker-sync too. I have copied clean drupal files into my folder and the files are synced to containers. Then i try to start drupal installation but i have permission errors on creating folders or files. I notice that the nginx container uses uid/gid 1000/1000 but the php container uid/gid is 501/20 (i use php-dev-macos). Do you think this may cause the permission issue ? I send you screenshots with errors and the console of containers.

Output of docker info

Containers: 8
 Running: 8
 Paused: 0
 Stopped: 0
Images: 9
Server Version: 18.03.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.87-linuxkit-aufs
Operating System: Docker for Mac
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.855GiB
Name: linuxkit-025000000001
ID: H6IU:DU3O:R75Y:WYAP:AKPZ:LA2D:FFXN:VL2M:X2K5:OVBN:YLKJ:EIBW
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 113
 Goroutines: 136
 System Time: 2018-05-17T16:34:41.922085012Z
 EventsListeners: 3
HTTP Proxy: docker.for.mac.http.internal:3128
HTTPS Proxy: docker.for.mac.http.internal:3129
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Contents of your docker-compose.yml

version: "2"

services:
  mariadb:
    image: wodby/mariadb:$MARIADB_TAG
    container_name: "${PROJECT_NAME}_mariadb"
    stop_grace_period: 30s
    environment:
      MYSQL_ROOT_PASSWORD: $DB_ROOT_PASSWORD
      MYSQL_DATABASE: $DB_NAME
      MYSQL_USER: $DB_USER
      MYSQL_PASSWORD: $DB_PASSWORD
#    volumes:
#      - ./mariadb-init:/docker-entrypoint-initdb.d # Place init .sql file(s) here.
#      - /path/to/mariadb/data/on/host:/var/lib/mysql # I want to manage volumes manually.

  php:
    image: wodby/drupal-php:$PHP_TAG
    container_name: "${PROJECT_NAME}_php"
    environment:
      PHP_SENDMAIL_PATH: /usr/sbin/sendmail -t -i -S mailhog:1025
      DB_HOST: $DB_HOST
      DB_USER: $DB_USER
      DB_PASSWORD: $DB_PASSWORD
      DB_NAME: $DB_NAME
      DB_DRIVER: $DB_DRIVER
## Read instructions at https://wodby.com/stacks/drupal/docs/local/xdebug/
##      PHP_XDEBUG: 1
##      PHP_XDEBUG_DEFAULT_ENABLE: 1
##      PHP_XDEBUG_REMOTE_CONNECT_BACK: 0
##      PHP_IDE_CONFIG: serverName=my-ide
#      PHP_XDEBUG_REMOTE_HOST: host.docker.internal # Docker 18.03+ & Linux/Mac/Win
#      PHP_XDEBUG_REMOTE_HOST: 172.17.0.1 # Linux, Docker < 18.03
#      PHP_XDEBUG_REMOTE_HOST: 10.254.254.254 # macOS, Docker < 18.03
#      PHP_XDEBUG_REMOTE_HOST: 10.0.75.1 # Windows, Docker < 18.03
    volumes:
#      - ./:/var/www/html
## For macOS users (https://wodby.com/stacks/drupal/docs/local/docker-for-mac/)
#      - /Users/perarg/Projects/drupal853:/var/www/html:cached # User-guided caching
      - docker-sync:/var/www/html # Docker-sync
## For Xdebug profiler files
#      - files:/mnt/files

  nginx:
    image: wodby/drupal-nginx:$NGINX_TAG
    container_name: "${PROJECT_NAME}_nginx"
    depends_on:
      - php
    environment:
      NGINX_STATIC_CONTENT_OPEN_FILE_CACHE: "off"
      NGINX_ERROR_LOG_LEVEL: debug
      NGINX_BACKEND_HOST: php
      NGINX_SERVER_ROOT: /var/www/html/web
#      NGINX_DRUPAL_FILE_PROXY_URL: http://example.com
    volumes:
#      - ./:/var/www/html
# For macOS users (https://wodby.com/stacks/drupal/docs/local/docker-for-mac/)
#      - /Users/perarg/Projects/drupal853:/var/www/html:cached # User-guided caching
      - docker-sync:/var/www/html # Docker-sync
    labels:
      - 'traefik.backend=nginx'
      - 'traefik.port=80'
      - 'traefik.frontend.rule=Host:${PROJECT_BASE_URL}'

  mailhog:
    image: mailhog/mailhog
    container_name: "${PROJECT_NAME}_mailhog"
    labels:
      - 'traefik.backend=mailhog'
      - 'traefik.port=8025'
      - 'traefik.frontend.rule=Host:mailhog.${PROJECT_BASE_URL}'

#  postgres:
#    image: wodby/postgres:$POSTGRES_TAG
#    container_name: "${PROJECT_NAME}_postgres"
#    stop_grace_period: 30s
#    environment:
#      POSTGRES_PASSWORD: $DB_PASSWORD
#      POSTGRES_DB: $DB_NAME
#      POSTGRES_USER: $DB_USER
#    volumes:
#      - ./postgres-init:/docker-entrypoint-initdb.d # Place init file(s) here.
#      - /path/to/postgres/data/on/host:/var/lib/postgresql/data # I want to manage volumes manually.

#  apache:
#    image: wodby/php-apache:$APACHE_TAG
#    container_name: "${PROJECT_NAME}_apache"
#    depends_on:
#      - php
#    environment:
#      APACHE_LOG_LEVEL: debug
#      APACHE_BACKEND_HOST: php
#      APACHE_SERVER_ROOT: /var/www/html/web
#    volumes:
#      - ./:/var/www/html
## For macOS users (https://wodby.com/stacks/drupal/docs/local/docker-for-mac/)
##      - ./:/var/www/html:cached # User-guided caching
##      - docker-sync:/var/www/html # Docker-sync
#    labels:
#      - 'traefik.backend=apache'
#      - 'traefik.port=80'
#      - 'traefik.frontend.rule=Host:${PROJECT_BASE_URL}'

#  varnish:
#    image: wodby/drupal-varnish:$VARNISH_TAG
#    container_name: "${PROJECT_NAME}_varnish"
#    depends_on:
#      - nginx
#    environment:
#      VARNISH_SECRET: secret
#      VARNISH_BACKEND_HOST: nginx
#      VARNISH_BACKEND_PORT: 80
#    labels:
#      - 'traefik.backend=varnish'
#      - 'traefik.port=6081'
#      - 'traefik.frontend.rule=Host:varnish.${PROJECT_BASE_URL}'

#  redis:
#    container_name: "${PROJECT_NAME}_redis"
#    image: wodby/redis:$REDIS_TAG

  adminer:
    container_name: "${PROJECT_NAME}_adminer"
    image: wodby/adminer:$ADMINER_TAG
    environment:
      ADMINER_SALT: adminer-salt
    labels:
      - 'traefik.backend=adminer'
      - 'traefik.port=9000'
      - 'traefik.frontend.rule=Host:adminer.${PROJECT_BASE_URL}'

#  pma:
#    image: phpmyadmin/phpmyadmin
#    container_name: "${PROJECT_NAME}_pma"
#    environment:
#      PMA_HOST: $DB_HOST
#      PMA_USER: $DB_USER
#      PMA_PASSWORD: $DB_PASSWORD
#      PHP_UPLOAD_MAX_FILESIZE: 1G
#      PHP_MAX_INPUT_VARS: 1G
#    labels:
#      - 'traefik.backend=pma'
#      - 'traefik.port=80'
#      - 'traefik.frontend.rule=Host:pma.${PROJECT_BASE_URL}'

#  solr:
#    image: wodby/drupal-solr:$SOLR_TAG
#    container_name: "${PROJECT_NAME}_solr"
#    environment:
#      SOLR_HEAP: 1024m
#    labels:
#      - 'traefik.backend=solr'
#      - 'traefik.port=8983'
#      - 'traefik.frontend.rule=Host:solr.${PROJECT_BASE_URL}'

#  nodejs:
#    image: wodby/drupal-node:$DRUPAL_NODE_TAG
#    container_name: "${PROJECT_NAME}_drupal_nodejs"
#    environment:
#       NODE_SERVICE_KEY: node-service-key
#    labels:
#      - 'traefik.backend=nodejs'
#      - 'traefik.port=8080'
#      - 'traefik.frontend.rule=Host:nodejs.${PROJECT_BASE_URL}'
#    volumes:
#      - ./path/to/your/single-page-app:/app
#    command: sh -c 'npm install && npm run start'

#  memcached:
#    container_name: "${PROJECT_NAME}_memcached"
#    image: wodby/memcached:$MEMCACHED_TAG

#  rsyslog:
#    container_name: "${PROJECT_NAME}_rsyslog"
#    image: wodby/rsyslog:$RSYSLOG_TAG

#  athenapdf:
#    image: arachnysdocker/athenapdf-service:$ATHENAPDF_TAG
#    container_name: "${PROJECT_NAME}_athenapdf"
#    environment:
#      WEAVER_AUTH_KEY: weaver-auth-key
#      WEAVER_ATHENA_CMD: "athenapdf -S"
#      WEAVER_MAX_WORKERS: 10
#      WEAVER_MAX_CONVERSION_QUEUE: 50
#      WEAVER_WORKER_TIMEOUT: 90
#      WEAVER_CONVERSION_FALLBACK: "false"

#  node:
#    image: wodby/node:$NODE_TAG
#    container_name: "${PROJECT_NAME}_node"
#    working_dir: /app
#    labels:
#      - 'traefik.backend=node'
#      - 'traefik.port=3000'
#      - 'traefik.frontend.rule=Host:front.${PROJECT_BASE_URL}'
#    expose:
#      - "3000"
#    volumes:
#      - ./path/to/your/single-page-app:/app
#    command: sh -c 'npm install && npm run start'

#  blackfire:
#    image: blackfire/blackfire
#    container_name: "${PROJECT_NAME}_blackfire"
#    environment:
#      BLACKFIRE_SERVER_ID: XXXXX
#      BLACKFIRE_SERVER_TOKEN: YYYYY

#  webgrind:
#    image: wodby/webgrind:$WEBGRIND_TAG
#    container_name: "${PROJECT_NAME}_webgrind"
#    environment:
#      WEBGRIND_PROFILER_DIR: /mnt/files/xdebug/profiler
#    labels:
#      - 'traefik.backend=webgrind'
#      - 'traefik.port=8080'
#      - 'traefik.frontend.rule=Host:webgrind.${PROJECT_BASE_URL}'
#    volumes:
#      - files:/mnt/files

#  elasticsearch:
#    image: wodby/elasticsearch:$ELASTICSEARCH_TAG
#    environment:
#      ES_JAVA_OPTS: "-Xms500m -Xmx500m"
#    ulimits:
#      memlock:
#        soft: -1
#        hard: -1

#  kibana:
#    image: wodby/kibana:$KIBANA_TAG
#    depends_on:
#      - elasticsearch
#    labels:
#      - 'traefik.backend=kibana'
#      - 'traefik.port=5601'
#      - 'traefik.frontend.rule=Host:kibana.php.docker.localhost'

  portainer:
    image: portainer/portainer
    container_name: "${PROJECT_NAME}_portainer"
    command: --no-auth -H unix:///var/run/docker.sock
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    labels:
      - 'traefik.backend=portainer'
      - 'traefik.port=9000'
      - 'traefik.frontend.rule=Host:portainer.${PROJECT_BASE_URL}'

  traefik:
    image: traefik
    container_name: "${PROJECT_NAME}_traefik"
    command: -c /dev/null --web --docker --logLevel=INFO
    ports:
      - '8000:80'
#      - '8080:8080' # Dashboard
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

volumes:
## Docker-sync for macOS users
  docker-sync:
    external: true
## For Xdebug profiler
#  files:

Contents of your .env

### Full documentation available at https://wodby.com/stacks/drupal/docs/local
###
### IMAGES TAGS CONSIST OF [VERSION]-[STABILITY_TAG]
###
### [VERSION] is usually a version of application running in a container
### [VERSION] sometimes additionally includes major drupal version (see nginx)
### [STABILITY_TAG] is a version of image (not application)
### [STABILITY_TAG] correspond to git tag of corresponding image repository
###
### EXAMPLE: wodby/mariadb:10.2-3.1.2 has MariaDB 10.2 and stability tag 3.0.2
### New stability tags include patch updates for applications and other fixes/improvements
### Changelog for stability tag can be found at https://github.com/wodby/mariadb/releases
###
### !!! For better reliability NEVER USE image without stability tag


### PROJECT SETTINGS

PROJECT_NAME=my_drupal8_project
PROJECT_BASE_URL=drupal.docker.localhost

DB_NAME=drupal
DB_USER=drupal
DB_PASSWORD=drupal
DB_ROOT_PASSWORD=password
DB_HOST=mariadb
DB_DRIVER=mysql

### --- MARIADB ----

MARIADB_TAG=10.2-3.1.3
#MARIADB_TAG=10.1-3.1.3

### --- VANILLA DRUPAL ----
### [DRUPAL_VERSION]-[PHP_VERSION]-[STABILITY_TAG]

DRUPAL_TAG=8-7.1-4.4.4
#DRUPAL_TAG=8-7.0-4.4.4
#DRUPAL_TAG=7-7.1-4.4.4
#DRUPAL_TAG=7-7.0-4.4.4
#DRUPAL_TAG=7-5.6-4.4.4
#DRUPAL_TAG=6-5.6-4.4.4
#DRUPAL_TAG=6-5.3-4.4.4

### --- PHP ----

#PHP_TAG=7.1-dev-4.4.2
#PHP_TAG=7.0-dev-4.4.2
#PHP_TAG=5.6-dev-4.4.2
#PHP_TAG=5.3-dev-4.4.2
PHP_TAG=7.1-dev-macos-4.4.2
#PHP_TAG=7.0-dev-macos-4.4.2
#PHP_TAG=5.6-dev-macos-4.4.2
#PHP_TAG=5.3-dev-macos-4.4.2

### --- NGINX ----
### [DRUPAL_VERSION]-[NGINX_VERSION]-[STABILITY_TAG]

NGINX_TAG=8-1.13-4.1.0
#NGINX_TAG=7-1.13-4.1.0
#NGINX_TAG=6-1.13-4.1.0
#NGINX_TAG=8-1.14-4.1.0
#NGINX_TAG=7-1.14-4.1.0
#NGINX_TAG=6-1.14-4.1.0

### --- SOLR ---
### [DRUPAL_VERSION]-[SOLR_VERSION]-[STABILITY_TAG]

SOLR_TAG=8-6.6-2.4.0
#SOLR_TAG=8-6.5-2.4.0
#SOLR_TAG=8-6.4-2.4.0
#SOLR_TAG=8-6.3-2.4.0
#SOLR_TAG=8-5.5-2.4.0
#SOLR_TAG=8-7.0-2.4.0
#SOLR_TAG=8-7.1-2.4.0
#SOLR_TAG=8-7.2-2.4.0
#SOLR_TAG=7-5.4-2.4.0

### --- ELASTICSEARCH ---

ELASTICSEARCH_TAG=6.2-2.2.0
#ELASTICSEARCH_TAG=6.1-2.2.0
#ELASTICSEARCH_TAG=6.0-2.2.0
#ELASTICSEARCH_TAG=5.6-2.2.0
#ELASTICSEARCH_TAG=5.5-2.2.0
#ELASTICSEARCH_TAG=5.4-2.2.0

### --- KIBANA ---

KIBANA_TAG=6.2-2.1.0
#KIBANA_TAG=6.1-2.1.0
#KIBANA_TAG=6.0-2.1.0
#KIBANA_TAG=5.6-2.1.0
#KIBANA_TAG=5.5-2.1.0
#KIBANA_TAG=5.4-2.1.0

### --- REDIS ---

REDIS_TAG=4.0-2.1.5
#REDIS_TAG=3.2-2.1.5

### --- NODE ---

NODE_TAG=9.11-0.2.0
#NODE_TAG=8.11-0.2.0

### --- POSTGRESQL ----

POSTGRES_TAG=10.1-1.3.0
#POSTGRES_TAG=9.6-1.3.0
#POSTGRES_TAG=9.5-1.3.0
#POSTGRES_TAG=9.4-1.3.0
#POSTGRES_TAG=9.3-1.3.0

### OTHERS

ADMINER_TAG=4.3-1.1.0
APACHE_TAG=2.4-3.0.5
ATHENAPDF_TAG=2.10.0
DRUPAL_NODE_TAG=1.0-2.0.0
MEMCACHED_TAG=1.4-2.0.0
RSYSLOG_TAG=latest
VARNISH_TAG=4.1-2.4.0
WEBGRIND_TAG=1.5-1.3.0

[csandanov]

It seems your codebase owned by 1000:root which is very unusual, normally the default user on macOS has uid 501. Trying using a usual -dev PHP image instead of -dev-macos (wodby user has uid 1000 like in Linux) and you should probably want to fix this root group as well.

[perarg]

Hmm.. i have renamed docker-compose.override.yml to .bak so to omit this file. This file mounts a volume codebase that i don't need it as i want the docker-sync to mount my own folder. Is this something that has any relation to my issue ?

[csandanov]

docker-compose.override.yml used for vanilla Drupal. If you want to mount your own codebase you don't need it at all. It has nothing to do with the permissions issue

[Niklan]

I have the same issue on Linux. Any suggestions? It was not presented month ago, didn't know what happens, but I bet on new version of Docker4Drupal 5.0.6 where is new image versions.

$ id
uid=1000(nikita) gid=1000(nikita) groups=1000(nikita),4(adm),24(cdrom),27(sudo),30(dip),33(www-data),46(plugdev),82(alpine-www-data),113(lpadmin),126(sambashare),999(docker)

If I set to folder 777 chmod, than install is passed, but the files is not accessable on the host anymore.

Months ago I use:

sudo setfacl -dR -m u:$(whoami):rwX -m u:82:rwX -m u:100:rX $(pwd)
sudo setfacl -R -m u:$(whoami):rwX -m u:82:rwX -m u:100:rX $(pwd)

But this fix was for 4.x, not for 5.x where is uid/guid is became 1000.

[Niklan]

Created the same new project, but on 5.0.5, there is no problem.
So, this is only for 5.0.6. I bet something changed inside PHP images that pulls for 5.0.6

UPD. Files, for translations, for example, not editable, but webserver can create folder by itself and can edit settings.php, which also has problems with permissions on 5.0.6

[csandanov]

See wodby/php#22 to understand better the permissions problem. In short, fpm should not have write permissions to your codebase except for specific directories like sites/*/files. Also, we provide helper scripts when you occasionally need to change ownership or add write access for group (wodby is a part of www-data group) for some reason.

[perarg]

@csandanov i use -dev in php container. Now it seems everything normal. I console every container and the wodby user is the owner of every file and folder. It seems great but it isn't :(

In Drupal Status report, it says that the directory sites/default/files is not writable. But nginx and php container have the right permissions...

UPDATE
I am going crazy... I have two main containers (nginx and php) that are connected with a docker-sync volume.

Let's take as example the folder sites/default/files. Giving ln -lashn sites/default i have:

In nginx container: drwxrwxr-x 1000 1000 files (1000 is wodby)
In php container: drwxrwxr-x 1000 1000 files (1000 is wodby)
In docker-sync container: drwxrwxr-x 1000 0 files (1000 is dockersy)
In my local folder: drwxrwxr-x 501 20 files (501 is default mac user)

The drupal says that the directory sites/default/files is not writable !! Which user is responsible to drupal installation for accessing files folder ? The fact that the nginx and php container can write into this folder is not enough ?

If i give 777 permissions to that folder, the message is gone. But i want to find the root of the problem and not just bypass it.

[csandanov]

PHP-FPM runs from www-data:www-data user, your codebase owned by wodby. That's all described in the issue I linked in my previous post.

[Niklan]

@csandanov

In short, fpm should not have write permissions to your codebase except for specific directories like sites/*/files

But in fact, 5.0.6 hasn't access to sites/*/files, but 5.0.5 has. That's odd. On clean project 5.0.6 can't create translation folder in sites/*/files, but 5.0.5 can. This also affects settings.php, which 5.0.6 can't edit.

I think the problem is only for newer version of php container. I just use 5.0.5 for new project and haven't any problem with it, using of 5.0.6 is not possible due to permission problems.

[csandanov]

This is the latest vanilla Drupal:

$ docker run --rm -ti wodby/drupal:8-7.1-4.4.4 sh                                                                                                   [14:08:03]
Drupal 8 not found in /var/www/html - copying now...
Complete! Drupal 8 has been successfully copied to /var/www/html
[email protected]:/var/www/html $ ls -la web/sites/default/
total 84
drwxr-xr-x    3 wodby    wodby         4096 Apr 27 10:20 .
drwxr-xr-x    3 wodby    wodby         4096 Apr 27 10:20 ..
-rw-r--r--    1 wodby    wodby         6762 Apr 27 10:20 default.services.yml
-rw-r--r--    1 wodby    wodby        31561 Apr 27 10:20 default.settings.php
drwxrwxrwx    2 wodby    wodby         4096 Apr 27 10:20 files
-rw-rw-rw-    1 wodby    wodby        31951 May 21 07:08 settings.php

[perarg]

Hm.. i am getting to the point. So, i gave chown -R wodby:www-data sites/default/files and now the files folder is writable (because the group is www-data)... I am going to go further now so i don't know what i have to expect further more.

@csandanov you refer helper-scripts and i am trying to understand what /mnt/files is ? The helper scripts are working only under /mnt/files

In documentation we read

Codebase volume $APP_ROOT (/var/www/html) owned by wodby:wodby. Files volume $FILES_DIR (/mnt/files) owned by www-data:www-data with 775 mode.

What is the Files volume ? I thought every file of interest is located at /var/www/html and the sites/default/files folder is located there too. So what is Files volume for ?

[csandanov]

Ahh.. right, just ignore it. Files volume not used for local development, we use it when deploying apps via Wodby Platform to separate codebase and files.

If you don't have enough permission to manipulate with ownership you can always run sudo su in the dev versions (-dev tag) of php images.