Volume mounted inside another sometimes disappears after file change

[josh-byster]

Description

Not sure if this is a Compose-specific issue or Moby so I have the linked issue here in case it's not Compose-specific.

Summary: We mount our source code in a directory in the container, and within that directory, mount on top a named volume which contains data from the built image (node_modules). Sometimes this named volume seems to unmount, resulting in an underlying empty directory in the container.

Background: Essentially, I have the mount of the backend folder to the container source code located at /var/www/app. However, since I don't have node_modules installed on the host, we have a named volume mounted called be_modules. This has the effect of (when it's working) using the installed node_modules in the image, effectively mounting "on-top" of the source code in /var/www/app.

The key is here in the Docker Compose file, which gets the node_modules directory working usually:

- ./backend:/var/www/app
- be_modules:/var/www/app/node_modules

Issue: Running docker-compose up or docker-compose run backend sh (for debugging) shows correctly the node_modules retrieved from the image. They all show up just fine, and the date/time on the node_modules folder when running ls is whenever the volume got created. All is good.

There's an empty node_modules directory created on the host, which is normal since it mounted into the container and so the corresponding folder got created on the host.

However, occasionally (and more frequent on my coworker's machine) we will make a few changes in the source code while our container is up. The changes syncs up to the container (as it should), however node_modules then becomes an empty folder in the container! Moreover, the datetime of the directory changes to the time when the image was built (and is the same as the timestamp of the empty directory on the host)! In other words, it's as if the named volume just randomly unmounts for no reason. Doing nothing to the source but restarting docker-compose down -v and docker-compose up fixes it until it happens again.

What we tried: Restarting Docker, removing all images, containers, volumes, and starting a fresh build.

Why would this be happening? My suspicion is that it's either a Docker / Compose bug since I can't reproduce with the exact same code, or there's some subtlety that I'm missing. I'm almost seems the issue has to do with when a file changes that Docker is re-mounting the entire directory in the container, and this is over-over-writing the be_modules mount with that empty directory, since the timestamp between the host empty directory and container empty directory are the same.
It's not device specific.

Steps to reproduce the issue:

Can't write an MWE since it doesn't happen always (for me, almost never; my coworker: all the time) and frequency depends on the machine, which makes me suspect that's a bug rather than something else.

Link to docker-compose gist

Link to dockerfile gist

Additional information you deem important (e.g. issue happens only occasionally):

Issue seems to happen randomly on my machine along with some of my coworkers. However, on one coworker's machine, this issue happens almost every time they make a change.

I'm posting here instead of Docker Compose since I think this may be an issue with volume mounting.

Context information (for bug reports)

Output of docker-compose version

docker-compose version 1.24.1, build 4667896b
docker-py version: 3.7.3
CPython version: 3.6.8
OpenSSL version: OpenSSL 1.1.0j  20 Nov 2018

Output of docker version

Client: Docker Engine - Community
 Version:           19.03.2
 API version:       1.40
 Go version:        go1.12.8
 Git commit:        6a30dfc
 Built:             Thu Aug 29 05:26:49 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.2
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.8
  Git commit:       6a30dfc
  Built:            Thu Aug 29 05:32:21 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker-compose config

services:
  backend:
    build:
      context: /Users/joshbyster/Documents/repo_name/backend
    command: npm start
    depends_on:
    - db
    environment:
      PORT: '5000'
    ports:
    - published: 5000
      target: 5000
    volumes:
    - /Users/joshbyster/Documents/repo_name/backend:/var/www/app:rw
    - be_modules:/var/www/app/node_modules:rw
  db:
    image: mongo
    init: true
    logging:
      driver: none
    ports:
    - published: 27017
      target: 27017
    volumes:
    - mongo_data:/data/db:rw
  frontend:
    build:
      context: /Users/joshbyster/Documents/repo_name
/frontend
    command: npm start
    depends_on:
    - backend
    - db
    environment:
      PORT: '3000'
      REACT_APP_API_PORT: '5000'
    ports:
    - published: 3000
      target: 3000
    volumes:
    - /Users/joshbyster/Documents/repo_name/frontend:/var/www/app:rw
    - fe_modules:/var/www/app/node_modules:rw
version: '3.7'
volumes:
  be_modules: {}
  fe_modules: {}
  mongo_data: {}

Steps to reproduce the issue

Please see above

We run the following commands in order to reproduce:

docker-compose run backend sh 
ls -al node_modules #shows correct listing of all node_modules
# make some change in a file and save it
ls -al node_modules # empty directory with date from when empty node_modules folder on host was created

Additional information

OS version / distribution, docker-compose install method, etc.

Running OS X 10.14.6 but see this issue on Docker with WSL.

[jcsirot]

Hello @josh-byster Thank you for the report. We are going to investigate.

[marcthayer]

Any updates on this?

[georgecrawford]

This exactly describes the problem I'm seeing. Again with docker-compose, with a sub-volume containing my node_modules. I'd love this to be looked into and fixed!

[tuanalumi]

We have the same issue. Thank @josh-byster for reporting it in this detail.

[jshbrntt]

Me and my co-worker have also been experiencing this issue to varying degrees.

[Akhil-bridge]

Any update on this?

[timothy-thomas]

I have this exact same issue.

Output of docker-compose version

docker-compose version 1.26.2, build eefe0d31
docker-py version: 4.2.2
CPython version: 3.7.7
OpenSSL version: OpenSSL 1.1.1g  21 Apr 2020

Output of docker version

Client: Docker Engine - Community
 Version:           19.03.12
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        48a66213fe
 Built:             Mon Jun 22 15:41:33 2020
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.12
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       48a66213fe
  Built:            Mon Jun 22 15:49:27 2020
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker-compose config

networks:
  backend:
    driver: bridge
services:
  neo4j:
    environment:
      NEO4J_dbms.memory.heap.initial_size: 1G
      NEO4J_dbms_memory_heap_max__size: 1G
      NEO4J_dbms_memory_pagecache_size: 1G
    image: neo4j:4.1.1
    networks:
      backend: null
    ports:
    - 7474/tcp
    - 7687/tcp
    restart: unless-stopped
    volumes:
    - /Users/Tim/Documents/Programming/Repos/app-node/neo4j/conf:/conf:rw
    - /Users/Tim/Documents/Programming/Repos/app-node/neo4j/data:/data:rw
    - /Users/Tim/Documents/Programming/Repos/app-node/neo4j/import:/import:rw
    - /Users/Tim/Documents/Programming/Repos/app-node/neo4j/logs:/logs:rw
    - /Users/Tim/Documents/Programming/Repos/app-node/neo4j/plugins:/plugins:rw
  nodejs:
    build:
      context: /Users/Tim/Documents/Programming/Repos/app-node/backend
    command: nodemon server.js
    networks:
      backend: null
    ports:
    - 49160:8080/tcp
    volumes:
    - /Users/Tim/Documents/Programming/Repos/app-node/backend:/usr/src/app:rw
    - nodejs_node_modules:/usr/src/app/node_modules:rw
version: '3.0'
volumes:
  nodejs_node_modules: {}

[marcelovani]

I had this issue before, now on Docker Desktop Community 2.4.0.0 for MacOs I have an issue that when I modify the mounted file on the host machine, it doesn't update the internal/mounted file, unless I restart docker engine completely.

[cullylarson]

I'm having this problem as well. I'm using an older version of Docker Desktop on macos because the newer versions have a persistent issue that I can't resolve (c.f. #3927). So I'm running 2.5.0.1. My volumes are mounted like this:

volumes:
  - "./.www:/var/www/html"
  - "./theme:/var/www/html/wp-content/themes/theme-name"

Whenever I change files in the ./theme folder (or strangely even in the ./.www folder), the /var/www/html/wp-content/themes/theme-name volume will become empty in the container (no files).

EDIT: I downgraded to 2.4.0.0. The issue doesn't happen on every edit, but does happen at frequent, random intervals.

EDIT: Downgraded to 2.3.0.5 and the problem went away.

[muka]

this happens on docker-ce on linux

      - ./libs:/app/libs
      - ./app:/app

the content of /app/libs is still the one in ./app/libs (a symlink in my case)

[putrafajarh]

I'm having this problem as well. I'm using an older version of Docker Desktop on macos because the newer versions have a persistent issue that I can't resolve (c.f. #3927). So I'm running 2.5.0.1. My volumes are mounted like this:

volumes:
  - "./.www:/var/www/html"
  - "./theme:/var/www/html/wp-content/themes/theme-name"

Whenever I change files in the ./theme folder (or strangely even in the ./.www folder), the /var/www/html/wp-content/themes/theme-name volume will become empty in the container (no files).

EDIT: I downgraded to 2.4.0.0. The issue doesn't happen on every edit, but does happen at frequent, random intervals.

EDIT: Downgraded to 2.3.0.5 and the problem went away.

did u fix this or solution may share?

[cullylarson]

@putrafajarh I never found a solution other than downgrading. Though, I haven't worked on the project that exhibited this behavior in a while.

[DmitriyLyalyuev]

@putrafajarh I never found a solution other than downgrading. Though, I haven't worked on the project that exhibited this behavior in a while.

Which version is working fine?

We are currently experiencing this issue. It's very annoying.

[DmitriyLyalyuev]

Maybe this will help to someone.
Resolved with:

    volumes:
      - ./app:/app
      - ./shared:/shared:delegated
    command: bash -c "ln -sf /shared /app/shared && ./manage.py runserver 0.0.0.0:3001"

It mounts shared not as subvolume but as separate volume and creates symlink inside app volume to shared.

[Ralle]

Just experienced this on a server in production.

Docker version 20.10.21, build baeda1f

docker-compose version 1.25.0, build unknown

# cat /etc/*release*
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.5 LTS"
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

[ndeloof]

@Ralle Docker compose v1 is deprecated and will reach End of Life in a few month, you should consider adoption of Compose v2 for your production server(s).

[Ralle]

Ouch. Yeah. Thanks.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.