Add bind mount exclusion to prevent overwriting generated files with local files

[steevehook]

It's always annoying when I want to create a bind mount in order to synchronize local source code I'm editing, with code from docker container and all of the generated files.

Whether it's vendor directory or a binary file which eventually is used as an ENTRYPOINT or CMD they are getting removed when the bind mount is done. And since these files/directories are not available on local machine to avoid copying different binaries for different OS & ARCH inside linux container, docker removes them inside container as well when binding the source from local with the one from container

A solution to this might be to to do the bind mount first then use the snapshot fs which was created when the image was build, which will first bind mount the sources then allow the scripts which generate the binaries/directories to execute without being removed

Another solution would be to indicate in the volumes section some sort of exclude list when binding mount for the first time, somethings that would look like this:

volumes:
      - .:/usr/app:
          exclude:
            - fileA
            - fileB
            - dirA/

As a workaround to avoid this situation with volumes instead of running the generated binary in CMD or ENTRYPOINT we run some sort of shell script which generates again the binary and then executes the binary and now the CMD/ENTRYPOINT is referencing the shell script.
However would be a nice option with for initial bind mount some files which we choose would not be overwritten/removed in order to sync with local source code

docker-compose.yaml

version: "3.7"

services:
  redis:
    image: redis
    restart: always
    volumes:
      - redis-data:/data
    networks:
      - visits
    command: redis-server --appendonly yes
  visits-app:
    build: .
    restart: always
    ports:
      - "8080:8080"
    volumes:
      - ./:/usr/app
    networks:
      - visits
    

    # if this custom command is missing then we get an error that there is no node_modules
    # but in Dockerfile we indeed genereated node_modules by running npm install
    command: sh /usr/app/init.sh

networks:
  visits:

volumes:
  redis-data:

init.sh

npm install
npm start

Dockerfile

FROM node:alpine

ENV PORT 8080

EXPOSE $PORT

WORKDIR /usr/app

COPY ./package.json ./
RUN npm install
COPY . .

CMD npm start

Thank you for the awesome project and let me know if such a feature is a valid one and does not come against docker philosophy or if you need any help in implementing it

[ulyssessouza]

Thanks for the suggestion @steevehook !

One thing you can do is restructuring your source tree so that you can mount (in docker-compose.yml:18) just important part which is, lets say a src folder. Leading to something like ./:/usr/app/src in the volume mounting line.

I think implementing volume exclusions could lead to awkward corner cases.

If the mounting is for a performance issue then, looking at your Dockerfile, if you use buildkit your image layers should be cached until the RUN npm install.

[montanaflynn]

The biggest problem that this would alleviate is for node_modules.

There are countless stackoverflow threads and blog posts about this, here's a few:

https://stackoverflow.com/questions/29181032/add-a-volume-to-docker-but-exclude-a-sub-folder
https://burnedikt.com/dockerized-node-development-and-mounting-node-volumes/

The basic gist is that when doing local development you want to have the files except for node modules shared between host and container. The docker image already installs the dependencies for the linux version, which is then overwritten with either an empty node_modules or in some cases node_modules for a different OS like windows or mac:

  example:
    build: example
    volumes:
      - ./example:/usr/app
    ports:
      - "8080:3000"

The common workaround is a named volume:

volumes:
  example_node_modules:

services:
  example:
    build: example
    volumes:
      - ./example:/usr/app
      - example_node_modules:/usr/app/node_modules/
    ports:
      - "8080:3000"

The big downside is then you have some extra volume to deal with and remember to remove when updating dependencies.

[adamkdean]

This would be a great feature to add, something we've lacked for almost a decade now. There was a workaround with symlinks and a second container but, sadly, as of 2020 that has also stopped working.

[malachid]

I'd second this. We are currently mounting our local work directory into the docker container so we can do some validations before pushing them up to Jenkins. Unfortunately, there is 1 file that has to be different locally vs in the container (local.properties) and as such we have had to resort to using the entrypoint to symlink every file/directory once mounted then remove that entry. That was working fine until AGP (Android Gradle) 7 which no longer works correctly with symlinks. As such, I'm once again looking for a workaround to the missing exclude file option.

[pcjmfranken]

There seem to be many posts and comments that are essentially requesting this exact same feature, describe a problem that this feature would solve, or they're running into problems with the commonly suggested 'submount' workaround. All these discussions are using different wording/terminology, and they're scattered across multiple Docker repositories, the legacy Docker forum, StackOverflow, and various other places.

Just a few in random order:

• [Node.js docs feedback] Install dependencies inside container docs#13227
• Volume mounted inside another sometimes disappears after file change #6975
• Nested mounts disappear for-mac#3479
• Volume ownership on creation inconsistent between docker and docker-compose #8002
• Add bind mount exclusion to prevent overwriting generated files with local files #6470
• Ability to exlude certain files and subfolders from being mounted/synced. #6997
• Missing documentation on how named volumes works in docker docs#11193
• adding readonly flag to bind mount failing anonymous volume creation awesome-compose#199
• docker-compose run without mounting volume #1682
• node_module's content is hidden on Ubuntu #7858
• File access in mounted volumes extremely slow for-mac#77
• https://stackoverflow.com/questions/29181032/add-a-volume-to-docker-but-exclude-a-sub-folder
• https://stackoverflow.com/questions/68325244/docker-compose-volumes
• https://stackoverflow.com/questions/53570499/make-docker-compose-ignore-folder-within-volume
• https://stackoverflow.com/questions/57512490/create-dockers-own-node-modules

The ability to (dynamically) exclude specific bind mount contents (like any and all node_modules or cache directories) from existing on the host filesystem would be very nice for DevContainer workflows. There are too many caching/performance/permission problems with the common workaround, plus having to manage all the exclusions in the composefile is such a hassle and on top of that very error-prone.

Maybe someone from the Docker team could create a new Docker Roadmap item so we have a central place for reference and discussion?

[ndeloof]

bind mount is managed by docker engine, not compose. And those are actually Linux features to define a mount for container-visible filesystem to access some parts of host's filesystem. We only expose the available lower-level features. Linux mounts does not offer anything like "exclusions" we could rely on.

[ndeloof]

Closing as there's nothing we can offer here from Docker Compose, until Docker engine offers such an "exclude" feature for bind mounts.

[nullbio]

Is there a way to do this yet?

[ndeloof]

@nullbio there's no such thing as a "mount exclusion" in docker engine API (nor linux mount used under the hood AFAICT)