Compose doesn't reuse anonymous volumes when recreating a container using an image generated by BuildKit

[jpetazzo]

Description of the issue

On some of my services, when I make a minor change to the service configuration (for instance, changing an environment variable or a label in the Compose file), Compose recreates the service (which is expected) but it also creates new volumes, instead of reusing the existing volumes.

After trying to reproduce the problem, I noticed that it happened only on images generated with BuildKit; probably because of a difference between the Config and the ContainerConfig fields in the image manifest.

Context information

Docker Compose version 1.29.2
Docker Engine 20.10.8

Problem initially identified with Docker Compose 1.24-1.25 as well as Docker Engine 20.10.7

Output of docker version

$ docker version
Client:
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d28e
 Built:             Wed Aug  4 10:59:01 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d88bc
  Built:            Wed Aug  4 10:58:48 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.5.5
  GitCommit:        72cec4be58a9eb6b2910f5d10f1c01ca47d231c0.m
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Steps to reproduce the issue

Create a directory with the following Dockerfile and Compose file:

mkdir -p /tmp/volume-issue && cd /tmp/volume-issue

cat >Dockerfile <<EOF
FROM busybox
VOLUME /volume
EOF

cat >docker-compose.yml <<EOF
version: "3"
services:
  test:
    build: .
    environment:
    - VAR
EOF

Now check how many volumes we have; build without BuildKit; and up the service a couple of times:

docker volume ls | wc -l

# Start the service a first time
DOCKER_BUILDKIT=0 VAR=1 docker-compose up --build

# We should now have one more volume, as expected
docker volume ls | wc -l

# Start the service again with a different configuration
DOCKER_BUILDKIT=0 VAR=2 docker-compose up

# We should have the same number of volumes (Compose reuses the volume, as expected)
docker volume ls | wc -l

Now let's do the same with BuildKit:

# Rebuild the service with BuildKit and start it again
DOCKER_BUILDKIT=1 VAR=3 docker-compose up --build

# We still have the same number of volumes; so far so good
docker volume ls | wc -l

# Restart the service one more time with a different configuration
DOCKER_BUILDKIT=1 VAR=4 docker-compose up

# We have an extra volume; that's unexpected
docker volume ls | wc -l

We can repeat these last couple of steps multiple times (changing the value of VAR to force Compose to recreate the container) and each time a new volume is created.

Observed result

Compose creates a volume each time we change the value of VAR.

Expected result

Compose should not create a volume when we change the value of VAR.

Stacktrace / full error message

N/A

Additional information

See repro steps above.

When building with BuildKit, it looks like Config and ContainerConfig are different in the resulting image:

$ docker inspect volume-issue_test | jq .[0].Config.Volumes
{
  "/volume": {}
}
[jp@большой (⎈ |kind-kind:default) volume-issue]
$ docker inspect volume-issue_test | jq .[0].ContainerConfig.Volumes
null

When building without BuildKit, they're the Volumes section is identical.

[jpetazzo]

Just in case you have the same (or a similar) issue and wonder how to work around it: just define the volumes explicitly in the Compose file; then Compose will do the right thing (and use your explicit volumes). If your app is already running with "anonymous" volumes and you need to update the container while retaining the volumes: identify the volume(s) that your app is using (with their long Docker ID, visible in e.g. docker volume ls or if you docker inspect the app container) then add that volume ID as the volume name in the Compose file and declare that volume as an external volume. Compose will then keep using that volume when you update the container. Voilà! ✨

[ndeloof]

I tried to reproduce on Compose v2:

$ VAR=1 docker compose up --build -d
[+] Building 0.1s (5/5) FINISHED                                                                                                                                               
 => [internal] load build definition from Dockerfile                                                                                                                      ...
[+] Running 1/1
 ⠿ Container truc-test-1  Started                                                                                                                                         

$ VAR=2 docker compose up --build --force-recreate -d
[+] Building 0.2s (5/5) FINISHED                                                                                                                                               
 => [internal] load build definition from Dockerfile                                                                                                                      ...
[+] Running 1/1
 ⠿ Container truc-test-1  Started   

Whatever change I apply to service configuration, I still get the exact same unique volume on my system attached to my compose container

$ docker volume ls
DRIVER    VOLUME NAME
local     6c59f190ea03e51dc4644e7a7cfaf50d3c327387069b9e0f9d5acbaef7f200bd

Would you have a chance to give Compose V2 a try?

[thaJeztah]

When building with BuildKit, it looks like Config and ContainerConfig are different in the resulting image

I think that part is expected; (also see docker/cli#2868) ContainerConfig is mostly an artifact from the classic way of building images (create a container for each step, and commit the result); the "Config" is the result of the step, and "ContainerConfig" what happened to be used to create it (I guess in the past we never bothered with cleaning up those bits in the JSON).

BuildKit only uses a container for certain kind of operations, and for metadata-only changes will apply those changes directly (without spinning up a container for that).

Not sure what's causing the volume issue though (perhaps compose is looking at the "wrong" bits in the image JSON?)

[jpetazzo]

I tried to run that stack with Compose v2, but unfortunately, I ran into another issue (#8741).
While trying to reproduce the issue described in #8741, I ran into another issue (#8740).
Which may or may not be related to #8626.
Hopefully all the issues will go away when the .env parsing library gets fixed 😅

... Meanwhile, I still haven't had the time to test the original stack with Compose v2, but I hope to give it a try soon.
(En espérant que je tombe pas sur 3 autres bugs vicelards au passage, ça serait vraiment pas de bol!)

Merci ! ♥

[jpetazzo]

I tried that Compose file with Compose v2, and when I make changes to the service that uses volumes, the volumes are properly carried over. So it looks like this is specific to Compose v1!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it had not recent activity during the stale period.

[cluick]

Unfortunately this problem caused a lot of trouble for me and led to some serious data losses in an old Docker-Compose v1 based demo infrastructure which I´m responsible for. May it be possible to extend Docker-Compose to at least refuse to work with the combination of Docker Compose v1 and Buildkit Image and fail early instead of creating a setup that is inevitably subject to data loss? As Buildkit is the new default in Docker 23 (https://docs.docker.com/engine/release-notes/23.0/#new), i guess this problem will arise in a lot of old installations.

[glours]

@cluick Compose V1 is EOL, you should switch to Compose V2
https://www.docker.com/blog/new-docker-compose-v2-and-v1-deprecation/

[cluick]

@cluick Compose V1 is EOL, you should switch to Compose V2 https://www.docker.com/blog/new-docker-compose-v2-and-v1-deprecation/

Yeah, you are right, and this is what I will do. But I would prefer if Docker-Compose would refuse to work in this combination and fail early instead of corrupting the compose setup.