dockerfile:1.7-labs: COPY fails where there exists broken symbolic link and --exclude is specified

[dyume99]

Overview:

Hi, i've found that COPY instruction fails where there exists broken symbolic link and --exclude is specified. Wondering if it's a bug.

How to repoduce:

Dockerfile:

# syntax=docker/dockerfile:1.7-labs

FROM alpine as stage0
RUN mkdir /test && ln -s /test/terget /test/link

FROM alpine 
COPY --from=stage0 --exclude=*.git /test /test

build:

buildctl --addr tcp://0.0.0.0:1234 build  --frontend dockerfile.v0  --local context=. --local dockerfile=.

docker build can also repoduce:

docker build .

Result:

[+] Building 1.9s (8/8) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                     0.0s
 => => transferring dockerfile: 208B                                                                                                                                                                                                                                     0.0s
 => resolve image config for docker.io/docker/dockerfile:1.7-labs                                                                                                                                                                                                        1.2s
 => CACHED docker-image://docker.io/docker/dockerfile:1.7-labs@sha256:b99fecfe00268a8b556fad7d9c37ee25d716ae08a5d7320e6d51c4dd83246894                                                                                                                                   0.0s
 => => resolve docker.io/docker/dockerfile:1.7-labs@sha256:b99fecfe00268a8b556fad7d9c37ee25d716ae08a5d7320e6d51c4dd83246894                                                                                                                                              0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                                                                                                                                                                         0.4s
 => [internal] load .dockerignore                                                                                                                                                                                                                                        0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                          0.0s
 => CACHED [stage0 1/2] FROM docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd                                                                                                                                     0.0s
 => => resolve docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd                                                                                                                                                   0.0s
 => CACHED [stage0 2/2] RUN mkdir /test && ln -s /test/terget /test/link                                                                                                                                                                                                 0.0s
 => ERROR [stage-1 2/2] COPY --from=stage0 --exclude=*.git /test /test                                                                                                                                                                                                   0.0s
------
 > [stage-1 2/2] COPY --from=stage0 --exclude=*.git /test /test:
------
Dockerfile:7
--------------------
   5 |
   6 |     FROM alpine
   7 | >>> COPY --from=stage0 --exclude=*.git /test /test
   8 |
--------------------
error: failed to solve: failed to compute cache key: failed to calculate checksum of ref 6w8ichx5f65vnre17ghbupzxu::u36c25l8glxphd3ebgyvnb820: "/test/terget": not found

It should be:

If i remove the --exclude flag, then the build will succeed without error:

[+] Building 1.9s (8/8) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                     0.0s
 => => transferring dockerfile: 192B                                                                                                                                                                                                                                     0.0s
 => resolve image config for docker.io/docker/dockerfile:1.7-labs                                                                                                                                                                                                        1.3s
 => CACHED docker-image://docker.io/docker/dockerfile:1.7-labs@sha256:b99fecfe00268a8b556fad7d9c37ee25d716ae08a5d7320e6d51c4dd83246894                                                                                                                                   0.0s
 => => resolve docker.io/docker/dockerfile:1.7-labs@sha256:b99fecfe00268a8b556fad7d9c37ee25d716ae08a5d7320e6d51c4dd83246894                                                                                                                                              0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                                                                                                                                                                         0.4s
 => [internal] load .dockerignore                                                                                                                                                                                                                                        0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                          0.0s
 => [stage0 1/2] FROM docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd                                                                                                                                            0.0s
 => => resolve docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd                                                                                                                                                   0.0s
 => CACHED [stage0 2/2] RUN mkdir /test && ln -s /test/terget /test/link                                                                                                                                                                                                 0.0s
 => CACHED [stage-1 2/2] COPY --from=stage0 /test /test

[jsternberg]

Currently investigating this but I think --exclude may be tangential to the problem. I suspect that the problem will also happen if there's a wildcard or include.

buildkit/cache/contenthash/checksum.go

Lines 409 to 411 in 265d18f

	if !opts.Wildcard && len(opts.IncludePatterns) == 0 && len(opts.ExcludePatterns) == 0 {
	return cc.checksumFollow(ctx, m, p, opts.FollowLinks)
	}

Haven't tested either yet though.

[jsternberg]

Seems to be an issue with checksumFollow when the original path is a broken symlink. If you attempt to copy the broken symlink directory with nothing else, it also breaks with the same error message. The reason why the COPY --from=stage0 /test /test seems to work is because it's computing the cache key using only /test, but if you modify it to point directly to the broken symlink it breaks.

The reason why this seems to affect --exclude is because exclude is forcing the checksum to compute which paths it's allowed to follow and the broken symlink just happens to be included.

We likely just need to avoid following the symlink if there's an error with reading it.

[jsternberg]

@aaronlehmann I was wondering if you had any opinions on what the behavior of this should be.

To put the problem simply, attempting to compute the checksum of a broken symlink results in an error when it tries to follow the symlink. The --exclude causes the broken symlink to be part of the checksum computation here:

buildkit/cache/contenthash/checksum.go

Line 413 in 7419743

includedPaths, err := cc.includedPaths(ctx, m, p, opts)

My thought is to cause broken symlinks fallback to being recorded as themselves rather than following the path when the symlink is broken. This will cause the symlink to be checksummed without an error and an error will get returned later if we attempt to copy a directory that's from a broken symlink.

[aaronlehmann]

That sounds reasonable to me.

[jsternberg]

The current plan is to fix this after #4896 has been merged.

[ericsmalling]

Just ran into this bug too. Now that #4896 is merged, any plans to tackle this?

[jsternberg]

I'm planning to tackle this but #4896 made the logic different so it wasn't a simple rebase and I got pulled into some other work in the interim. I am planning to start working on this again soon.

[gavinltomra]

This also triggers if --parents is used, even without --exclude. Using neither flag results in a successful copy when broken symlinks are present.

See also this MCVE with --exclude.