Building with --attest=type=sbom --tag <tag> --push results in non-reclaimable layers

[pavolmarko]

TLDR:
When using
docker buildx build --builder buildkit17 --tag <registry:tag> --push --attest=type=sbom .
non-reclaimable layers remain on the buildkit instance. buildctl prune or buildctl prune --all don't remove those.

My expectation was that after the build has finished, it should be possible to garbage-collect all corresponding layers.

Repro:
Running with buildkit 0.17.0 (moby/buildkit container):

$ buildctl --addr unix:///${HOME}/buildkit/buildkitd.sock debug info
BuildKit: github.com/moby/buildkit v0.17.0 fd61877fa73693dcd4ef64c538f894ec216409a3

$ docker buildx create --name buildkit17 --driver remote unix:///${HOME}/buildkit/buildkitd.sock

$ docker buildx inspect buildkit17
Name:          buildkit17
Driver:        remote
Last Activity: 2024-11-07 13:16:56 +0000 UTC

Nodes:
Name:             buildkit170
Endpoint:         unix:///home/pmarko/buildkit/buildkitd.sock
Status:           running
BuildKit version: v0.17.0
Platforms:        linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
Labels:
 org.mobyproject.buildkit.worker.executor:         oci
 org.mobyproject.buildkit.worker.hostname:         c0ff2ad9381b
 org.mobyproject.buildkit.worker.network:          host
 org.mobyproject.buildkit.worker.oci.process-mode: sandbox
 org.mobyproject.buildkit.worker.selinux.enabled:  false
 org.mobyproject.buildkit.worker.snapshotter:      native
GC Policy rule#0:
 All:           false
 Filters:       type==source.local,type==exec.cachemount,type==source.git.checkout
 Keep Duration: 48h0m0s
GC Policy rule#1:
 All:           false
 Keep Duration: 1440h0m0s
 Keep Bytes:    9.313GiB
GC Policy rule#2:
 All:        false
 Keep Bytes: 9.313GiB
GC Policy rule#3:
 All:        true
 Keep Bytes: 9.313GiB

The current directory has a Dockerfile with the contents

FROM alpine

Then performing the build:

$ docker buildx build --builder buildkit17 --tag <my-registry>/test/pavol_test:latest --push --attest=type=sbom .
[+] Building 14.0s (9/9) FINISHED                                                                                                                                                                                                                                         remote:buildkit17
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                   0.0s
 => => transferring dockerfile: 49B                                                                                                                                                                                                                                                    0.0s
 => resolve image config for docker-image://docker.io/docker/buildkit-syft-scanner:stable-1                                                                                                                                                                                            1.6s
 => [internal] load metadata for docker.io/library/alpine:latest                                                                                                                                                                                                                       0.9s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                        0.0s
 => [1/1] FROM docker.io/library/alpine:latest@sha256:beefdbd8a1da6d2915566fde36db9db0b524eb737fc57cd1367effd16dc0d06d                                                                                                                                                                 0.9s
 => => resolve docker.io/library/alpine:latest@sha256:beefdbd8a1da6d2915566fde36db9db0b524eb737fc57cd1367effd16dc0d06d                                                                                                                                                                 0.0s
 => => sha256:43c4264eed91be63b206e17d93e75256a6097070ce643c5e8f0379998b44f170 3.62MB / 3.62MB                                                                                                                                                                                         0.7s
 => => extracting sha256:43c4264eed91be63b206e17d93e75256a6097070ce643c5e8f0379998b44f170                                                                                                                                                                                              0.1s
 => docker-image://docker.io/docker/buildkit-syft-scanner:stable-1                                                                                                                                                                                                                     2.4s
 => => resolve docker.io/docker/buildkit-syft-scanner:stable-1                                                                                                                                                                                                                         0.9s
 => => sha256:8f55b7fda2c88820456a8687c5a0032f59bc1247451cfdbc968d773124f5da01 24.35MB / 24.35MB                                                                                                                                                                                       1.3s
 => => extracting sha256:8f55b7fda2c88820456a8687c5a0032f59bc1247451cfdbc968d773124f5da01                                                                                                                                                                                              0.2s
 => [linux/amd64] generating sbom using docker.io/docker/buildkit-syft-scanner:stable-1                                                                                                                                                                                                4.9s
 => exporting to image                                                                                                                                                                                                                                                                 4.1s
 => => exporting layers                                                                                                                                                                                                                                                                0.0s
 => => exporting manifest sha256:465f515dfba9a241c3af16b4154e4161db1e8de1055b753434874158e7ae05d9                                                                                                                                                                                      0.0s
 => => exporting config sha256:e75ff1d806b99c41ff0f797f6800a85546d5ebf447d69796ec1db166809a41a2                                                                                                                                                                                        0.0s
 => => exporting attestation manifest sha256:a621b2141dc5f638852e2f60471f6f235123f4a2f33ede0da4b7cda6b864bd11                                                                                                                                                                          0.0s
 => => exporting manifest list sha256:6b12b4f9131ee414e0fda5d3fc60fcfe1e58149043e424a6fdd7b04621275dba                                                                                                                                                                                 0.0s
 => => pushing layers                                                                                                                                                                                                                                                                  2.8s
 => => pushing manifest for <my-registry>/test/pavol_test:latest@sha256:6b12b4f9131ee414e0fda5d3fc60fcfe1e58149043e424a6fdd7b04621275dba                                                                                                                          1.2s
 => [auth] sharing credentials for <my-registry>
0.0s

After this, examining disk usage on the buildkit side:

$ buildctl --addr unix:///home/pmarko/buildkit/buildkitd.sock du -v
...

ID:             38amqtb13xmthi1uyb4ya8av4
Created at:     2024-11-07 13:20:33.038863904 +0000 UTC
Mutable:        false
Reclaimable:    false
Shared:         false
Size:           4.15MB
Description:    pulled from docker.io/library/alpine:latest@sha256:beefdbd8a1da6d2915566fde36db9db0b524eb737fc57cd1367effd16dc0d06d
Usage count:    1
Last used:      2024-11-07 13:20:44.491769216 +0000 UTC
Type:           regular

...

This is marked as "Reclaimable: false". Consequently, buildctl prune does not remove this layer. Same for buildctl prune --all.

Note that running

$ docker buildx build --builder buildkit17 --tag <my-registry>/test/pavol_test:latest --push .

or

$ docker buildx build --builder buildkit17 --attest=type=sbom .

does not result in this issue - all layers are Reclaimable after that and prune can remove them.

It looks like this only happens when pushing to a registry and using --attest.

Other notes:

• Also observed on 0.16.0.
• This could be related to Some records seem to never become reclaimable #4525 .