github action cache error when pushing cache

[txomon]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

When building images and pushing the build cache to github actions cache in mode=max, there are many times similar but different errors:

1. Error: buildx failed with: ERROR: failed to solve: error writing layer blob: Patch "https://artifactcache.actions.githubusercontent.com/..../_apis/artifactcache/caches/11111": read tcp 172.17.0.2:40814->142.250.141.82:443: use of closed network connection seems to come from Long running push sometimes fail with "use of closed network connection" #367
2. Gets stuck in exporting layer

And trust me, it really gets stuck:

Expected behaviour

There shouldn't be any timeouts or reuse of a closed connection.

Actual behaviour

It works unpredictably

Buildx version

github.com/docker/buildx 0.10.4+azure-1 c513d34

Docker info

Client:
   Context:    default
   Debug Mode: false
   Plugins:
    buildx: Docker Buildx (Docker Inc., 0.10.4+azure-1)
    compose: Docker Compose (Docker Inc., 2.17.2+azure-1)
  
  Server:
   Containers: 0
    Running: 0
    Paused: 0
    Stopped: 0
   Images: 18
   Server Version: 20.10.23+azure-2
   Storage Driver: overlay2
    Backing Filesystem: extfs
    Supports d_type: true
    Native Overlay Diff: false
    userxattr: false
   Logging Driver: json-file
   Cgroup Driver: cgroupfs
   Cgroup Version: 2
   Plugins:
    Volume: local
    Network: bridge host ipvlan macvlan null overlay
    Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
   Swarm: inactive
   Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
   Default Runtime: runc
   Init Binary: docker-init
   containerd version: 1e1ea6e986c6c86565bc33d52e34b81b3e2bc71f
   runc version: f19387a6bec4944c770f7668ab51c4348d9c2f38
   init version: 
   Security Options:
    apparmor
    seccomp
     Profile: default
    cgroupns
   Kernel Version: 5.15.0-1035-azure
   Operating System: Ubuntu 22.04.2 LTS
   OSType: linux
   Architecture: x86_64
   CPUs: 2
   Total Memory: 6.781GiB
   Name: fv-az251-622
   ID: 3Q5N:TB6N:KRBU:S5UK:CFVR:VASN:GQOQ:7YNN:ZAWD:NZJQ:ZO4T:DFAG
   Docker Root Dir: /var/lib/docker
   Debug Mode: false
   Username: githubactions
   Registry: https://index.docker.io/v1/
   Labels:
   Experimental: false
   Insecure Registries:
    127.0.0.0/8
   Live Restore Enabled: false

Builders list

The one created by the github action:

docker/setup-buildx-action@v2

Configuration

# Github actions yaml
      - name: Build docker dev
        uses: docker/build-push-action@v4
        with:
          # load: true makes the image available to docker compose
          load: true
          tags: |
            my-image:latest
          # temp workaround for newer buildx where images outputed by default are not cloud run compatible https://github.com/docker/buildx/issues/1533
          provenance: false
          cache-from: |
            type=gha,scope=my-image/${{ github.base_ref }}
            type=gha,scope=my-image/${{ github.ref_name }}
            type=gha,scope=my-image/dev-${{ github.base_ref }}
            type=gha,scope=my-image/dev-${{ github.ref_name }}
            type=registry,ref=registry.url/cache/my-image:latest
            type=registry,ref=registry.url/cache/my-image:newest
            type=registry,ref=registry.url/cache/my-image:dev-latest
            type=registry,ref=registry.url/cache/my-image:dev-newest
          cache-to: |
            type=gha,scope=my-image/dev-${{ github.ref_name }},mode=max
            type=registry,ref=registry.url/cache/my-image:dev-newest,mode=max

Build logs

#33 exporting to GitHub cache
#33 preparing build cache for export 6.1s done
#33 writing layer sha256:aaaaaaaaaaaaaaaaaaaaaaaaa 1.3s done
#33 writing layer sha256:bbbbbbbbbbbbbbbbbbbbbbbbb 1.2s done
#33 writing layer sha256:ccccccccccccccccccccccccc 0.6s done
#33 writing layer sha256:ddddddddddddddddddddddddd 2.6s done
#33 ERROR: error writing layer blob: Patch "https://artifactcache.actions.githubusercontent.com/.................../_apis/artifactcache/caches/13632": read tcp 172.17.0.2:36796->142.251.163.82:443: use of closed network connection

Additional info

It seems like most of the trouble is in github actions cache

[Roguelazer]

This has been happening today on all of my repos, the full error looks like

ERROR: failed to solve: error writing layer blob: Patch "https://acghubeus2.actions.githubusercontent.com/<redacted>/_apis/artifactcache/caches/<redacted>": read tcp 172.17.0.2:47880->20.209.227.129:443: use of closed network connection

any way to tell whether this is a github issue or an issue with this action? none of the other actions that use actions caches have had problems.

For now, I'm periodically deleting all the caches, which seems to fix it for a while.