Random 403 errors when pushing to GHCR

[jauderho]

Troubleshooting

Pushing to GHCR seems to randomly resulting in 403s in the last 24 hours. Rerunning the GH action seems to resolve things (partially).

Behaviour

See above.

Failure message : buildx failed with: error: failed to solve: unexpected status: 403 Forbidden

Steps to reproduce this issue

1. Kick off GH action to rebuild Docker images
2. See https://github.com/jauderho/dockerfiles/actions
3. Check one of the recent failed runs. For example, https://github.com/jauderho/dockerfiles/actions/runs/1255324536

Expected behaviour

Builds should complete with no issue.

Actual behaviour

See https://github.com/jauderho/dockerfiles/actions/runs/1255324536

Configuration

• Repository URL (if public): https://github.com/jauderho/dockerfiles
• Build URL (if public): https://github.com/jauderho/dockerfiles/actions/runs/1255324536

https://github.com/jauderho/dockerfiles/blob/main/.github/workflows/tailscale.yml

[jauderho]

To be clear, I have multiple actions that each build docker images for different apps failing, see https://github.com/jauderho/dockerfiles

I currently count 5 so it does not appear to an isolated issue.

[revolunet]

same issue here, not on all images but some and its quite systematic

[crazy-max]

@markphelps Looks similar to #251

[jauderho]

Just saw a suggestion on the orange site: https://news.ycombinator.com/item?id=28607735

I'm trying it out now and will report back.

[crazy-max]

As the ephemeral secrets.GITHUB_TOKEN would be invalidated when the job is completed, I wonder if there is a limited token pool per repo? Any idea @chrispat?

We recently faced this; if you are using the docker/login action I'd give that a check as it turned out it was logging us out by default at the end of each job; resulting in some race conditions when running multiple runners on the same machine (sharing the same docker daemon).

As I can see you're not using a self-hosted runner so it's not your case.

Simple fix was to add logout: false to the action options.

Don't think it will change smth as docker logout only erases token from the credential store.

[jauderho]

Nope, it did not work. Still fails.

[onedr0p]

Also facing this issue, it seems to happen intermittently but more times than not.

[onedr0p]

I have verified the following steps work for me to resolve the issue, however, this is not ideal.

• Delete the image from your image packages that has the 403 problem.
• Trigger the CI to build and push the image again.

Maybe this can shed a light into what's happening @chrispat ?

Build that failed:
https://github.com/k8s-at-home/container-images/runs/3656105847

Build that passed after deleting the image from GHCR:
https://github.com/k8s-at-home/container-images/runs/3674834677

[onedr0p]

I have also tried toggling Inherit access from source repository (recommended) and toggling the image visibility between Public and Private to no avail. This is definitely not an issue we cannot resolve on our end besides completely deleting the image and pushing it again.

[ondrejmirtes]

I'm experiencing the same problem. I'm logging with docker-login-action:

      - name: Login to ghcr
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ secrets.GITHUB_TOKEN }}

And trying to push with this action:

      - name: Build and push
        uses: docker/build-push-action@v2
        with:
          builder: ${{ steps.buildx.outputs.name }}
          context: ./docker
          file: ./docker/Dockerfile
          build-args: |
            PHPSTAN_VERSION=dev-master#${{ github.sha }}
          push: true
          tags: ghcr.io/phpstan/phpstan:nightly
          platforms: linux/amd64,linux/arm64
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,dest=/tmp/.buildx-cache

This is how the output tail looks like:

#27 [auth] phpstan/phpstan:pull,push token for ghcr.io
#27 DONE 0.0s

#28 [auth] phpstan/phpstan:pull,push token for ghcr.io
#28 DONE 0.0s

#29 [auth] phpstan/phpstan:pull,push token for ghcr.io
#29 DONE 0.0s

#26 exporting to image
#26 pushing layers 0.5s done
#26 ERROR: unexpected status: 403 Forbidden
------
 > exporting to image:
------
error: failed to solve: unexpected status: 403 Forbidden
Error: buildx failed with: error: failed to solve: unexpected status: 403 Forbidden

[onedr0p]

After a couple more hours of debugging I fixed it with this under the package settings, or the problem resolved itself.

It doesn't make sense why this would be needed for one image package in my repo and not the other image packages in the same repo.

[jauderho]

@onedr0p How are you getting to that dialog? My google-fu is failing me today and I am unable to find the location so that I can check that setting. Thanks.

[revolunet]

You can find it in https://github.com/orgs/[XXX]/packages/container/[YYY]/settings

or by going to the package page then "package settings"

[jauderho]

I can confirm that @onedr0p 's solution seems to work. I have now rebuilt two images with no issue and are rerunning jobs on my remaining failed actions.

To reiterate, the steps are:

1. Go to the packages page
2. Pick a package and set the following (eg https://github.com/users/jauderho/packages/container/dnscrypt-proxy/settings)
3. "Add a repository" - in my case, it is jauderho/dockerfiles and enable write access
4. Enable "Inherit access from source repository (recommended)"

What's odd is that as I went through and checked the settings for all of my packages (21!), it seemed some were already set this way and seem to be the ones not having issues.

I wonder if there was some backend maintenance and somehow these settings were lost. Can someone from GitHub investigate and confirm?

Update: All of my images have successfully built.

[filipesilva]

I'm also seeing this problem on push to a PR, but not on push for tags. So it might be somehow related to the metadata, which is different between PR push and tag?

In my case these were the tags for a build that didn't work:

Docker tags
  ghcr.io/athensresearch/athens-server:pr-1684

but these tags worked:

Docker tags
  ghcr.io/athensresearch/athens-server:mock-v0.1
  ghcr.io/athensresearch/athens-server:latest

In my case the image did not exist at all to begin with, so I couldn't follow the steps in #463 (comment).

[crazy-max]

@filipesilva

on push to a PR

Not related to that issue but check permissions of the GITHUB_TOKEN in your workflow on PR. Afaik it will not work without a pull_request_target event but be very careful with it.

---

Closing this issue as it seems to be solved.

[jrschumacher]

Switching to a PAT worked for me. I used this action and redhat-actions/buildah-build and redhat-actions/push-to-registry and both resulted in a 403. Even with adding the repo to the package, granting explicit access, etc.

Once I returned to a PAT it worked flawlessly.

[joostschriek]

I think this 403 has to do with how the github packages registry resolves. Using any varient of ghcr.io just in the tags or as the registry property doesn't work. Using the registry: docker.pkg.github.com how ever solves my issues.

[rupe120]

I can confirm that @onedr0p 's solution seems to work. I have now rebuilt two images with no issue and are rerunning jobs on my remaining failed actions.

To reiterate, the steps are:

1. Go to the packages page

2. Pick a package and set the following (eg https://github.com/users/jauderho/packages/container/dnscrypt-proxy/settings)

3. "Add a repository" - in my case, it is jauderho/dockerfiles and enable write access

4. Enable "Inherit access from source repository (recommended)"

What's odd is that as I went through and checked the settings for all of my packages (21!), it seemed some were already set this way and seem to be the ones not having issues.

I wonder if there was some backend maintenance and somehow these settings were lost. Can someone from GitHub investigate and confirm?

Update: All of my images have successfully built.

I'm feeling dumb. Where is the packages page ?

[filipesilva]

@rupe120 it's on the user/org page. In your case:

4th tab on the top. https://github.com/rupe120?tab=packages

[snowdream]

After a couple more hours of debugging I fixed it with this under the package settings, or the problem resolved itself.

It doesn't make sense why this would be needed for one image package in my repo and not the other image packages in the same repo.

It works,Thank you.

[JoshMcCullough]

What's odd is that as I went through and checked the settings for all of my packages (21!), it seemed some were already set this way and seem to be the ones not having issues.

I read somewhere in the GHA docs that if you publish a package via a workflow, but that workflow did not initially create the package, then it "would" or "could" fail -- don't remember which. I believe it had to do with how the package is linked to the repo. I tried to find these docs again but could not.

[ChristopherRabotin]

This issue should seriously be reopened. I've been struggling trying to fix this for at least three hours now, and I'm well versed in both Github actions and Docker.

I have no packages... so I can't access the basically hidden "package settings" link: for people reading this, you must click on any package that is listed if you have any, scroll two thirds down the page and you'll see on the right a "package settings" link.

My repo allows for all github actions to be executed.

And the GITHUB TOKEN has all permissions

So yup, here I am unable to figure out why this isn't working but really, this is a Github UX and Docker action problem.

[mdsitton]

Yeah i'm also having the same issue as @ChristopherRabotin how do we create a new package? When trying to push a new package that doesn't exist for the first time it just errors...

[ChristopherRabotin]

@mdsitton what's your yaml like? Despite the permissions above, I had to add the write permissions too : https://github.com/nyx-space/nyx/blob/master/.github/workflows/python.yml#L153

[captainhook]

@mdsitton what's your yaml like? Despite the permissions above, I had to add the write permissions too : https://github.com/nyx-space/nyx/blob/master/.github/workflows/python.yml#L153

Probably not ideal to have write-all permission. I'm still seeing this very issue today.

[tfrijsewijk]

I ended up deleting the package:

Instant satisfaction 🤷

[phyzical]

thx @onedr0p (small world)

If you face this issue either someone already created a package with this name so it doesn't "inherit" from the repo or you may have inherit permissions off on the org/repo so you just have to do it yourself

[jb747]

Deleting the images created using another token worked for me.

[H-ADJI]

as the above comments mentioned i had an old package that i created using a PAT, and my repo didn't had permissions to write for same packages. that's the reason behind the permission denied error. Just delete the old package 💃