Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deployment failing due to "Failed to upload secrets" #240

Closed
vladinator1000 opened this issue Mar 5, 2024 · 9 comments
Closed

Deployment failing due to "Failed to upload secrets" #240

vladinator1000 opened this issue Mar 5, 2024 · 9 comments
Labels
tracked elsewhere A valid issue that is tracked elsewhere (such as workers-sdk)

Comments

@vladinator1000
Copy link

vladinator1000 commented Mar 5, 2024

I just migrated to wrangler-action v3 and this started happening to me. It fails every time I run it.

image
Here's my workflow file:

name: Deploy

on:
  workflow_dispatch:
  push:
    branches: [main]

jobs:
  build_and_deploy:
    name: Build and deploy
    runs-on: ubuntu-latest

    strategy:
      matrix:
        node-version: [21.x]

    env:
      ENVIRONMENT: production
      APP_SECRET: ${{ secrets.APP_SECRET }}
      DATABASE_URL: ${{ secrets.DATABASE_URL }}
      CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}

    steps:
      - uses: actions/checkout@v4
      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}

      - uses: c-hive/gha-yarn-cache@v2

      - name: Get version
        id: version
        run: echo "::set-output name=version::$(date +'%Y-%m-%dT%H:%M:%S')-${{ github.sha }}"

      - name: Install dependencies
        run: yarn --frozen-lockfile

      - name: 🔨📦 Build and deploy
        uses: cloudflare/wrangler-action@v3
        with:
          apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
          environment: 'production'
          secrets: |
            APP_SECRET
            DATABASE_URL

I ran it in debug mode, but didn't see any useful logging
image

I tried pinning wranglerVersion to the one in my package.json, but that didn't change anything.

@github-project-automation github-project-automation bot moved this to Untriaged in workers-sdk Mar 5, 2024
@vladinator1000 vladinator1000 changed the title Deployment failing due to secrets not uploading Deployment failing due to "Failed to upload secrets" Mar 5, 2024
@AdiRishi
Copy link
Contributor

AdiRishi commented Mar 6, 2024

Hey 👋
These issues are always quite tricky to debug so let me ask some simple debugging questions first

  • Is your worker already deployed? Or is it yet to be created? If not, you should deploy it without secrets first
  • Second, let's make sure the secret values are being passed in to your action, can you try to expand the group for cloudflare/wrangler-action@3, here's an example of what of of my projects looks like (note the inclusion of secret values in env)
    Run cloudflare/wrangler-action@v3
      with:
        accountId: ***
        apiToken: ***
        wranglerVersion: 3.26.0
        secrets: TURBO_TOKEN
        command: deploy --minify
        quiet: false
      env:
        PNPM_HOME: /home/runner/setup-pnpm/node_modules/.bin
        TURBO_TOKEN: ***
    
  • Have you tried adding the secrets manually via the CLI? Does it work?
  • In the output you pasted (running in debug mode), when you expand the Uploading secrets... group, is there any additional debugging information? Remember groups can have additional information inside them, you have to expand to find out more.
  • Instead of specifying CLOUDFLARE_ACCOUNT_ID in the environment, specify it directly in the options for wrangler-action

@enfipy
Copy link

enfipy commented Apr 7, 2024

Any news on this? I also can't make the wrangler-action upload secrets (I use wrangler v3.48.0).

Here's my code:

      - name: Deploy to Cloudflare
        uses: cloudflare/wrangler-action@v3
        with:
          accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
          apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
          workingDirectory: ./apps/api/
          packageManager: bun
          environment: production
          quit: false
          secrets: |
            RESEND_API_KEY
        env:
          RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}

And the result is always the same:

 Finished processing secrets JSON file:
  ✨ 0 secrets successfully uploaded
  
  ✘ [ERROR] 🚨 1 secrets failed to upload

@admah admah removed this from workers-sdk Apr 8, 2024
@caffeinated-nick
Copy link

I was able to resolve this issue by reverting to the "legacy" secrets upload method (i.e. setting wranglerVersion: '3.3.0' to get more detailed logging. Doing this means you will get logging per attempted secret upload, so if any individual secret has a problem you should get a clear message of why (rather than bulk upload failing with a generic error).

Example with bulk upload with wrangler >= 3.4.0 (I used 3.50.0)
Screenshot 2024-04-14 at 11 13 49 AM

Example with wrangler <=3.3.0
Screenshot 2024-04-14 at 11 15 32 AM

To "properly" fix it I guess CF would have to improve the error output on wrangler secret:bulk

@mpint
Copy link

mpint commented Apr 22, 2024

Thanks Nick, reverting to wranglerVersion: 3.3.0 helped with logging, although in my case, it's still not clear what's causing the error. My action logs look like this:

✨ Success! Uploaded secret SUPABASE_SERVICE_ROLE_KEY
✘ [ERROR] A request to the Cloudflare API (/accounts/123/workers/scripts/im--135348507-example_com-staging/secrets) failed.

  global variable USER_PAGERDUTY_API_KEY already set [code: 10053]

The logs seem to indicate USER_PAGERDUTY_API_KEY is set as a variable instead of a secret, but inspecting the Worker Variables tab in the dashboard confirms USER_PAGERDUTY_API_KEY is a secret as expected.

USER_PAGERDUTY_API_KEY Value encrypted

It's not clear to me why the SUPABASE_SERVICE_ROLE_KEY secret gets uploaded correctly but USER_PAGERDUTY_API_KEY fails when both of them are secrets that have been previously set using Wrangler (when the script was created).

@gentlementlegen
Copy link

Happening to me as well. In my case, deleting the worker and creating it again fixed the issue.

I was able to break it again by manually adding variables within Cloudflare Dashboard, anything random like test = foobar and try to deploy again afterwards through GitHub Action. Adding the environment variables would fail with the error

  Finished processing secrets JSON file:
  ✨ 0 secrets successfully uploaded
  
  ✘ [ERROR] 🚨 4 secrets failed to upload

Deleting again the manually added value allowed me to deploy again successfully, so my suspicion is any discrepancy with the variables in the dashboard and the Action file are the cause of the issue.

@willin
Copy link

willin commented Aug 21, 2024

config:

      - name: Deploy
        uses: cloudflare/wrangler-action@v3
        with:
          apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
          accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
          secrets: |
            RESEND_API_KEY
            EMAIL_USERNAME
            TEST_EMAIL_ADDRESS
        env:
          RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}
          EMAIL_USERNAME: ${{ secrets.EMAIL_USERNAME }}
          TEST_EMAIL_ADDRESS: ${{ secrets.TEST_EMAIL_ADDRESS }}

log: https://github.com/willin/resend-cloudflare-service-worker/actions/runs/10491270196/job/29059995916

  ✅ Wrangler installed
🔑 Uploading secrets...
  /home/runner/.bun/bin/bunx wrangler secret:bulk
   ⛅️ wrangler 3.13.2 (update available 3.72.1)
  ---------------------------------------------
  🌀 Creating the secrets for the Worker "email-sender-worker" 
  ✘ [ERROR] uploading secret for key: RESEND_API_KEY:
        A request to the Cloudflare API (/accounts/***/workers/scripts/email-sender-worker/secrets) failed.
  ✘ [ERROR] uploading secret for key: TEST_EMAIL_ADDRESS:
        A request to the Cloudflare API (/accounts/***/workers/scripts/email-sender-worker/secrets) failed.
  ✘ [ERROR] uploading secret for key: EMAIL_USERNAME:
        A request to the Cloudflare API (/accounts/***/workers/scripts/email-sender-worker/secrets) failed.
  Finished processing secrets JSON file:
  ✨ 0 secrets successfully uploaded
  ✘ [ERROR] 🚨 3 secrets failed to upload
  If you think this is a bug then please create an issue at https://github.com/cloudflare/workers-sdk/issues/new/choose
  Error: The process '/home/runner/.bun/bin/bunx' failed with exit code 1
Error: Failed to upload secrets.
Error: 🚨 Action failed

@kocendavid
Copy link

I just spent my portion on this issue. Found the solution. If you have already defined keys in your worker they need to be type "secret" (add -> value -> encrypt) or delete them. This way the secret bulk is able to create them OR update them. If key is not secret it cannot be updated thus the action will fail. Of course secret:bulk delete doesnt exist. Thats also why @gentlementlegen was successfull with deleting his worker.

@afdalwahyu
Copy link

thanks @kocendavid It worked on the first try, but when I redeployed, it got replaced again with plaintext.

My workaround for supporting multiple custom vars and secrets:

Make sure your secret vars are already using secret type in your cloudflare Variables and Secrets configuration.

name = "xxxxxxxx"
compatibility_date = "2024-11-06"
main = "./dist/worker/index.js"
assets = { directory = "./dist/public", binding = "ASSETS" }

[vars]
NUXT_OAUTH_AUTH0_CLIENT_ID = ""
NUXT_OAUTH_AUTH0_DOMAIN = ""
# make sure to exclude vars inside the TOML file when using secrets so that when using this github action, your vars are not replaced with plaintext
# NUXT_OAUTH_AUTH0_CLIENT_SECRET = ""
# NUXT_SESSION_PASSWORD = ""

github yaml

- name: Deploy
  uses: cloudflare/wrangler-action@v3
  env:
    NUXT_SESSION_PASSWORD: ${{ secrets.NUXT_SESSION_PASSWORD }}
    NUXT_OAUTH_AUTH0_CLIENT_ID: ${{ secrets.NUXT_OAUTH_AUTH0_CLIENT_ID }}
    NUXT_OAUTH_AUTH0_CLIENT_SECRET: ${{ secrets.NUXT_OAUTH_AUTH0_CLIENT_SECRET }}
    NUXT_OAUTH_AUTH0_DOMAIN: ${{ secrets.NUXT_OAUTH_AUTH0_DOMAIN }}
  with:
    apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
    accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
    environment: 'production'
    vars: |
      NUXT_OAUTH_AUTH0_CLIENT_ID
      NUXT_OAUTH_AUTH0_DOMAIN
    secrets: |
      NUXT_SESSION_PASSWORD
      NUXT_OAUTH_AUTH0_CLIENT_SECRET

Also, it seems that vars does not exist in the README.md, so I need to check the code myself to confirm that it actually exists for gihub action input.

jahands added a commit that referenced this issue Nov 18, 2024
`wrangler secret:bulk` is deprecated and will be removed in a future
version. This also improves logging in cases where a secret is failing
to upload because an environment variable with the same name already
exists (see: #240).
@jahands
Copy link
Contributor

jahands commented Nov 18, 2024

Hi! As others have pointed out, this can happen if there is already a binding/environment variable set on the Worker with the same name as the secret. This includes non-secret text variables set on the Worker.

To fix this, remove the conflicting binding/environment variable from your Worker and try again.

The logging that wrangler outputs is admittedly not helpful here, so I've raised an issue to improve that in wrangler itself: cloudflare/workers-sdk#7287

Closing this out in favor of the workers-sdk issue.

@jahands jahands closed this as completed Nov 18, 2024
@jahands jahands added the tracked elsewhere A valid issue that is tracked elsewhere (such as workers-sdk) label Nov 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tracked elsewhere A valid issue that is tracked elsewhere (such as workers-sdk)
Projects
None yet
Development

No branches or pull requests

10 participants