upload-plan-destination: github is using deprecated Artifact API scheduled for EOL on November 30, 2024

[minamijoyo]

I noticed the following deprecation warning when using upload-plan-destination: github.

Deprecation notice: v1, v2, and v3 of the artifact actions
The following artifacts were uploaded using a version of actions/upload-artifact that is scheduled for deprecation: "environments_dev_dir1", "environments_prod_dir2".
Please update your workflow to use v4 of the artifact actions.
Learn more: https://github.blog/changelog/2024-04-16-deprecation-notice-v3-of-the-artifact-actions/

The minimum reproduction case is as follows:

name: Digger Workflow

on:
  pull_request:
    branches: [ main ]
    types: [ opened, synchronize ]
  issue_comment:
    types: [created]
  workflow_dispatch:

jobs:
  digger-job:
    runs-on: ubuntu-latest
    permissions:
      contents: write      # required to merge PRs
      actions: write       # required for plan persistence
      id-token: write      # required for workload-identity-federation
      pull-requests: write # required to post PR comments
      statuses: write      # required to validate combined PR status

    steps:
      - uses: diggerhq/digger@vLatest
        with:
          no-backend: true
          disable-locking: true
          upload-plan-destination: github
        env:
          GITHUB_CONTEXT: ${{ toJson(github) }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

The current vLatest commit ID is b552ebd, but looking at action.yml it doesn't look like we are depending on actions/upload-artifact.
https://github.com/diggerhq/digger/blob/b552ebd7556eed0a297af16f44042177c6b755ba/action.yml

Curious, I read the source and found that the current implementation is calling Artifact's API directly.

digger/libs/storage/plan_storage.go

Line 32 in b552ebd

artifactBase := fmt.Sprintf("%s_apis/pipelines/workflows/%s/artifacts?api-version=6.0-preview", actionsRuntimeURL, githubRunID)

api-version=6.0-preview looked suspicious, so I went to read the actions/upload-artifact implementation and found that actions/upload-artifact@v4 uses npm's @actions/artifact v2.
actions/upload-artifact@0f57369

At a quick glance @actions/artifact v2 endpoints appear to be completely different from @actions/artifact v1.
https://github.com/actions/toolkit/blob/%40actions/artifact%401.1.1/packages/artifact/src/internal/utils.ts#L222
https://github.com/actions/toolkit/blob/%40actions/artifact%402.0.1/packages/artifact/src/internal/shared/artifact-twirp-client.ts#L57

As I understand it, even if we are not using actions/upload-artifact directly, this old Artifact API will also be likely to unavailable after November 30, 2024, so it looks like something needs to be done before EOL.
https://github.blog/changelog/2024-04-16-deprecation-notice-v3-of-the-artifact-actions/

[motatoes]

Hey @minamijoyo thanks for this investigation! You are right about your analysis. Since we want to upload multiple artefacts with digger step directly we could not use the upload-artefact official step (actually we tried to use this in earlier version by saving to folder and group uploading but it didn't work very well). So we have reversed the steps and converted into golang requests

So yes, we need to perform the same reversing again unless GH have started allowing invoking the upload via some official api

[bsmlindner]

Hi,

As artifact-upload@v3 is now finally deprecated as mentioned here, we cannot use github as plan upload destination anymore.

The run always result in the following error:

url: https://pipelinesghubeus10.actions.githubusercontent.com/cuk4563n3GfNeAiLw9paOgdZCAD6w1HyMdvF6Drcw3TGJDw8CI/_apis/pipelines/workflows/13062225117/artifacts?api-version=6.0-previewRequest failed with status code: 400
body: {"$id":"1","innerException":null,"message":"Artifact Versions v1 through v3 are deprecated.","typeName":"Microsoft.Azure.Pipelines.Server.DeprecatedFeatureException, Microsoft.Azure.Pipelines.Server","typeKey":"DeprecatedFeatureException","errorCode":0,"eventId":3000}Error storing artifact file: could not create artifact with github error creating request: <nil>
Failed to Run digger plan command. error storing artifact file: could not create artifact with github error creating request: <nil>
error while running command digger plan for project default: Failed to Run digger plan command. error storing artifact file: could not create artifact with github error creating request: <nil>
Project default command digger plan failed, skipping job
Failed to run commands. %!s(<nil>)

We already tried to upgrade to the latest digger version v0.6.83 (at the time of writing the comment).

Are there already any plans when this issue will be fixed?

Thanks and best regards,
Manuel

[minamijoyo]

To clarify my position to anyone interested in this issue, I ended up using aws as a workaround, so it is no longer my priority. However, github is definitely easier to set up with no dependencies, and it would be helpful to have it as before!

[motatoes]

Hi everyone, we will take a look at this but might need to involve someone at github to help us out with it so it might take longer, no eta yet

In the meantime it seems that using s3 is a good workaround while a fix is out

[joerg]

Regarding S3, there is an open request to implement KMS: #1783
Since the plan explicitly contains sensitive data having a backend as secure as possible for storage would be a good idea. ;-)

[0x2b3bfa0]

I'm also going to switch to AWS S3, because I don't have time to write/contribute a fix.

[0x2b3bfa0]

Migration to AWS S3

https://docs.digger.dev/ce/howto/store-plans-in-a-bucket#aws

      - uses: diggerhq/[email protected]
        with:
          ...
          upload-plan-destination: aws
          upload-plan-destination-s3-bucket: ...

Tip

If you get 301 errors, check that your e.g. AWS_REGION environment variable is set to the same region where the bucket was created.

[motatoes]

Hi everyone, a quick update regarding this functionality we have originally implemented it by reverse engineering the github API since we needed to perform artefact updates per digger projects within the cli and doing it before/after was not feasible in many cases as in backendless mode we don't know what is the impacted projects upfront. In the meantime if you have the option to move to AWS S3 or GCP buckets to mitigate feel free to do so. Encryption for S3 bucket support was added in #1882

We have started updating our code to support v4 of artefacts api but it uses RPC and protobufs so it makes is a bit more challenging.