[CI] CodeCov no longer works after upgrade to v4 of the action

[dbwiddis]

What is the bug?

#484 upgraded our codecov GitHub Action dependency from v3 to v4. One of the listed Breaking Changes is:

Tokenless uploading is unsupported. However, PRs made from forks to the upstream public repos will support tokenless (e.g. contributors to OS projects do not need the upstream repo's Codecov token). This doc shows instructions on how to add the Codecov token.

This results in failures in our CI workflow:

==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov create-commit'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov create-commit -C 634e8d8
==> Uploader SHASUM verified (103bfefcc56f76473179e600b96eb8150b0f349ad94836b0f63f03ffac469ad7 codecov)
info - 2024-02-04 00:27:18,396 -- ci service found: github-actions
Error: Codecov token not found. Please provide Codecov token with -t flag.
Warning: Codecov: Failed to properly create commit: The process '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov' failed with exit code 1

How can one reproduce the bug?

1. Submit a PR.
2. Observe lack of a code coverage comment.

What is the expected behavior?

Code coverage, as established org-wide, see opensearch-project/opensearch-plugins#46 and related issues.

What is your host/environment?

GitHub Actions

Do you have any additional context?

The resolution instructions at the CodeCov repo indicate adding an appropriate Secret to the repo will resolve this. We would add a token: field under the with: here referencing that secret:

      - name: Upload Coverage Report
        if: ${{ matrix.codecov }}
        uses: codecov/codecov-action@v4
        with:
          file: ./build/reports/jacoco/test/jacocoTestReport.xml

However, individual repo maintainers do not have permissions to edit secrets, so resolving this will require EE team to add the secrets both here and org-wide.

I did experiment on my fork (which, as a fork, the docs say could have used the upstream public repo's token) with the standard-named token, but it doesn't seem to be present, so needs to be added.

        with:
          file: ./build/reports/jacoco/test/jacocoTestReport.xml
          token: ${{ secrets.CODECOV_TOKEN }}

CC @bbarani

[dbwiddis]

Failure on a PR from a fork, looks like upload succeeds but gets 500 error, no coverage report:

==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov create-commit'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov create-commit -C a36368cbead7375243de682205fa6bb2ffd9cadf
info - 2024-02-05 18:45:41,773 -- ci service found: github-actions
info - 2024-02-05 18:45:42,104 -- The PR is happening in a forked repo. Using tokenless upload.
info - 2024-02-05 18:45:43,068 -- Process Commit creating complete
error - 2024-02-05 18:45:43,068 -- Commit creating failed: {"error": "Server Error (500)"}
==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov create-report'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov create-report -C a36368cbead7375243de682205fa6bb2ffd9cadf
info - 2024-02-05 18:45:43,775 -- ci service found: github-actions
info - 2024-02-05 18:45:44,564 -- Process Report creating complete
error - 2024-02-05 18:45:44,564 -- Report creating failed: ["Commit SHA not found"]
==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov do-upload'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov do-upload -f ./build/reports/jacoco/test/jacocoTestReport.xml -C a36368cbead7375243de682205fa6bb2ffd9cadf
info - 2024-02-05 18:45:45,[28](https://github.com/opensearch-project/flow-framework/actions/runs/7789320303/job/21240683335?pr=494#step:5:29)0 -- ci service found: github-actions
warning - 2024-02-05 18:45:45,[29](https://github.com/opensearch-project/flow-framework/actions/runs/7789320303/job/21240683335?pr=494#step:5:30)1 -- xcrun is not installed or can't be found.
warning - 2024-02-05 18:45:45,[30](https://github.com/opensearch-project/flow-framework/actions/runs/7789320303/job/21240683335?pr=494#step:5:31)3 -- No gcov data found.
warning - 2024-02-05 18:45:45,303 -- coverage.py is not installed or can't be found.
info - 2024-02-05 18:45:45,[34](https://github.com/opensearch-project/flow-framework/actions/runs/7789320303/job/21240683335?pr=494#step:5:35)7 -- Found 1 coverage files to upload
info - 2024-02-05 18:45:45,347 -- > /home/runner/work/flow-framework/flow-framework/build/reports/jacoco/test/jacocoTestReport.xml
info - 2024-02-05 18:[45](https://github.com/opensearch-project/flow-framework/actions/runs/7789320303/job/21240683335?pr=494#step:5:46):[46](https://github.com/opensearch-project/flow-framework/actions/runs/7789320303/job/21240683335?pr=494#step:5:47),134 -- Process Upload complete
error - 2024-02-05 18:45:46,134 -- Upload failed: ["Commit SHA not found"]

[owaiskazi19]

@opensearch-project/admin can you help to add a new token here?

[prudhvigodithi]

Looks to me like its a broader issues and only impacts the backports, because they are not from forked repos. From what I understood Tokenless uploading is still supported. https://github.com/codecov/codecov-action?tab=readme-ov-file#breaking-changes

Tokenless uploading is unsupported. However, PRs made from forks to the upstream public repos will support tokenless (e.g. contributors to OS projects do not need the upstream repo's Codecov token)

We should try to address this to all repos and see how this can be fixed when updated to v4.

Thanks

[dbwiddis]

From what I understood Tokenless uploading is still supported

That's what I initially thought but the evidence in PRs shows otherwise.

It says "Tokenless uploading is unsupported." What it does allow is a fork of an upstream to use the upstream's token in a PR to the upstream.

This is a trivial exercise when one has access to add secrets to the repo.

1. I can go here to see my token: https://app.codecov.io/gh/opensearch-project/flow-framework/settings
2. I can click one button with my mouse and copy it to my clipboard
3. I could, if I had permissions, go to "Secrets" on my repo and paste the value and name it CODECOV_TOKEN.

This would take me about 30 seconds.

[gaiksaya]

Token has been added under SECRETS.CODECOV_TOKEN Thanks!

[dbwiddis]

Thanks @gaiksaya !