ci: tell codecov to wait_for_ci to avoid flappy reports #1160
Conversation
Codecov ReportPatch coverage has no change and project coverage change:
Additional details and impacted files@@ Coverage Diff @@
## master #1160 +/- ##
==========================================
- Coverage 47.08% 47.03% -0.05%
==========================================
Files 365 365
Lines 61156 61156
==========================================
- Hits 28793 28765 -28
- Misses 30011 30039 +28
Partials 2352 2352
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
Signed-off-by: moul <[email protected]>
Signed-off-by: moul <[email protected]>
Signed-off-by: moul <[email protected]>
| @@ -1,7 +1,7 @@ | |||
| codecov: | |||
| require_ci_to_pass: false | |||
| notify: | |||
| wait_for_ci: false | |||
| wait_for_ci: true | |||
There was a problem hiding this comment.
this is the trick; making codecov wait for all jobs to finish before generating a report.
.github/codecov.yml
Outdated
| project: | ||
| default: | ||
| target: auto | ||
| threshold: 0.5% |
There was a problem hiding this comment.
i think this one should stay low like 0.5, or even smaller.
.github/codecov.yml
Outdated
| patch: | ||
| default: | ||
| target: auto | ||
| threshold: 1% |
There was a problem hiding this comment.
while i think this one could potentially be increased, because I think it works per file, and sometimes, adding a unique line may be seen as 5% change.
Signed-off-by: moul <[email protected]>
| @@ -20,7 +20,7 @@ jobs: | |||
| strategy: | |||
| fail-fast: false | |||
| matrix: | |||
| go-version: | |||
| goversion: | |||
There was a problem hiding this comment.
removing the dash to allow using the variable if if conditions
| - "1.20.x" | ||
| - "1.21.x" | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 30 | ||
| steps: | ||
| - uses: actions/checkout@v4 |
There was a problem hiding this comment.
moving the actions/checkout before setup-go, because setup-go is trying to cache dependencies based on go.mod and go.sum
| - name: test | ||
| working-directory: gno.land | ||
| run: | | ||
| export GOPATH=$HOME/go | ||
| export GOTEST_FLAGS="-v -p 1 -timeout=30m -coverprofile=coverage.out -covermode=atomic" | ||
| make ${{ matrix.args }} | ||
| - if: runner.os == 'Linux' | ||
| - if: ${{ runner.os == 'Linux' && matrix.goversion == '1.21.x' }} |
There was a problem hiding this comment.
to prevent inconsistent results, let's only upload the coverage of the latest supported version, while still unit testing with the previous version.
| uses: codecov/codecov-action@v3 | ||
| with: | ||
| token: ${{ secrets.CODECOV_TOKEN }} | ||
| name: gno.land | ||
| flags: gno.land-${{matrix.args}} | ||
| files: ./gno.land/coverage.out | ||
| #fail_ci_if_error: ${{ github.repository == 'gnolang/gno' }} | ||
| fail_ci_if_error: false # temporarily | ||
| fail_ci_if_error: ${{ github.repository == 'gnolang/gno' }} |
There was a problem hiding this comment.
i think this is what generated failed CI because of codecov 502 errors; would be nice to setup an auto-retry mechanism
Signed-off-by: moul <[email protected]>
| - name: test | ||
| working-directory: gnovm | ||
| run: | | ||
| export GOPATH=$HOME/go | ||
| export GOTEST_FLAGS="-v -p 1 -timeout=30m -coverprofile=coverage.out -covermode=atomic" | ||
| make ${{ matrix.args }} | ||
| - if: runner.os == 'Linux' | ||
| - if: ${{ runner.os == 'Linux' && matrix.goversion == '1.21.x' }} |
There was a problem hiding this comment.
this might be a problem when we update go versions and 1.21 is not supported anymore, we can lose coverage execution. Maybe add go version to the flags?
There was a problem hiding this comment.
yeah, I'm the one who made the 3 last go version bumps and I basically made sure to update all the references I can find.
- go1.21: chore: bump golang at go1.21 #1093
- go1.20: chore: switch to go1.20 #518
- go1.19: chore: bump go1.19 #310
at some point, I was considering adding a variable.yml file with "latest_go" and "previous_go" variable; or maybe we can add a quick misc/check-go-versions.sh script to assist us or to check in the CI.
Signed-off-by: moul <[email protected]>
|
The flakiness issue appears to be resolved, as seen in my fork:
Edit: I might have been a bit hasty in my assessment. I've just initiated new builds. I'll keep you updated. Edit 2: Upon review, I believe this PR effectively makes Codecov visible while minimizing false positives. I recommend we proceed with this to resolve the current impasse. Subsequently, we can dedicate more time to refining QA and DX. |
Signed-off-by: moul <[email protected]>
I've conducted tests on my fork, which you can find at https://github.com/moul/gno. The 'if' condition draws inspiration from @ajnavarro's contributions, so a shoutout to him for that. Additionally, @ajnavarro pointed out gaps in our testing for certain packages. We need to strategize on expanding our test coverage to address these overlooked areas. I suggest waiting for his alternative work before considering merging mine. --------- Signed-off-by: moul <[email protected]>
I've conducted tests on my fork, which you can find at https://github.com/moul/gno.
The 'if' condition draws inspiration from @ajnavarro's contributions, so a shoutout to him for that.
Additionally, @ajnavarro pointed out gaps in our testing for certain packages. We need to strategize on expanding our test coverage to address these overlooked areas.
I suggest waiting for his alternative work before considering merging mine.