Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: tell codecov to wait_for_ci to avoid flappy reports #1160

Merged
merged 12 commits into from
Sep 22, 2023
+95 −53
Merged

ci: tell codecov to wait_for_ci to avoid flappy reports #1160

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
a37ba6a
Update codecov.yml
moul Sep 21, 2023
4416a4b
chore: fixup
moul Sep 21, 2023
f2f708c
chore: fixup
moul Sep 21, 2023
cfa1fae
Create tmp.go
moul Sep 21, 2023
db978bb
chore: fixup
moul Sep 21, 2023
f749046
chore: fixup
moul Sep 21, 2023
1e60225
chore: fixup
moul Sep 21, 2023
1ccf42d
chore: fixup
moul Sep 21, 2023
aa344d6
chore: fixup
moul Sep 21, 2023
cd08a31
Update tm2.yml
moul Sep 21, 2023
c3dbc92
Update codecov.yml
moul Sep 22, 2023
968c04c
Update codecov.yml
moul Sep 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 23 additions & 3 deletions .github/codecov.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
codecov:
require_ci_to_pass: false
notify:
wait_for_ci: false
wait_for_ci: true
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the trick; making codecov wait for all jobs to finish before generating a report.


ignore:
- misc
Expand All @@ -12,16 +12,36 @@ comment:
coverage:
round: down
precision: 2
status:
project:
default:
target: auto
threshold: 10 # Let's decrease this later.
base: parent
if_no_uploads: error
if_not_found: success
if_ci_failed: error
only_pulls: false
patch:
default:
target: auto
threshold: 100 # Allows PRs without tests, overall stats count.
base: auto
if_no_uploads: error
if_not_found: success
if_ci_failed: error
only_pulls: false

flag_management:
default_rules:
carryforward: true
statuses:
- type: project
target: auto
threshold: 0.5%
threshold: 10 # Let's decrease this later.
- type: patch
target: auto
target: auto # Allows PRs without tests, overall stats count.
threshold: 100
individual_flags:
- name: tm2
paths:
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/benchmark.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ on:
type: string
jobs:
benchmarks:
if: ${{ github.repository == 'gnolang/gno' }}
runs-on: [self-hosted, Linux, X64, benchmark-v1]
steps:
- name: checkout
Expand Down
7 changes: 4 additions & 3 deletions .github/workflows/db-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ jobs:
strategy:
fail-fast: false
matrix:
go-version:
goversion:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removing the dash to allow using the variable if if conditions

- "1.20.x"
- "1.21.x"
tags:
Expand All @@ -29,10 +29,12 @@ jobs:
- fsdb
- boltdb
steps:
- uses: actions/checkout@v4

# golang
- uses: actions/setup-go@v4
with:
go-version: ${{ matrix.go-version }}
go-version: ${{ matrix.goversion }}

# leveldb
- name: install leveldb
Expand All @@ -46,6 +48,5 @@ jobs:
sudo dpkg -i *.deb

# test ./pkgs/db
- uses: actions/checkout@v4
- name: test ./tm2/pkg/db
run: go test -tags ${{ matrix.tags }} ./tm2/pkg/db/...
18 changes: 9 additions & 9 deletions .github/workflows/examples.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,51 +21,51 @@ jobs:
strategy:
fail-fast: false
matrix:
go-version:
goversion:
- "1.20.x"
- "1.21.x"
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@v4
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moving the actions/checkout before setup-go, because setup-go is trying to cache dependencies based on go.mod and go.sum

- uses: actions/setup-go@v4
with:
go-version: ${{ matrix.go-version }}
- uses: actions/checkout@v4
go-version: ${{ matrix.goversion }}
- run: go install -v ./gnovm/cmd/gno
- run: go run ./gnovm/cmd/gno precompile --verbose ./examples
- run: go run ./gnovm/cmd/gno build --verbose ./examples
test:
strategy:
fail-fast: false
matrix:
go-version:
goversion:
- "1.20.x"
- "1.21.x"
# unittests: TODO: matrix with contracts
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v4
with:
go-version: ${{ matrix.go-version }}
- uses: actions/checkout@v4
go-version: ${{ matrix.goversion }}
- run: go install -v ./gnovm/cmd/gno
- run: go run ./gnovm/cmd/gno test --verbose ./examples
lint:
strategy:
fail-fast: false
matrix:
go-version:
goversion:
- "1.20.x"
- "1.21.x"
# unittests: TODO: matrix with contracts
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v4
with:
go-version: ${{ matrix.go-version }}
- uses: actions/checkout@v4
go-version: ${{ matrix.goversion }}
- run: go install -v ./gnovm/cmd/gno
# testing official directories, basically examples/ minus examples/.../x/.
- run: go run ./gnovm/cmd/gno lint --verbose ./examples/gno.land/p
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/gh-pages.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ concurrency:

jobs:
build:
if: ${{ github.repository == 'gnolang/gno' }} # Alternatively, validate based on provided tokens and permissions.
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
Expand All @@ -30,6 +31,7 @@ jobs:
path: ./misc/gendocs/godoc

deploy:
if: ${{ github.repository == 'gnolang/gno' }} # Alternatively, validate based on provided tokens and permissions.
runs-on: ubuntu-latest
environment:
name: github-pages
Expand Down
23 changes: 14 additions & 9 deletions .github/workflows/gnoland.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,12 @@ on:
- "tm2/**.go"
- "gno.land/**"
- ".github/workflows/gnovm.yml"
# Until the codecov issue is resolved, it's essential to run the tests for gnovm, tm2, and gno.land concurrently.
- "gnovm/**"
- "tm2/**"
- "gno.land/**"
- "examples/**"
- ".github/workflows/**"
push:
branches: [ "master" ]

Expand All @@ -21,7 +27,7 @@ jobs:
strategy:
fail-fast: false
matrix:
go-version:
goversion:
- "1.20.x"
- "1.21.x"
goarch: [ "amd64" ]
Expand All @@ -35,10 +41,10 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v4
with:
go-version: ${{ matrix.go-version }}
- uses: actions/checkout@v4
go-version: ${{ matrix.goversion }}
- name: go install
working-directory: gno.land
run: GOOS=${{ matrix.goos }} GOARCH=${{ matrix.goarch }} go install ./cmd/${{ matrix.program }}
Expand All @@ -47,7 +53,7 @@ jobs:
strategy:
fail-fast: false
matrix:
go-version:
goversion:
- "1.20.x"
- "1.21.x"
args:
Expand All @@ -58,25 +64,24 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v4
with:
go-version: ${{ matrix.go-version }}
- uses: actions/checkout@v4
go-version: ${{ matrix.goversion }}
- name: test
working-directory: gno.land
run: |
export GOPATH=$HOME/go
export GOTEST_FLAGS="-v -p 1 -timeout=30m -coverprofile=coverage.out -covermode=atomic"
make ${{ matrix.args }}
- if: runner.os == 'Linux'
- if: ${{ runner.os == 'Linux' && matrix.goversion == '1.21.x' }}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to prevent inconsistent results, let's only upload the coverage of the latest supported version, while still unit testing with the previous version.

uses: codecov/codecov-action@v3
with:
token: ${{ secrets.CODECOV_TOKEN }}
name: gno.land
flags: gno.land-${{matrix.args}}
files: ./gno.land/coverage.out
#fail_ci_if_error: ${{ github.repository == 'gnolang/gno' }}
fail_ci_if_error: false # temporarily
fail_ci_if_error: ${{ github.repository == 'gnolang/gno' }}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think this is what generated failed CI because of codecov 502 errors; would be nice to setup an auto-retry mechanism


docker-integration:
strategy:
Expand Down
24 changes: 15 additions & 9 deletions .github/workflows/gnovm.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,12 @@ on:
- "gnovm/Makefile"
- "tm2/**.go"
- ".github/workflows/gnovm.yml"
# Until the codecov issue is resolved, it's essential to run the tests for gnovm, tm2, and gno.land concurrently.
- "gnovm/**"
- "tm2/**"
- "gno.land/**"
- "examples/**"
- ".github/workflows/**"
push:
branches: [ "master" ]

Expand All @@ -23,7 +29,7 @@ jobs:
strategy:
fail-fast: false
matrix:
go-version: # two latest versions
goversion: # two latest versions
- "1.20.x"
- "1.21.x"
goenv: # TODO: replace with pairs, so it's easier to read in the GH interface.
Expand All @@ -34,10 +40,10 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v4
with:
go-version: ${{ matrix.go-version }}
- uses: actions/checkout@v4
go-version: ${{ matrix.goversion }}
- name: go install
working-directory: gnovm
run: ${{ matrix.goenv }} go install ./cmd/${{ matrix.program }}
Expand All @@ -46,7 +52,7 @@ jobs:
strategy:
fail-fast: false
matrix:
go-version:
goversion:
- "1.20.x"
- "1.21.x"
args:
Expand All @@ -62,22 +68,22 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v4
with:
go-version: ${{ matrix.go-version }}
- uses: actions/checkout@v4
go-version: ${{ matrix.goversion }}
- name: test
working-directory: gnovm
run: |
export GOPATH=$HOME/go
export GOTEST_FLAGS="-v -p 1 -timeout=30m -coverprofile=coverage.out -covermode=atomic"
make ${{ matrix.args }}
- if: runner.os == 'Linux'
- if: ${{ runner.os == 'Linux' && matrix.goversion == '1.21.x' }}
Copy link
Contributor

@ajnavarro ajnavarro Sep 21, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this might be a problem when we update go versions and 1.21 is not supported anymore, we can lose coverage execution. Maybe add go version to the flags?

Copy link
Member Author

@moul moul Sep 21, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I'm the one who made the 3 last go version bumps and I basically made sure to update all the references I can find.

at some point, I was considering adding a variable.yml file with "latest_go" and "previous_go" variable; or maybe we can add a quick misc/check-go-versions.sh script to assist us or to check in the CI.

uses: codecov/codecov-action@v3
with:
token: ${{ secrets.CODECOV_TOKEN }}
name: gnovm
verbose: true
flags: gnovm-${{matrix.args}}
files: ./gnovm/coverage.out
#fail_ci_if_error: ${{ github.repository == 'gnolang/gno' }}
fail_ci_if_error: false # temporarily
fail_ci_if_error: ${{ github.repository == 'gnolang/gno' }}
22 changes: 11 additions & 11 deletions .github/workflows/misc.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,14 @@ jobs:
lint:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Install Go
uses: actions/setup-go@v4
with:
go-version: 1.21.x

- name: Checkout code
uses: actions/checkout@v4

- name: Lint
uses: golangci/golangci-lint-action@v3
with:
Expand All @@ -31,17 +31,17 @@ jobs:
fmt:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Install Go
uses: actions/setup-go@v4
with:
go-version: 1.20.x
go-version: 1.21.x

- name: Install make
run: sudo apt-get install -y make

- name: Checkout code
uses: actions/checkout@v4

# prefill dependencies so that mod messages don't show up in make output
- name: Fetch dependencies
run: go mod download -modfile ./misc/devdeps/go.mod -x
Expand All @@ -61,13 +61,13 @@ jobs:
modtidy:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Install Go
uses: actions/setup-go@v4
with:
go-version: 1.20.x

- name: Checkout code
uses: actions/checkout@v4
go-version: 1.21.x

- name: Check go.mods
run: |
Expand Down
Loading
Loading