[Merged by Bors] - Ultra Fast Super Slick CI #4755

AgeManning · 2023-09-19T13:17:11Z

Attempting to improve our CI speeds as its recently been a pain point.

Major changes:

Use a github action to pull stable/nightly rust rather than building it each run
Shift test suite to nexttest https://github.com/nextest-rs/nextest for CI

UPDATE:

So I've iterated on some changes, and although I think its still not optimal I think this is a good base to start from. Some extra things in this PR:

Shifted where we pull rust from. We're now using this thing: https://github.com/moonrepo/setup-rust . It's got some interesting cache's built in, but was not seeing the gains that Jimmy managed to get. In either case tho, it can pull rust, cargofmt, clippy, cargo nexttest all in < 5s. So I think it's worthwhile.
I've grouped a few of the check-like tests into a single test called code-test. Although we were using github runners in parallel which may be faster, it just seems wasteful. There were like 4-5 tests, where we would pull lighthouse, compile it, then run an action, like clippy, cargo-audit or fmt. I've grouped these into a single action, so we only compile lighthouse once, then in each step we run the checks. This avoids compiling lighthouse like 5 times.
Ive made doppelganger tests run on our local machines to avoid pulling foundry, building and making lcli which are all now baked into the images.
We have sccache and do not incremental compile lighthouse

Misc bonus things:

Cargo update
Fix web3 signer openssl keys which is required after a cargo update
Use mock_instant in an LRU cache test to avoid non-deterministic test
Remove race condition in building web3signer tests

There's still some things we could improve on. Such as downloading the EF tests every run and the web3-signer binary, but I've left these to be out of scope of this PR. I think the above are meaningful improvements.

.github/workflows/test-suite.yml

according to https://github.com/actions/runner-images/blob/ubuntu22/20230911.1/images/linux/Ubuntu2204-Readme.md (closes version matches: go 1.20.8 .net 6.0.413)

unblock #4755 Co-authored-by: Paul Hauner <[email protected]> Co-authored-by: realbigsean <[email protected]>

paulhauner · 2023-10-03T04:56:49Z

bors r+

bors · 2023-10-03T05:11:52Z

Canceled.

AgeManning · 2023-10-03T05:11:58Z

bors retry

bors · 2023-10-03T05:12:02Z

🕐 Waiting for PR status (GitHub check) to be set, probably by CI. Bors will automatically try to run when all required PR statuses are set.

Attempting to improve our CI speeds as its recently been a pain point. Major changes: - Use a github action to pull stable/nightly rust rather than building it each run - Shift test suite to `nexttest` https://github.com/nextest-rs/nextest for CI UPDATE: So I've iterated on some changes, and although I think its still not optimal I think this is a good base to start from. Some extra things in this PR: - Shifted where we pull rust from. We're now using this thing: https://github.com/moonrepo/setup-rust . It's got some interesting cache's built in, but was not seeing the gains that Jimmy managed to get. In either case tho, it can pull rust, cargofmt, clippy, cargo nexttest all in < 5s. So I think it's worthwhile. - I've grouped a few of the check-like tests into a single test called `code-test`. Although we were using github runners in parallel which may be faster, it just seems wasteful. There were like 4-5 tests, where we would pull lighthouse, compile it, then run an action, like clippy, cargo-audit or fmt. I've grouped these into a single action, so we only compile lighthouse once, then in each step we run the checks. This avoids compiling lighthouse like 5 times. - Ive made doppelganger tests run on our local machines to avoid pulling foundry, building and making lcli which are all now baked into the images. - We have sccache and do not incremental compile lighthouse Misc bonus things: - Cargo update - Fix web3 signer openssl keys which is required after a cargo update - Use mock_instant in an LRU cache test to avoid non-deterministic test - Remove race condition in building web3signer tests There's still some things we could improve on. Such as downloading the EF tests every run and the web3-signer binary, but I've left these to be out of scope of this PR. I think the above are meaningful improvements. Co-authored-by: Paul Hauner <[email protected]> Co-authored-by: realbigsean <[email protected]> Co-authored-by: antondlr <[email protected]>

bors · 2023-10-03T06:10:45Z

Build failed:

release-tests-windows

paulhauner · 2023-10-03T06:12:01Z

bors retry

Attempting to improve our CI speeds as its recently been a pain point. Major changes: - Use a github action to pull stable/nightly rust rather than building it each run - Shift test suite to `nexttest` https://github.com/nextest-rs/nextest for CI UPDATE: So I've iterated on some changes, and although I think its still not optimal I think this is a good base to start from. Some extra things in this PR: - Shifted where we pull rust from. We're now using this thing: https://github.com/moonrepo/setup-rust . It's got some interesting cache's built in, but was not seeing the gains that Jimmy managed to get. In either case tho, it can pull rust, cargofmt, clippy, cargo nexttest all in < 5s. So I think it's worthwhile. - I've grouped a few of the check-like tests into a single test called `code-test`. Although we were using github runners in parallel which may be faster, it just seems wasteful. There were like 4-5 tests, where we would pull lighthouse, compile it, then run an action, like clippy, cargo-audit or fmt. I've grouped these into a single action, so we only compile lighthouse once, then in each step we run the checks. This avoids compiling lighthouse like 5 times. - Ive made doppelganger tests run on our local machines to avoid pulling foundry, building and making lcli which are all now baked into the images. - We have sccache and do not incremental compile lighthouse Misc bonus things: - Cargo update - Fix web3 signer openssl keys which is required after a cargo update - Use mock_instant in an LRU cache test to avoid non-deterministic test - Remove race condition in building web3signer tests There's still some things we could improve on. Such as downloading the EF tests every run and the web3-signer binary, but I've left these to be out of scope of this PR. I think the above are meaningful improvements. Co-authored-by: Paul Hauner <[email protected]> Co-authored-by: realbigsean <[email protected]> Co-authored-by: antondlr <[email protected]>

bors · 2023-10-03T06:32:03Z

Build failed:

release-tests-windows

paulhauner · 2023-10-03T06:32:57Z

bors retry

Attempting to improve our CI speeds as its recently been a pain point. Major changes: - Use a github action to pull stable/nightly rust rather than building it each run - Shift test suite to `nexttest` https://github.com/nextest-rs/nextest for CI UPDATE: So I've iterated on some changes, and although I think its still not optimal I think this is a good base to start from. Some extra things in this PR: - Shifted where we pull rust from. We're now using this thing: https://github.com/moonrepo/setup-rust . It's got some interesting cache's built in, but was not seeing the gains that Jimmy managed to get. In either case tho, it can pull rust, cargofmt, clippy, cargo nexttest all in < 5s. So I think it's worthwhile. - I've grouped a few of the check-like tests into a single test called `code-test`. Although we were using github runners in parallel which may be faster, it just seems wasteful. There were like 4-5 tests, where we would pull lighthouse, compile it, then run an action, like clippy, cargo-audit or fmt. I've grouped these into a single action, so we only compile lighthouse once, then in each step we run the checks. This avoids compiling lighthouse like 5 times. - Ive made doppelganger tests run on our local machines to avoid pulling foundry, building and making lcli which are all now baked into the images. - We have sccache and do not incremental compile lighthouse Misc bonus things: - Cargo update - Fix web3 signer openssl keys which is required after a cargo update - Use mock_instant in an LRU cache test to avoid non-deterministic test - Remove race condition in building web3signer tests There's still some things we could improve on. Such as downloading the EF tests every run and the web3-signer binary, but I've left these to be out of scope of this PR. I think the above are meaningful improvements. Co-authored-by: Paul Hauner <[email protected]> Co-authored-by: realbigsean <[email protected]> Co-authored-by: antondlr <[email protected]>

bors · 2023-10-03T07:08:17Z

Pull request successfully merged into unstable.

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

unblock sigp#4755 Co-authored-by: Paul Hauner <[email protected]> Co-authored-by: realbigsean <[email protected]>

Attempting to improve our CI speeds as its recently been a pain point. Major changes: - Use a github action to pull stable/nightly rust rather than building it each run - Shift test suite to `nexttest` https://github.com/nextest-rs/nextest for CI UPDATE: So I've iterated on some changes, and although I think its still not optimal I think this is a good base to start from. Some extra things in this PR: - Shifted where we pull rust from. We're now using this thing: https://github.com/moonrepo/setup-rust . It's got some interesting cache's built in, but was not seeing the gains that Jimmy managed to get. In either case tho, it can pull rust, cargofmt, clippy, cargo nexttest all in < 5s. So I think it's worthwhile. - I've grouped a few of the check-like tests into a single test called `code-test`. Although we were using github runners in parallel which may be faster, it just seems wasteful. There were like 4-5 tests, where we would pull lighthouse, compile it, then run an action, like clippy, cargo-audit or fmt. I've grouped these into a single action, so we only compile lighthouse once, then in each step we run the checks. This avoids compiling lighthouse like 5 times. - Ive made doppelganger tests run on our local machines to avoid pulling foundry, building and making lcli which are all now baked into the images. - We have sccache and do not incremental compile lighthouse Misc bonus things: - Cargo update - Fix web3 signer openssl keys which is required after a cargo update - Use mock_instant in an LRU cache test to avoid non-deterministic test - Remove race condition in building web3signer tests There's still some things we could improve on. Such as downloading the EF tests every run and the web3-signer binary, but I've left these to be out of scope of this PR. I think the above are meaningful improvements. Co-authored-by: Paul Hauner <[email protected]> Co-authored-by: realbigsean <[email protected]> Co-authored-by: antondlr <[email protected]>

unblock sigp#4755 Co-authored-by: Paul Hauner <[email protected]> Co-authored-by: realbigsean <[email protected]>

Attempting to improve our CI speeds as its recently been a pain point. Major changes: - Use a github action to pull stable/nightly rust rather than building it each run - Shift test suite to `nexttest` https://github.com/nextest-rs/nextest for CI UPDATE: So I've iterated on some changes, and although I think its still not optimal I think this is a good base to start from. Some extra things in this PR: - Shifted where we pull rust from. We're now using this thing: https://github.com/moonrepo/setup-rust . It's got some interesting cache's built in, but was not seeing the gains that Jimmy managed to get. In either case tho, it can pull rust, cargofmt, clippy, cargo nexttest all in < 5s. So I think it's worthwhile. - I've grouped a few of the check-like tests into a single test called `code-test`. Although we were using github runners in parallel which may be faster, it just seems wasteful. There were like 4-5 tests, where we would pull lighthouse, compile it, then run an action, like clippy, cargo-audit or fmt. I've grouped these into a single action, so we only compile lighthouse once, then in each step we run the checks. This avoids compiling lighthouse like 5 times. - Ive made doppelganger tests run on our local machines to avoid pulling foundry, building and making lcli which are all now baked into the images. - We have sccache and do not incremental compile lighthouse Misc bonus things: - Cargo update - Fix web3 signer openssl keys which is required after a cargo update - Use mock_instant in an LRU cache test to avoid non-deterministic test - Remove race condition in building web3signer tests There's still some things we could improve on. Such as downloading the EF tests every run and the web3-signer binary, but I've left these to be out of scope of this PR. I think the above are meaningful improvements. Co-authored-by: Paul Hauner <[email protected]> Co-authored-by: realbigsean <[email protected]> Co-authored-by: antondlr <[email protected]>

unblock sigp#4755 Co-authored-by: Paul Hauner <[email protected]> Co-authored-by: realbigsean <[email protected]>

Attempting to improve our CI speeds as its recently been a pain point. Major changes: - Use a github action to pull stable/nightly rust rather than building it each run - Shift test suite to `nexttest` https://github.com/nextest-rs/nextest for CI UPDATE: So I've iterated on some changes, and although I think its still not optimal I think this is a good base to start from. Some extra things in this PR: - Shifted where we pull rust from. We're now using this thing: https://github.com/moonrepo/setup-rust . It's got some interesting cache's built in, but was not seeing the gains that Jimmy managed to get. In either case tho, it can pull rust, cargofmt, clippy, cargo nexttest all in < 5s. So I think it's worthwhile. - I've grouped a few of the check-like tests into a single test called `code-test`. Although we were using github runners in parallel which may be faster, it just seems wasteful. There were like 4-5 tests, where we would pull lighthouse, compile it, then run an action, like clippy, cargo-audit or fmt. I've grouped these into a single action, so we only compile lighthouse once, then in each step we run the checks. This avoids compiling lighthouse like 5 times. - Ive made doppelganger tests run on our local machines to avoid pulling foundry, building and making lcli which are all now baked into the images. - We have sccache and do not incremental compile lighthouse Misc bonus things: - Cargo update - Fix web3 signer openssl keys which is required after a cargo update - Use mock_instant in an LRU cache test to avoid non-deterministic test - Remove race condition in building web3signer tests There's still some things we could improve on. Such as downloading the EF tests every run and the web3-signer binary, but I've left these to be out of scope of this PR. I think the above are meaningful improvements. Co-authored-by: Paul Hauner <[email protected]> Co-authored-by: realbigsean <[email protected]> Co-authored-by: antondlr <[email protected]>

AgeManning and others added 17 commits September 18, 2023 13:12

First attempt will never work. Here's to hoping

613dd7b

Correct simple mistakes

789ec23

Fix makefile

805d5f4

More fixes

f985f3a

Add configuration to nextest

be10057

detect self-hostedness through runner name

a1c1a4b

try different syntax for if statement

fdaf199

test latest windows runner image

9ccda0d

Don't run block processing tests in debug

28aab54

Looser name matching

b0059e1

What is this env?

bfe7be1

Fix builder name matching

0fea018

Cleanup if statements

fad2e43

give us some sccache stats

34f2ac3

Trial a new rust builder

39b0655

Fix slasher tests

84cbf33

now actually show cache stats

76871bb

jimmygchen reviewed Sep 19, 2023

View reviewed changes

.github/workflows/test-suite.yml Outdated Show resolved Hide resolved

antondlr added 2 commits September 19, 2023 20:49

proceed if folder exists

ea891ee

rename mkdir to avoid clash

82134a9

jimmygchen mentioned this pull request Sep 19, 2023

CI Job Caching #4750

Closed

AgeManning and others added 8 commits September 20, 2023 10:32

Update caching

1170b6c

Improve workflow and correct web3 tests

204138b

Use mock-instant for lru-time based tests

0cabbf8

Fix racey tests and doppleganger ci

13034ab

show the stats forreal this time

72be592

show cache stats on windows

f16ae0d

this should already be present in the runner

b050579

according to https://github.com/actions/runner-images/blob/ubuntu22/20230911.1/images/linux/Ubuntu2204-Readme.md (closes version matches: go 1.20.8 .net 6.0.413)

See how grouping runs goes

ebd97bf

jimmygchen added the infra-ci label Sep 21, 2023

bors bot pushed a commit that referenced this pull request Sep 29, 2023

Clean bors.toml (#4795)

7b28ae6

unblock #4755 Co-authored-by: Paul Hauner <[email protected]> Co-authored-by: realbigsean <[email protected]>

bors bot pushed a commit that referenced this pull request Sep 29, 2023

Clean bors.toml (#4795)

1572308

unblock #4755 Co-authored-by: Paul Hauner <[email protected]> Co-authored-by: realbigsean <[email protected]>

bors bot pushed a commit that referenced this pull request Sep 29, 2023

Clean bors.toml (#4795)

38a9caf

unblock #4755 Co-authored-by: Paul Hauner <[email protected]> Co-authored-by: realbigsean <[email protected]>

bors bot pushed a commit that referenced this pull request Sep 29, 2023

Clean bors.toml (#4795)

111338f

unblock #4755 Co-authored-by: Paul Hauner <[email protected]> Co-authored-by: realbigsean <[email protected]>

bors bot pushed a commit that referenced this pull request Oct 2, 2023

Clean bors.toml (#4795)

5fbb73a

unblock #4755 Co-authored-by: Paul Hauner <[email protected]> Co-authored-by: realbigsean <[email protected]>

bors bot pushed a commit that referenced this pull request Oct 2, 2023

Clean bors.toml (#4795)

2c3eceb

unblock #4755 Co-authored-by: Paul Hauner <[email protected]> Co-authored-by: realbigsean <[email protected]>

bors bot pushed a commit that referenced this pull request Oct 2, 2023

Clean bors.toml (#4795)

b4c2081

unblock #4755 Co-authored-by: Paul Hauner <[email protected]> Co-authored-by: realbigsean <[email protected]>

Merge sigp#4795

9627bce

bors bot pushed a commit that referenced this pull request Oct 3, 2023

Clean bors.toml (#4795)

a935dae

unblock #4755 Co-authored-by: Paul Hauner <[email protected]> Co-authored-by: realbigsean <[email protected]>

Merge unstable

e4818e7

bors bot changed the title ~~Ultra Fast Super Slick CI~~ [Merged by Bors] - Ultra Fast Super Slick CI Oct 3, 2023

bors bot closed this Oct 3, 2023

Woodpile37 pushed a commit to Woodpile37/lighthouse that referenced this pull request Jan 6, 2024

Clean bors.toml (sigp#4795)

864ab24

unblock sigp#4755 Co-authored-by: Paul Hauner <[email protected]> Co-authored-by: realbigsean <[email protected]>

Woodpile37 pushed a commit to Woodpile37/lighthouse that referenced this pull request Jan 6, 2024

Clean bors.toml (sigp#4795)

a150d71

unblock sigp#4755 Co-authored-by: Paul Hauner <[email protected]> Co-authored-by: realbigsean <[email protected]>

Woodpile37 pushed a commit to Woodpile37/lighthouse that referenced this pull request Jan 6, 2024

Clean bors.toml (sigp#4795)

39509da

unblock sigp#4755 Co-authored-by: Paul Hauner <[email protected]> Co-authored-by: realbigsean <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Merged by Bors] - Ultra Fast Super Slick CI #4755

[Merged by Bors] - Ultra Fast Super Slick CI #4755

AgeManning commented Sep 19, 2023 •

edited

Loading

paulhauner commented Oct 3, 2023

bors bot commented Oct 3, 2023

AgeManning commented Oct 3, 2023

bors bot commented Oct 3, 2023

bors bot commented Oct 3, 2023

paulhauner commented Oct 3, 2023

bors bot commented Oct 3, 2023

paulhauner commented Oct 3, 2023

bors bot commented Oct 3, 2023

[Merged by Bors] - Ultra Fast Super Slick CI #4755

[Merged by Bors] - Ultra Fast Super Slick CI #4755

Conversation

AgeManning commented Sep 19, 2023 • edited Loading

paulhauner commented Oct 3, 2023

bors bot commented Oct 3, 2023

AgeManning commented Oct 3, 2023

bors bot commented Oct 3, 2023

bors bot commented Oct 3, 2023

paulhauner commented Oct 3, 2023

bors bot commented Oct 3, 2023

paulhauner commented Oct 3, 2023

bors bot commented Oct 3, 2023

AgeManning commented Sep 19, 2023 •

edited

Loading