Speedup CI

[ValarDragon]

Background

We should invest time in speeding up CI. Lots of time is lost in just waiting for CI to finish executing. If theres any reason we cannot do all of the above in github actions, imo we should investigate moving to a paid CI service, perhaps Circle, where they can all be done. Getting the feedback loop right for rapid merging multiple related PR's is important.

First where we stand on latencies. As of time of issue creation, we are blocked on:

• CodeQL (Takes 10-15 minutes to complete) https://github.com/osmosis-labs/osmosis/runs/6593339656?check_suite_focus=true
• Two simultaneous end to end tests (takes around 10 minutes to complete, 3.5 minutes of which is in docker building, and by unreliable visual inspection 1.5 minute seems to be in downloading go packages)
• (Queued for awhile) the two go tests, each of which takes around 7 minutes to complete, 1.5 min of which I unreliably eyeball estimate to be spent in downloading go packages. A large chunk is also in CLI tests.

Furthermore, many github actions are stuck in 'queued', and take awhile before they begin execution, further increasing this dead time cost.

Components

• Remove duplicate push / pull request jobs. Make CI only run on pushes to main / release branches. Comment out less useful CLI tests #1573 , hopefully improves queue delay
• Remove less useful CLI tests (~2 minute saving on inspection of CLI logs)
• Cache / incrementally Docker build #1601
• Cache go tests Share Go's test cache in CI between jobs to reduce latency for getting the results  #1229
• Cache go package downloads in CI (Add CI cache #1583)
• Figure out how to improve CodeQL times, or just remove it.

Acceptance Criteria

• CI completes in under 5 minutes

[ValarDragon]

This repo has examples for how we can cache go package downloads in CI: https://github.com/mvdan/github-actions-golang

It also has the caching needed to get incremental builds, which would likely also help some of this CI time.

Seems like the caching for incremental go builds may also help with the docker image build time?

[p0mvn]

We can also run e2e upgrade tests on merge to main only:

osmosis/tests/e2e/e2e_setup_test.go

Line 155 in c8ac95c

if str := os.Getenv("OSMOSIS_E2E_SKIP_UPGRADE"); len(str) > 0 {

Upgrade testing takes a long time and it doesn't add much value running it on every PR / commit to PR. As long as we know that upgrade tests pass on main, that should be sufficient.

We should still continue running e2e tests on every PR, just the upgrade part can be skipped

[ValarDragon]

oh thats a great idea. Perhaps we can make that variable also get run if theres any change in the upgrades sub folder as well?

[ValarDragon]

For CodeQL, here are there docs on speeding it up: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/troubleshooting-the-codeql-workflow#the-build-takes-too-long

TL;DR:

• if self hosting a runner, parallelize
• Exclude many files from the building, e.g. all _test files. (And in our case, also likely CLI files)
• Just do it on push to main or on a schedule, not on every PR commit

But to be honest, I don't really see much point in keeping it. I'd rather just use gosec, which is AST based code scanning

[ValarDragon]

For docker build times, I think we can replace the make docker-build-debug step in test.yml with a direct docker build invocation, and use cache fields, e.g. https://github.com/docker/build-push-action/blob/master/docs/advanced/cache.md#cache-backend-api

cc @nikever

[niccoloraspa]

Added the cache option in #1602
It gives a nice speed up but I would like to be even faster.

There are some redundant work between go_test step and e2e_test that could be improved.
I'm working on re-structuring a little bit the setup to avoid redoing work.

[ValarDragon]

We've completed everything thats not test-e2e related!

Roman did a great job at summarizing a path to tackle for doing that in #1646 , nothing comes to my mind immediately for speedup steps beyond that. So going to close this in favor of 1646 for now!