Share Go's test cache in CI between jobs to reduce latency for getting the results

[p0mvn]

Context

We would like to investigate and implement, if possible, the solution for sharing Go's test cache between test CI jobs to reduce the number of redundant runs

Initial Investigation

• The cache is stored under GOCACHE environment variable
  • Confirmed this by deleting the folder and rerunning the tests to see of they get cached.
• GitHub actions have artifacts and cache, we can probably store it there.
• Here's some info about caching and artifacts in GitHub Actions: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows

Risks

• The proposed solution has never been tested
• From quick google, there are some issues with getting it work in Gitlab: https://stackoverflow.com/questions/45537023/gitlab-ci-cache-keep-golang-packages-between-stages
• However, nothing specific to GitHub

Acceptance Criteria

• investigate the solution and alternatives
• implement reusable test cache
• the solution is stable
  • it reduces the number of test runs across jobs
  • it runs the tests when a change was made in a package

[ValarDragon]

This PR caches the entire go build directory, I don't know if the folder pointed to contains test cache on the CI system: #1583

On inspection of jobs in that PR, tests weren't cached on repeated runs of the same logic. If we see test runs speedup dramatically below 3 minutes, that'd be a good sign caching of tests is happening. The default go test cache location does appear to be getting cached in that PR though. Perhaps the -coverprofile forces re-running of all tests, similar to count=1? https://github.com/osmosis-labs/osmosis/blob/main/Makefile#L233-L234

Though tbh, if this stays at/under 3 minutes, I think its fine. Will get sped up further when we reduce the overhead of CLI tests.

[ValarDragon]

Ah indeed, golang/go#23565

coverprofile does not get cached with tests at the moment, hence tests are not getting cached. So the decision matrix for this issue is then:

• Keep coverprofile, close this issue, accept no test caching
• Remove coverprofile, go tests will be cached / execute extremely fast

I don't think I ever look at codecov comments on PRs / coverprofile personally, but I'm also fine with the current CI time / further speedups by eliminating more CI tests. How do others feel?

[alexanderbez]

I personally don't either, but it can be good to know how much coverage we have at any given moment. I'm in favor of caching, i.e. removing coverprofile

[ValarDragon]

Sweet, I'm down to remove coverprofile as well

[ValarDragon]

Ah the other benefit of removing this: No more noise notifications from codecov on your PRs 😂