Add CI cache

[ValarDragon]

What is the purpose of the change

Try adding a cache for go build data, to help speedup CI. Copied from mvdan here: https://github.com/mvdan/github-actions-golang/blob/master/.github/workflows/test.yml

cref #1578

Brief Changelog

• Adds go cache to CI

Testing and Verifying

We will need to keep an eye on if there are edge cases that make this not work / give incorrect results, or things don't speedup. A bit hard to know this without it being in place though.

Documentation and Release Note

• Does this pull request introduce a new feature or user-facing behavior changes? yes
• Is a relevant changelog entry added to the Unreleased section in CHANGELOG.md? no (I don't think CI is in scope)
• How is the feature or change documented? We need to figure out a strategy for if/how we want to document CI.

[codecov-commenter]

Codecov Report

Merging #1583 (b1668ef) into main (31a28e3) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1583   +/-   ##
=======================================
  Coverage   19.53%   19.53%           
=======================================
  Files         242      242           
  Lines       32262    32262           
=======================================
  Hits         6303     6303           
  Misses      24804    24804           
  Partials     1155     1155           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 31a28e3...b1668ef. Read the comment docs.

[ValarDragon]

Compare the two runs

1. (First run, cache miss) https://github.com/osmosis-labs/osmosis/runs/6591778494?check_suite_focus=true
2. (Second run, cache hit) https://github.com/osmosis-labs/osmosis/runs/6591903758?check_suite_focus=true

This cut out over half the time! 7min -> 3min 13sec (No tests were cached here, presumably due to me doing a change in app.go which under current infra requires re-running every test! So this is just eliminating build times)

This suggests that the go build time was around 3 minutes in every CI run, perhaps getting something similar for docker tests would help as well! We'd likely want a different cache key for the docker tests though, since we don't want it to override the go test cache.

[ValarDragon]

Last commit is trying to add a cache to test-e2e, may need a couple of go edits before we can see if it helps

[ValarDragon]

Awesome, tests are still getting rebuilt / re-run! (Evident by last commit)

[ValarDragon]

For Docker, Compare the two runs

(First run, cache miss) https://github.com/osmosis-labs/osmosis/runs/6592152484?check_suite_focus=true
(Second run, cache hit) https://github.com/osmosis-labs/osmosis/runs/6592518772?check_suite_focus=true

Time went from 9m31s -> 8m23s. However, concurrently it appeared github actions is having network latency problems. (The lint job and codeQL job was intermittently failing to connect to the internet for docker pull, and checkout the repo)

The e2e upgrade testing time shaved 1minute 40 seconds, however docker image build increased by 30 seconds. So we still need to separately figure out how to speedup docker image building (perhaps incremental build) and further speeding up the e2e tests.

[ValarDragon]

Linter speed difference:

(Run with no cache) https://github.com/osmosis-labs/osmosis/runs/6591900239?check_suite_focus=true
(First run, cache miss) https://github.com/osmosis-labs/osmosis/runs/6593255081?check_suite_focus=true
(Second run, cache hit) https://github.com/osmosis-labs/osmosis/runs/6593340103?check_suite_focus=true

Time difference is 2m14s -> 36s. (The no cache run took 2m6s, so huge time saving regardless.)