chore(ci): various optimizations for build processing, caching and concurrency

[jakobmoellerdev]

Our caches are getting too big:

Approaching total cache storage limit (58.09 GB of 10 GB Used)

On top we dont make use of makes inbuilt parallelism (-j) and also we do not make use of shells forking mechanism for concurrent execution (& execution chain followed by wait)

What this PR does / why we need it

Limits the cache save to only the main branch in the build step, all other branches/PRs and Jobs will reuse that cache instead of computing their own

What does this mean:

PRO:

• only one cache that should roundabout have 4 (build files for all platforms) + 2 (build+test files for unit tests on amd64) Gi of modules and builds that get updated on every commit to main
• The build cache is likely unaffected or largely still valid for most if not every PR. This leads to compile time improvements upwards of 600% depending on PR size without killing our cache usage.

CON:

• a rebuild or dependency changes will be able to "invalidate" that cache and could lead to longer build times faster. However in general, most builds should still perform the same. If the go.mod / go.sum is not touched, the build cache will be restored for all files except the ones touched or impacted by the PR.

Which issue(s) this PR fixes

fix #999 #1000

[jakobmoellerdev]

@hilmarf could you take another look here

[hilmarf]

General question: what do we want to build as fast as possible? (What requires really the best cache setup?)

• Release builds?
• PR builds?
  • Dependency updates?
  • Feature builds?
  • Bug fixes?
• Hotfixes?
• ... ?

[jakobmoellerdev]

General question: what do we want to build as fast as possible? (What requires really the best cache setup?)

• Release builds?

• PR builds?

  • Dependency updates?
  • Feature builds?
  • Bug fixes?

• Hotfixes?

• ... ?

All build steps using go dependencies should be accelerated as much as possible. I would like to keep build times in the low 5 minutes, so also all PR builds.

However due to our exorbitantly large cache, the tradeoff is to reuse only one build cache from release builds as well as test steps on main.