Use caching to speed up functional tests

[oskirby]

Description

The functional tests are slow, and it sucks. Here are some tweaks to make it go faster:

• Install ccache which we'll use to cache compiled objects.
• Setup actions/cache@v4 to save the results of the ccache folder on every push
• Setup actions/cache@v4 to pull the ccache folder from the PR's base commit.
• Refactor the test addon build into a reusable workflow and cache the addons since they rarely change.

Some future work to make it go even faster:

• The rust libraries don't benefit from ccache and they are slow to build. However, they don't change very often so maybe we should just cache the generated libs?
• Miniconda is sloooooow. Unfortunately, caching it is complicated because of how it interracts with pip packages.
• Miniconda is kinda too big to cache. Github workers tend to have 1-2GB of cache space available and the Miniconda package cache is something like 1.1GB in size.

Reference

i.e Jira or Github issue URL

Checklist

• My code follows the style guidelines for this project
• I have not added any packages that contain high risk or unknown licenses (GPL, LGPL, MPL, etc. consult with DevOps if in question)
• I have performed a self review of my own code
• I have commented my code PARTICULARLY in hard to understand areas
• I have added thorough tests where needed

[mcleinman]

Thanks!

[strseb]

Miniconda is kinda too big to cache. Github workers tend to have 1-2GB of cache space available and the Miniconda package cache is something like 1.1GB in size.

Looking at the env, maybe we should continue splitting up the files to avoid downloading things we dont need, there are a lot of dependencies we pull that are depending on the target you need :/

  - clang=16.0.6 <- win only 
  - clang-tools=16.0.6<- win only 
  - clangxx=16.0.6<- win only 
  - python=3.9
  - nodejs=18.16.*
  - pip=22.3.1
  - rust=1.75
  - rust-std-aarch64-apple-darwin=1.75 <- osx only
  - rust-std-x86_64-apple-darwin=1.75<- osx only
  - rust-std-aarch64-apple-ios=1.75<- ios only
  - rust-std-x86_64-apple-ios=1.75<- ios only
  - rust-std-armv7-linux-androideabi=1.75 <- android only 
  - rust-std-x86_64-linux-android=1.75 <- android only 
  - rust-std-i686-linux-android=1.75<- android only 
  - rust-std-aarch64-linux-android=1.75 <- android only  
  - go=1.18 <- !windows && !android 
  - compiler-rt <- windows 
  - cmake=3.26.3
  - ninja=1.11.0
  - conda-forge::openjdk=17 <- only android 
  - pip:
      - -r requirements.txt
      - -r taskcluster/scripts/requirements.txt
variables:

Miniconda is sloooooow. Unfortunately, caching it is complicated because of how it interracts with pip packages.

Maybe time to switch to micromamba? Resolving and fetching is pretty fast! And its only one static executable :)