ci: ccache for all github worfklows

[ochafik]

Use https://github.com/hendrikmuhs/ccache-action in all our ci jobs (was already used for cuda / hip jobs)

Example:

• Before:
  • macOS-latest-cmake-x64 succeeded in 9m 57s (build = 5min15s)
  • ubuntu-cpu-cmake succeeded in 5m 16s (build = 2min13s)
• After:
  • macOS-latest-cmake-x64 succeeded in 5m 2s (build = 20s)
  • ubuntu-cpu-cmake succeeded in 3m 11s (build = 7s)

Notes:

• Default seems to be using single cache (Looks like each job gets its own cache by default.)
• May need sccache tweak on Windows?
• May need to share caches between compatible jobs (assuming we can serialize them by group, as concurrent ccache updates are likely counter-productive) if hitting cache limits (update: don’t think there are compatible builds between jobs)
• Expiring files unused in caches for more than 1d w/ Evict old files from the cache prior to saving hendrikmuhs/ccache-action#270 (cf. @slaren's suggestion below); hopefully will allow decent cross-PR caching (TBC in real life)

[slaren]

I would be wary about using too many caches, since there is a limited amount of cache space and it can be easily exceeded, which will cause other caches to be dropped.

Also the current caches keep growing until they hit the size limit, which makes this problem worse. I noticed that they added an option to the action recently that may help with this: hendrikmuhs/ccache-action#270

[ochafik]

I would be wary about using too many caches, since there is a limited amount of cache space and it can be easily exceeded, which will cause other caches to be dropped.

I'm not sure how concurrent "Post ccache" updates work, used distinct keys for fear of race conditions, otherwise we could just use a single cache (maybe let's do this if / when we hit the limits?)

[ggerganov]

This change might have broken the SYCL CI: https://github.com/ggerganov/llama.cpp/actions/runs/13062035808/job/36447068610. Not sure.

[ochafik]

This change might have broken the SYCL CI: https://github.com/ggerganov/llama.cpp/actions/runs/13062035808/job/36447068610. Not sure.

@ggerganov Indeed (example), thanks! I'll see if using sccache as recommended on https://github.com/hendrikmuhs/ccache-action for windows helps.

Also, ubuntu-22-cmake-vulkan & ubuntu-22-cmake-musa's ccache doesn't seem to work (seems to be amongst only jobs left to take 5min to build)

[ggerganov]

These failures started happening recently:

https://github.com/ggerganov/llama.cpp/actions/runs/13178901793/job/36784659823?pr=11213#step:3:38

[ochafik]

These failures started happening recently:

https://github.com/ggerganov/llama.cpp/actions/runs/13178901793/job/36784659823?pr=11213#step:3:38

@ggerganov SSL certificate revocation on windows seems to be stricter (at least w/ the schannel layer), cf. curl/curl#12239 (comment)), with these failures possibly a transient flare on Github's Windows workers (experienced by others.

Possible options i'm evaluating

• Disable ccache on windows (short term)
• Request ccache-action to support setting curl's --ssl-revoke-best-effort flag (or even --ssl-no-revoke)
• Install a different version of curl before running the action maybe (one using openssl instead of schannel)