Fix docker layers caching in CI

[diegoximenes]

Resolves NIT-2643

Fix docker layers caching in CI.

It doesn't improve CI running time by much though, something around 20% better.

You can check it working by re-running the CI workflows of this PR, or creating a new PR that doesn't modify Dockerfiles based on the branch of this PR, and check cache being hit.

[Tristan-Wilson]

This looks good but I'm trying to understand it. Why does docker compose not cache the layers and we have to explicitly set a cache dir and use buildx?

[diegoximenes]

This looks good but I'm trying to understand it. Why does docker compose not cache the layers and we have to explicitly set a cache dir and use buildx?

Here are some references related to docker layer caching not being natively supported in GitHub actions:

• Docker caching actions/cache#31
• https://stackoverflow.com/a/61504649

We need to explicitly set a cache dir so actions/cache@v3 and docker buildx can use it.
docker buildx will build images trying to retrieve docker layers from this dir, and will save built docker layers into this dir.
We are not able to replicate that behavior with docker compose.
To run a image, docker compose will use the images built through docker buildx.
In the end of this workflow, actions/cache@v3 will be responsible to save what is stored in this dir, so it will be able to restore into this dir in the beginning of this workflow in future GitHub actions runs.

[diegoximenes]

Requesting review again after resolving git conflicts

[joshuacolvin0]

LGTM