Improve artifact caching

[priyawadhwa]

Based on the following feedback from @dgageot, there are a few things we can do to make artifact caching better:

TODO:

• Even if there is no local daemon, check if images exist remotely
• Include filenames in hash
• Sort dependencies
• Loop on all images only once (Only go through images once for artifact caching #1743)
• Don't create a function/file for the hasher
• skaffold cache command to add images built by others to the cache (not doable without tagging with hash)
• images built locally but pushed (push:true) are pushed each time
• sometimes images cannot be retagged:
  - something like rm ~/.skaffold/cache && skaffold build --cache-artifacts=true in examples/kaniko should reproduce the issue
• Changes to the builder config or the artifact config should invalidate the cache
• only get image list from local docker daemon once per build loop
• check for remote images in parallel
• allow caching to work with remote builds (gcb, in-cluster via kaniko)
• store artifacts in cache as they are built
• tags provided with --images to skaffold deploy are kinda ignored.
• Should not show a red error if an image is not already built. (Unable to retrieve ... from cache; this image will be rebuilt.)
• TestNewCache unit test shouldn't depend on cluster the user is running
• In a fully local environment, using minikube or Docker Desktop, and with push either undefined or set to false, we should NOT check for remote digests.

[balopat]

allow caching to work with remote builds (gcb, in-cluster via kaniko) + jib - jib currently always rebuilds:

Retagging cached images...
WARN[0019] error retagging gcr.io/balintp-gcp-lab/gcr.io/k8s-skaffold/skaffold-jib-1:v0.25.0-154-gf75d251e-dirty@sha256:e5848ab3d7bd24b2e8b839f077fd1c35e9d873989ddeaafb890310b427152bb1 as gcr.io/balintp-gcp-lab/gcr.io/k8s-skaffold/skaffold-jib-1:bdfdbc623efcd7fcf2f2f696022d65fd112c62b3888496d681bc6ad01db507bf, caching for this image may not work: Error response from daemon: No such image: gcr.io/balintp-gcp-lab/gcr.io/k8s-skaffold/skaffold-jib-1:v0.25.0-154-gf75d251e-dirty@sha256:e5848ab3d7bd24b2e8b839f077fd1c35e9d873989ddeaafb890310b427152bb1 
WARN[0019] error retagging gcr.io/balintp-gcp-lab/gcr.io/k8s-skaffold/skaffold-jib-2:v0.25.0-154-gf75d251e-dirty@sha256:d6e3571fd4dd1f8df28c08c1661b2d538ad253c890f6d26d08c5ec7935286348 as gcr.io/balintp-gcp-lab/gcr.io/k8s-skaffold/skaffold-jib-2:4e5bd871502ed28e01996a0f8afb305037a8dff06c13d49a6251b9f8c757b60a, caching for this image may not work: Error response from daemon: No such image: gcr.io/balintp-gcp-lab/gcr.io/k8s-skaffold/skaffold-jib-2:v0.25.0-154-gf75d251e-dirty@sha256:d6e3571fd4dd1f8df28c08c1661b2d538ad253c890f6d26d08c5ec7935286348 
St

[zodiac1214]

Is --cache-artifacts checking cache on local only?
Step to reproduce:

1. skaffold build --cache-artifacts, i can see the cache image has been pushed to my remote registry

2.docker rmi -f $(docker images -aq)

3. skaffold build --cache-artifacts again:

Checking cache...
 - my/Image: Not found. Rebuilding.

in step 3, i can see that a new cache image with the same tag has been pushed to my remote registry again

build:
  artifacts:
    - image: myimage
      docker:
        dockerfile: Dockerfile
        target: stage1
  local:
    useDockerCLI: false
    useBuildkit: false

[priyawadhwa]

Hey @zodiac1214, that's correct, the cache only exists locally as of now.