Skip downloading base image layers if they exists in target repository

[chanseokoh]

Fixes #1673.

This will give substantial speedup for CI/CD environments where people usually don't persist the Jib base image cache.

Design

I tried to do intelligent, fine-grained BLOb checking on each layer. For example, if only a subset of the layers are missing in the target repo, it downloads and caches only those layers.

I also made it avoid checking the layer twice in the situation where it should push missing layers, because currently PushBlobStep always does a Blob check before pushing.

I spent a lot of time tinkering with several different designs and implementations in order to maintain generality and loose coupling between build steps and components, and I think I can settle with this design. So, other advantages from this are

• generality: any layer can be checked and skipped using the same framework with no change; whether or not the layer is from a base image doesn't matter (hence future proof)
• Therefore, checking/downloading/pushing layers can maintain the same level of parallelization as before. For example, it is possible that Jib is downloading one missing base image layer while pushing another missing base image layer as well as building/pushing application layers all at the same time. Potentially and theoretically in the future, the same story can be applied for application layers too. (I am saying theoretically here, because you have to cache applications layers once anyway.)

Speedup

For a large image like adoptopenjdk/openjdk9-openj9,

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  19.681 s
[INFO] Finished at: 2019-07-11T17:53:30-04:00
[INFO] ------------------------------------------------------------------------

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.420 s
[INFO] Finished at: 2019-07-11T17:53:36-04:00
[INFO] ------------------------------------------------------------------------

(However, I admit adoptopenjdk/openjdk9-openj9 is 466MB and way larger than openjdk:8-slim or Distroless.)

Interaction with --offline

As pointed out before, not caching the base image layers may cause a problem for --offline when layers are missing. However, in reality, this would be extremely rare, because --offline can be used only with jib:dockerBuild and jib:buildTar; for those goals, Jib always downloads base image layers. And people do understand they have to do an online build at least once before they can do --offline.

But in an extremely rare case where the user

1. never did jib:dockerBuild or jib:buildTar on their local dev machine
2. and the base image exists in the target repo
3. but did jib:build which skipped downloading layers (i.e., manifest JSON is cached but layers are not)
4. now does --offline jib:dockerBuild without ever trying online jib:dockerBuild

then this case is still covered, as Jib will show the following error message.

[ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:1.3.1-SNAPSHOT:dockerBuild (default-cli) on project helloworld: Cannot run Jib in offline mode; local Jib cache for base image is missing image layer sha256:.... Rerun Jib in online mode with "-Djib.cacheBaseImage=always" to re-download the base image layers. -> [Help 1]

(-Djib.cacheBaseImage=always is actually unnecessary because jib:dockerBuild and jib:buildTar always download layers and --offline is effective only for those goals, but I added it just for the unlikely case someone tries jib:build instead of jib:dockerBuild to do one-time online caching in this situation.)

[chanseokoh]

I have addressed all the comments, AFAIKT. Good for another review pass.

I decided to update CHANGELOG in a follow-up PR.

[briandealwis]

I wonder if this is as easy as adding buildConfiguration.getBaseImageRegistry().equals(buildConfiguration.getTargetImageRegistry()) on in the existing-checker below?

[chanseokoh]

I thought it involves more than that, like checking

JibSystemProperties.useCrossRepositoryBlobMounts()
    && canAttemptBlobMount(authorization, sourceRepository))

to ensure that we will attempt the blob mount. And I think it is that we can attempt blob mount, and it might be possible that a server may not allow/support the mount by returning 202 Accepted instead of 201 Created, in which case RegistryClient.pushBlob() will fall back to do the usual pushing. So I think this is actually complicated.

But I think we are already pretty good without blob mount. Most of the time, the base image will be in the target repository. The blob mount can be a further optimization, but I think it does not give a huge value.

[raizoor]

Hi team,

Have any time for release it? Sounds good to my CI/CD method.

Thanks !!!