Skaffold-Jib fails when trying to override image repository with docker.io/{account}

[ivanporty]

Example used: https://github.com/GoogleContainerTools/skaffold/tree/master/examples/jib
Command line: skaffold dev --default-repo docker.io/{account}

Note: this works fine for Dockerfile builds.

[INFO] Containerizing application to docker.io/{account}/gcr_io_k8s-skaffold_skaffold-jib:7212422d...
[INFO] Retrieving registry credentials for docker.io...
[INFO] Getting base image gcr.io/distroless/java:8...
[INFO] Building dependencies layer...
[INFO] Building resources layer...
[INFO] Building classes layer...
[INFO] 
[INFO] Container entrypoint set to [java, -Djava.security.egd=file:/dev/./urandom, -XX:+UnlockExperimentalVMOptions, -XX:+UseCGroupMemoryLimitForHeap, -cp, /app/resources:/app/classes:/app/libs/*, hello.Application]
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.206 s
[INFO] Finished at: 2019-03-11T16:13:55-04:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:1.0.2:build (default-cli) on project hello-spring-boot: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
[ERROR]  at [Source: (StringReader); line: 2, column: 2]
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
Exception in thread "pool-2-thread-3" java.lang.NoClassDefFoundError: com/google/common/collect/Sets$3
	at com.google.common.util.concurrent.AggregateFutureState.getOrInitSeenExceptions(AggregateFutureState.java:98)
	FATA[0006] exiting dev mode because first run failed: build failed: building [docker.io/{account}/gcr_io_k8s-skaffold_skaffold-jib]: build artifact: maven build failed: exit status 1 

[chanseokoh]

java.lang.NoClassDefFoundError: com/google/common/collect/Sets$3

We had a user with a similar problem with Guava in Gradle before. In that case, the Gradle build was pulling in a different Guava version than the version Jib needs, which doesn't have a method or a class. For Gradle, it is possible to force using a specific dependency version for building. For Maven, no idea, unfortunately. Jib 1.0.2 is using Guava 27.0.1-jre.

Failed to execute goal com.google.cloud.tools:jib-maven-plugin:1.0.2:build (default-cli) on project hello-spring-boot: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null'

This may be a different error.

[chanseokoh]

What if we call out the version in <build><plugins>?

<build>
  <plugins>
    ...
    <plugin>
      <groupId>com.google.guava</groupId>
      <artifactId>guava</artifactId>
      <version>27.0.1-jre</version>
    </plugin>
  </plugins>
</build>

[ivanporty]

I tried this, then this prints:

com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (StringReader); line: 2, column: 2]
	at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1804)
[	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:693)
ER	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:591)
ROR	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1902)
] 	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:757)
 a	at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4141)
t [	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4000)
Sou	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3004)
rce:	at com.google.cloud.tools.jib.json.JsonTemplateMapper.readJson(JsonTemplateMapper.java:101)
 (S	at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.call(RegistryEndpointCaller.java:248)
tring	at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.callWithAllowInsecureRegistryHandling(RegistryEndpointCaller.java:152)
Read	at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.call(RegistryEndpointCaller.java:142)
er);	at com.google.cloud.tools.jib.registry.RegistryClient.callRegistryEndpoint(RegistryClient.java:356)
 li	at com.google.cloud.tools.jib.registry.RegistryClient.pushBlob(RegistryClient.java:304)
ne:	at com.google.cloud.tools.jib.builder.steps.PushBlobStep.call(PushBlobStep.java:124)
 2, 	at com.google.cloud.tools.jib.builder.steps.PushBlobStep.call(PushBlobStep.java:40)
colu	at com.google.common.util.concurrent.CombinedFuture$CallableInterruptibleTask.runInterruptibly(CombinedFuture.java:180)
mn:	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
 2]	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

[chanseokoh]

Basically, Jib is trying to parse the JSON returned from the registry, but it fails because the output is not something parsable as JSON. This usually happens when the endpoint you believe to be a registry endpoint isn't actually a registry and returns an HTML page, which starts with <.

docker.io/{account}/gcr_io_k8s-skaffold_skaffold-jib:7212422d

Is account alive? Try using a different image reference to see if it works.

[ivanporty]

Account is alive, it works with non-Jib Skaffold example, verbatim --default-repo setting, I can see Skaffold images there.

[chanseokoh]

Confirmed that docker.io/some-repository/image doesn't work with Jib. We need to look into why this doesn't work.

As a workaround, use --default-repo registry-1.docker.io/.... (index.docker.io should also work, but AFAIK, this is a legacy name.) See docker/hub-feedback#1767 for more info.

[chanseokoh]

File docker/hub-feedback#1767, but since they have > 900 issues open, I am not so optimistic that they will get to that soon.

In the mean time, we could implement a workaround on the Jib side if the registry is docker.io.

[chanseokoh]

@ivanporty Jib 1.1.0 with the fix released. --default-repo docker.io/... should work.