InsecureRegistryException thrown for all registries including the default registry and distroless registry

[oscarnylander]

Description of the issue:
InsecureRegistryException thrown for all registries (including default where distroless image is hosted)

Expected behavior:
mvn jib:build completes without failing due to secure registries being insecure

Steps to reproduce:
mvn jib:build

Environment:
MacOS 10.14, Maven 3.5.4, Java 9.0.4

jib-maven-plugin Configuration:

      <plugin>
        <groupId>com.google.cloud.tools</groupId>
        <artifactId>jib-maven-plugin</artifactId>
        <version>0.9.13</version>
        <configuration>
          <from>
            <image>openjdk:10</image>
          </from>
          <to>
            <image>${env.REMOTE_IMAGE_URL}</image>
            <credHelper>ecr-login</credHelper>
            <tags>
              <tag>JIB-TEST-IMAGE</tag>
            </tags>
          </to>
        </configuration>
      </plugin>

Log output:

Caused by: com.google.cloud.tools.jib.registry.InsecureRegistryException: Failed to verify the server at https://registry.hub.docker.com/v2/library/openjdk/manifests/10 because only secure connections are allowed.
    at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.handleUnverifiableServerException (RegistryEndpointCaller.java:169)
    at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.callWithAllowInsecureRegistryHandling (RegistryEndpointCaller.java:154)
    at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.call (RegistryEndpointCaller.java:141)
    at com.google.cloud.tools.jib.registry.RegistryClient.callRegistryEndpoint (RegistryClient.java:355)
    at com.google.cloud.tools.jib.registry.RegistryClient.pullManifest (RegistryClient.java:225)
    at com.google.cloud.tools.jib.registry.RegistryClient.pullManifest (RegistryClient.java:233)
    at com.google.cloud.tools.jib.builder.steps.PullBaseImageStep.pullBaseImage (PullBaseImageStep.java:194)
    at com.google.cloud.tools.jib.builder.steps.PullBaseImageStep.call (PullBaseImageStep.java:116)
    at com.google.cloud.tools.jib.builder.steps.PullBaseImageStep.call (PullBaseImageStep.java:57)
    at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly (TrustedListenableFutureTask.java:127)
    at com.google.common.util.concurrent.InterruptibleTask.run (InterruptibleTask.java:57)
    at com.google.common.util.concurrent.TrustedListenableFutureTask.run (TrustedListenableFutureTask.java:80)
    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1167)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:641)
    at java.lang.Thread.run (Thread.java:844)

Additional Information:

[chanseokoh]

Hi @hedvigoscar,

We are aware that some JDKs cannot verify certain servers. We've seen it with certain JDK 9 and JDK 10 (like in this case). You said your environment is Java 9.0.4. I just downloaded OpenJDK 9.0.4 from jdk.java.net and noticed it cannot verify Docker Hub. (BTW, OpenJDK 9.0.4 does not include security fixes and no longer recommended for use in production.)

$ mkdir /tmp/test && cd /tmp/test
$ wget https://download.java.net/java/GA/jdk9/9.0.4/binaries/openjdk-9.0.4_linux-x64_bin.tar.gz
$ tar zxvf openjdk-9.0.4_linux-x64_bin.tar.gz
$ cat > A.java <<EOF
public class A {
  public static void main(String[] args) throws Exception {
    new java.net.URL("https://registry.hub.docker.com").openStream();
  }
}
EOF
$ ./jdk-9.0.4/bin/javac A.java
$ ./jdk-9.0.4/bin/java A

Then you get the following error:

Exception in thread "main" javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.ssl.Alerts.getSSLException(Alerts.java:198)
	at java.base/sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1969)
	at java.base/sun.security.ssl.Handshaker.fatalSE(Handshaker.java:345)
	at java.base/sun.security.ssl.Handshaker.fatalSE(Handshaker.java:339)
	at java.base/sun.security.ssl.ClientHandshaker.checkServerCerts(ClientHandshaker.java:1968)
	at java.base/sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1777)
	at java.base/sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:264)
	at java.base/sun.security.ssl.Handshaker.processLoop(Handshaker.java:1092)
	at java.base/sun.security.ssl.Handshaker.processRecord(Handshaker.java:1026)
	at java.base/sun.security.ssl.SSLSocketImpl.processInputRecord(SSLSocketImpl.java:1137)
	at java.base/sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1074)
	at java.base/sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
	at java.base/sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1402)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1429)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
	at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1581)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1509)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:245)
	at java.base/java.net.URL.openStream(URL.java:1117)
	at A.main(A.java:3)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
	at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:290)
	at java.base/sun.security.validator.Validator.validate(Validator.java:264)
	at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:343)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:226)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:133)
	at java.base/sun.security.ssl.ClientHandshaker.checkServerCerts(ClientHandshaker.java:1947)
	... 17 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
	at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380)
	... 23 more

[oscarnylander]

@chanseokoh That's odd!

I tried using Java 11 and for some reason that does appear to work.

[chanseokoh]

This is due to some popular root CA certs missing or incomplete in some OpenJDK versions. Looks like the very first OpenJDK 9 had an empty cacerts, and then, Oracle open-sourced a set of root certificates issued by the CAs who signed some Oracle agreement to include the certificates in later JDK builds. However, I remember some important CA certs were still missing in OpenJDK 11 Early Access, but maybe this last bit is resolved this for OpenJDK 11 GA, so hopefully everything works out of the box starting from OpenJDK 11.

[rosenk]

With what version of the JDK it is supposed to work? I can't find one that works.

[oscarnylander]

@rosenk I've gotten it to work with OpenJDK 11.

[rosenk]

@hedvigoscar Already tried that. Dosen't work with the distroless registry

[chanseokoh]

Hi @rosenk,

Does #1216 (comment) work if you use https://gcr.io instead of https://registry.hub.docker.com with your JDK? (Make sure you use the same JDK used by your build system. For example, mvn -v will show you the exact path to the JDK.) Also try that with Java 8.

A common root cause is due to proxy problems. Check if you are behind any proxy.

If it works, then the problem may be specific to Jib. In that case, please open a new issue with more details: your configs, environment, and detailed logs.

Lastly, check #1247 too.