Maven plugin fails due to mismatching registry certificates despite correct certs

[siamaksade]

When using Google's jib maven plugin to build and push images to OpenShift Online registry, the plugin complains about mismatching certificates:

Caused by: javax.net.ssl.SSLException: hostname in certificate didn't match: <registry.pro-us-east-1.openshift.com> != <*.b9ad.pro-us-east-1.openshiftapps.com> OR <*.b9ad.pro-us-east-1.openshiftapps.com> OR <b9ad.pro-us-east-1.openshiftapps.com>
    at org.apache.http.conn.ssl.AbstractVerifier.verify (AbstractVerifier.java:220)
    at org.apache.http.conn.ssl.BrowserCompatHostnameVerifier.verify (BrowserCompatHostnameVerifier.java:54)
    at org.apache.http.conn.ssl.AbstractVerifier.verify (AbstractVerifier.java:149)
    at org.apache.http.conn.ssl.AbstractVerifier.verify (AbstractVerifier.java:130)
    at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket (SSLSocketFactory.java:339)
    at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection (DefaultClientConnectionOperator.java:123)
    at org.apache.http.impl.conn.AbstractPoolEntry.open (AbstractPoolEntry.java:147)
    at org.apache.http.impl.conn.AbstractPooledConnAdapter.open (AbstractPooledConnAdapter.java:108)
    at org.apache.http.impl.client.DefaultRequestDirector.execute (DefaultRequestDirector.java:415)
    at org.apache.http.impl.client.AbstractHttpClient.execute (AbstractHttpClient.java:641)
    at org.apache.http.impl.client.AbstractHttpClient.execute (AbstractHttpClient.java:576)
    at org.apache.http.impl.client.AbstractHttpClient.execute (AbstractHttpClient.java:554)
    at com.google.api.client.http.apache.ApacheHttpRequest.execute (ApacheHttpRequest.java:65)
    at com.google.api.client.http.HttpRequest.execute (HttpRequest.java:981)
    at com.google.cloud.tools.jib.http.Connection.send (Connection.java:124)

Steps to Reproduce:

$ wget https://github.com/spring-projects/spring-petclinic/archive/master.zip
$ unzip master
$ cd spring-petclinic-master
$ export PRJ_NAME=testprj
$ oc login ONLINE
$ oc new-project $PRJ_NAME
$ mvn compile com.google.cloud.tools:jib-maven-plugin:0.9.6:build -Dimage=registry.pro-us-east-1.openshift.com/$PRJ_NAME/spring-petclinic

This is while the registry certificates are correct:

curl -v --head https://registry.pro-us-east-1.openshift.com
* Server certificate:
*  subject: C=US; ST=North Carolina; L=Raleigh; O=Red Hat Inc.; OU=RHC Cloud Operations; CN=*.pro-us-east-1.openshift.com
*  start date: May 19 00:00:00 2018 GMT
*  expire date: May 23 12:00:00 2019 GMT
*  subjectAltName: host "registry.pro-us-east-1.openshift.com" matched cert's "*.pro-us-east-1.openshift.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA

[chanseokoh]

I wonder where this openshiftapps.com comes from.

$ nslookup openshiftapps.com
Server:         127.0.0.1
Address:        127.0.0.1#53

Non-authoritative answer:
*** Can't find openshiftapps.com: No answer

Not an expert in this subject, so just ask wild questions: are you behind a proxy? Does the domain have DNS in your network?

[briandealwis]

@chanseokoh use dig: dig openshiftapps.com any

I don't have an OpenShift account (yet), so when I try to jib:deploy an image to that reference, Jib errors with a 401.

[chanseokoh]

dig openshiftapps.com ANY says the same. openshiftapps.com has no A type. The name servers that are supposed to give the answer have no entry for it, I believe. BTW, nslookup works too.

$ nslookup
> set type=ANY
> openshiftapps.com
Server:         127.0.0.1
Address:        127.0.0.1#53

Non-authoritative answer:
openshiftapps.com
        origin = ns1.p23.dynect.net
        mail addr = rhcloud-security.redhat.com
        serial = 649
        refresh = 3600
        retry = 600
        expire = 604800
        minimum = 1800
openshiftapps.com       nameserver = ns1.p23.dynect.net.
openshiftapps.com       nameserver = ns2.p23.dynect.net.
openshiftapps.com       nameserver = ns4.p23.dynect.net.
openshiftapps.com       nameserver = ns3.p23.dynect.net.
openshiftapps.com       text = "google-site-verification=hB07U0i727LYgqIyfRKVwej1D7v_PHBRX25GO9VNfRk"

Authoritative answers can be found from:
> set type=A
> server ns1.p23.dynect.net 
Default server: ns1.p23.dynect.net
Address: 2001:500:90:1::23#53
Default server: ns1.p23.dynect.net
Address: 208.78.70.23#53
> openshiftapps.com
Server:         ns1.p23.dynect.net
Address:        2001:500:90:1::23#53

*** Can't find openshiftapps.com: No answer
> 

[briandealwis]

@siamaksade I suspect this was a configuration problem that has since been fixed with OpenShift. I'm able to push an image to a slightly-different OpenShift registry with Jib 0.9.7-SNAPSHOT.

(0.9.6 has authentication bug that affects OpenShift, fixed with #614. You can build from head and use Jib 0.9.7-SNAPSHOT.)

We use the Google HTTP client under the hood which supports voluminous logging with java.util.logging. If you're still having a problem then please create a log.properties with the following (edited to use CONFIG to avoid including authorization data):

# must set handler level too <https://stackoverflow.com/a/470492/600339>
handlers = java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level=CONFIG
com.google.api.client.http.level=CONFIG

and then run your Maven command with:

mvn jib:build -Dimage=registry.pro-us-east-1.openshift.com/testprj/ \
  -Djava.util.logging.config.file=path/to/log.properties

We do a lot of requests in parallel so the output can be confusing to see; I wonder if we could serialize execution to simplify this.

---

By background, OpenShift provides a registry as registry.<location>.openshift.com. In my case, did the following:

1. Created an OpenShift starter account on ca-central-1
2. Created a testprj
3. Used docker login -u <login> -p $(oc whoami -t) registry.starter-ca-central-1.openshift.com.
4. Built an image to registry.starter-ca-central-1.openshift.com/testprj/foo with 0.9.7-SNAPSHOT.

$ mvn jib:build -Dimage=registry.starter-ca-central-1.openshift.com/testprj/foo
[…]
[INFO] Built and pushed image as registry.starter-ca-central-1.openshift.com/testprj/foo
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

[briandealwis]

Forgot to add: openshiftapps.com is used when creating a route to a deployed service. For example, I had a service called docker-registry that was exposed as docker-registry-testprj.193b.starter-ca-central-1.openshiftapps.com

[coollog]

@siamaksade Feel free to reopen if the issue persists.