Error when pushing image to a private Gitlab Registry with Maven plugin 0.9.7 #746
Labels
Comments
|
Hi @DidierSchonne, For the issue about 403 Forbidden, the log seems to show that the username and the password in you <settings>
...
<servers>
...
<server>
<id>gitlab.mydomain.com:4567</id>
<username>...</username>
<password>...</password>
</server>
</servers>
</settings> |
|
Hi @chanseokoh, <servers>
<server>
<id>gitlab.mydomain.com:4567</id>
<username>***</username>
<password>***</password>
</server>
</servers>I tried many times to copy / paste from a docker login command in order to avoid a misspelling. And don't understand why I have this: |
|
Hi @DidierSchonne , if you run |
FYI, the port part (
This is the sign that Jib was not able to find the credentials from your |
|
The other thing to try is to enable full-auth logging. Follow the instructions for enabling logging with the Google HTTP Client, but instead of "CONFIG" use "ALL". That should show the username and password being passed up to gitlab. |
|
@coollog, @chanseokoh, result is: <settings xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd">
<localRepository xmlns="http://maven.apache.org/SETTINGS/1.1.0">/home/did/.m2/repository</localRepository>
<servers xmlns="http://maven.apache.org/SETTINGS/1.1.0">
<server>
<username>...</username>
<password>***</password>
<id>gitlab.mydomain.com:4567</id>
</server>
</servers>
<pluginGroups xmlns="http://maven.apache.org/SETTINGS/1.1.0">
<pluginGroup>org.apache.maven.plugins</pluginGroup>
<pluginGroup>org.codehaus.mojo</pluginGroup>
</pluginGroups>
</settings>
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
Thank you @briandealwis, I'm going to try your suggestion. |
|
@DidierSchonne and can you confirm you still see |
|
Do either of your username or password contain any XML-relevant characters? Enabling Google HTTP logging should help reveal that. Basically you should see an item like: but to your gitlab instance. Base64-decoding that Authorization blog should reveal your username and password, separated with a colon. |
|
@chanseokoh, I confirm and still have it: @briandealwis, my password contained a special character but I changed my password to avoid any issue with it. I'm currently checking what the logs have to say. |
|
@briandealwis Using your tip, the authorization header is simply not present.
```
$ sudo mvn jib:build -Djava.util.logging.config.file=src/main/resources/log.properties
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/usr/share/maven/lib/guice.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building slow-response 0.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- jib-maven-plugin:0.9.7:build (default-cli) @ slow-response ---
[WARNING] Base image 'gcr.io/distroless/java' does not use a specific image digest - build may not be reproducible
[INFO]
[INFO] Containerizing application to gitlab.mydomain.com:4567/sandbox/slow-response...
[INFO]
[INFO] Retrieving registry credentials for gitlab.mydomain.com:4567...
[INFO] Getting base image gcr.io/distroless/java...
[INFO] Building dependencies layer...
[INFO] Building resources layer...
[INFO] Building classes layer...
juil. 31, 2018 6:28:06 PM com.google.api.client.http.HttpRequest execute
CONFIG: -------------- REQUEST --------------
GET https://gcr.io/v2/distroless/java/manifests/latest
Accept: application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json
Accept-Encoding: gzip
User-Agent: jib 0.9.7 jib-maven-plugin Google-HTTP-Java-Client/1.23.0 (gzip)
juil. 31, 2018 6:28:06 PM com.google.api.client.http.HttpRequest execute juil. 31, 2018 6:28:06 PM com.google.api.client.http.HttpRequest execute juil. 31, 2018 6:28:06 PM com.google.api.client.util.LoggingByteArrayOutputStream close |
|
Oh, I suspect you're communicating over HTTP, and not HTTPS. We don't send auth over HTTP by default, since it's insecure. You'll need to run with the |
|
@briandealwis I'm not sure if that's the case; if that were true, jib should log something like "Required credentials for xxxx/yyyy were not sent because the connection was over HTTP", but I don't see that anywhere in their logs. If that is the case here, though, that may be a bug with jib not displaying the correct error message. |
|
Hmm.. I added SSL through Let's Encrypt to Gitlab and parameterized the Gitlab Registry to use it. |
|
@DidierSchonne I noticed that you're running
or
? Do they give different results? For |
|
@DidierSchonne I see from gitlab's documentation that it supports the use of
|
Closed
|
You're right @chanseokoh, with sudo I don't have my right settings. So I added my user to the Docker group, because I guess that it is needed to push to the registry, right? But with I have this error: Error Log: |
Jib doesn't need privileged rights to push an image to a remote registry (unless you are pushing to your local Docker daemon, which usually requires That said, I'll look into the log and try to think what's happening with the following error. |
|
I am having same issue similar to DidierSchonne |
|
@adorearun yeah, it looks very similar. Interesting. Can you tell us which base and target registries you are using? |
|
@DidierSchonne and @adorearun are you behind proxies by any chance? |
|
GCR is our target registry and I am using maven plugin. It was working earlier only failing now. |
|
@briandealwis Nope. But It was strange for me because it used to work after I upgrade from 0.9.4 version to 0.9.7 in the morning and after an hour it started failing. Does it ring any bell? |
|
@chanseokoh @briandealwis To debug further if I set
I am using google container registry for storing images and I use |
I think you should be using the If that doesn't fix things, could you try enabling Google HTTP logging? It sounds like somehow the SSL certificate validation is failing and so jib is failing over to using HTTP. If you set the |
@adorearun is this still happening? We figured out that, if Jib 0.9.7 can't connect to the registry for reasons like the registry not listening or temporarily down (while @DidierSchonne same goes for you. If this is still happening (while
|
|
@chanseokoh so I confirm that I'm not behind a proxy and I still have the same issue. |
|
@chanseokoh @briandealwis I appreciate all your help and inputs here , please find the output below Open SSL output -
I don't have any certificates with me so here is the output for keytool - |
|
@DidierSchonne we've done some research and we have some evidence here and there to make us believe that your OpenJDK 10 cannot verify Google servers. For example, I downloaded OpenJDK 10 tar.gz, unpacked it, executed There seem to exist several related JDK bugs (e.g., https://bugs.openjdk.java.net/browse/JDK-8207255). @DidierSchonne so I expect if you downgrade to Java 8 for example, you should be able to verify Google servers. @adorearun I did notice you are not using OpenJDK 10, but I wonder if the build tool you are using is picking up some other JRE. |
|
@chanseokoh @briandealwis yep we are not using OpenJDK please find the output |
|
@adorearun can you do to see if your JDK has GlobalSign CA certs? It will ask a password, and if you have never modified your JDK, it should be "changeit". For my JDK for example, I get |
|
@DidierSchonne Probably the easiest thing to do is to use a tool like KeyStore Explorer and import either the GlobalSign Root CA - R2 certificate and the GTS certificates (Google Trust Services) from Google Trust Service's "sample PEM file" (see What roots should we trust for connecting to Google? in the Google Trust Service's FAQ). This is the same process required for using a registry with a self-signed certificate. |
I find using the I tried importing the Google Trust Service's "sample PEM file". Adding it didn't help. (UPDATE: @briandealwis later told me that the following command (the What worked for me was the GlobalSign Root R2 So, if you have permission to modify your JDK, importing missing certificates could be one of possibly many other workarounds. (If you don't have the permission to modify your JDK, another option could be to install one on your home. It's also possible to leave the JDK trusted keystore intact and load a different keystore per individual Java application.) |
|
@chanseokoh @briandealwis, thank you very much for your support, you found the issue. And you're right, with an 8 Java version it's ok. Next month Java 11 should be released, I hope that it will be fixed. Besides that I tried to add a Did I miss something, or did somehting wrong? Or should I open a new issue? Thanks a lot again for your great responsiveness. |
|
@chanseokoh @briandealwis please find the output for global signed certificate in my JDK |
|
@DidierSchonne I opened a new issue for the "Cannot find default setter in class com.google.cloud.tools.jib.maven.JibPluginConfiguration$FromConfiguration" error: #791 Using JDK 8 could be an easy workaround, but just keep in mind that there are other options or workarounds such as importing GlobalSign CA certs into your JDK's default trusted keystore or making your Java programs to load a different trusted keystore, if you do some research, but this might not worth your time. Lastly, it seems that OpenJDK-11 Early Access is also missing the root CA certs, so I also hope it will have them when officially released. |
|
@adorearun then your issue seems different from @DidierSchonne's. Can you confirm that 0.9.8 with |
|
@chanseokoh @briandealwis Once I set the Build image failed: Failed to authenticate with the registry because: peer not authenticated |
|
@adorearun that makes sense. For some unknown reason, you cannot still verify HTTPS servers. (You may wonder why I'm currently clueless. Does this also happen when you use GCR for both the from image and the to image? If possible, can you test with other registries? I'm wondering if you cannot verify only Google registries. |
|
@chanseokoh @briandealwis I believe both from and to will take from GCR .My configuration below <configuration>
<from>
<image>openjdk:8</image>
</from>
<to>
<image>us.gcr.io/projectid/imagename</image>
<credHelper>gcloud</credHelper>
</to>
<allowInsecureRegistries>true</allowInsecureRegistries>
<container>
<jvmFlags>
<jvmFlag>-Dspring.profiles.active=gcpcloud</jvmFlag>
</jvmFlags>
<mainClass>com.XXX.mm.rd.XXX.XXX.myApplication</mainClass>
<args>
<arg>APPLICATION_ARGS</arg>
</args>
<ports>
</ports>
</container>
</configuration> |
|
@adorearun no, Also, I see you are pushing to |
|
@chanseokoh I have seen a strange behavior , I changed us.gcr.io to gcr.io in project 1 then tried jib build and it works. Now I tried project 1 which was working earlier , is failing now. . very strange right. |
|
@adorearun based on all of the logs and info you gave us, here is what I think. But before that, let's examine some of your logs: The log was when You also said
and
So I'm thinking your local environment is showing some unpredictable behavior regarding being able to verify SSL certificates of some servers including |
|
@chanseokoh Thank you for continuous support. I am not sure about my local env because in 0.9.4 it was working perfectly fine earlier (but now even I downgrade it fails) and I am seeing the error from tuesday and upgraded to 0.9.8 still no luck.Is there any way if I use 0.9.4 and make it work because which was sufficient and working fine for me? Also Is there any cache problem ? |
|
@adorearun I don't think Jib is at fault, and the Jib versions seem irrelevant. You said now 0.9.4 is not working. I suggest trying Jib in a totally different environment, on many different laptop/workstation on a different network or WiFi. I also suggest to find and consult an SSL and network expert to look into your machine directly. I am not an expert in this matter, so I feel I'm limited in helping resolve your resolve. Since you said you are not behind a proxy, I even wondered if your machine or your network has been compromised and is under attack by an unknown party, or there might be some security software or firewall installed in your network that you are not aware of, but I'm by no means a security expert. Lastly, there are always some workarounds. Build your image and push it directly into your local Docker daemon, by |
|
@chanseokoh Thank you for the help and I totally understand your point. I will check with my security team over here in mean time,but gcloud docker push works without any problem. |
@adorearun thanks for letting us know. I don't rule out the possibility that it might be the case that the problem happens only for Java applications, applications using the Google HTTP Client Library, applications using a particular JRE setup, etc.
Can you try this? It will be really valuable to know, and I'd try my best effort to find such an environment. If you can find any machine (just using any small hello-world test project such as the one embedded in the Jib source repo) that works, that'd give you a huge lead. I'm sure you'll be able to find such an environment, and I'd start digging into the environmental differences. |
|
@chanseokoh Interesting !!! , the example project with different environment works. I believe the problem with my environment. |
|
Thanks to everyone for the efforts on this issue thread! It looks like the original issues have been resolved, so I removed the milestone. The only issue left is to figure out what environment difference led to the failure @adorearun was running into. |
|
@coollog Thanks all but even my SRE team and networking team were clueless what made this to happen in my machine, will keep you posted if I get any further information. |
|
Closing. Will reopen if any new information comes to light from @adorearun. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description of the issue: Error when pushing image to a private Gitlab Registry with Maven plugin 0.9.7
Expected behavior: An image is built and pushed to my private Gitlab Registry
Steps to reproduce:
settings.xmlin plain text to avoid additional issuessudo mvn compile -X com.google.cloud.tools:jib-maven-plugin:0.9.7:buildEnvironment:
jib-maven-pluginConfiguration:Log output:
Error Log:
Additional Information:
But
sudo mvn compile -X com.google.cloud.tools:jib-maven-plugin:0.9.7:dockerBuildWorks well. I locally have my image built and I'm able to push it to my private Gitlab Repo.
And
If I don't use
sudoI get this error :Thank you for your help.
The text was updated successfully, but these errors were encountered: