support for credsStore or credential-helper in new docker versions #657

johnflavin · 2017-03-02T14:33:55Z

The current implementation of RegistryAuth has several static builder methods that parse the credentials out of the supplied or default config.json file. That strategy does not work in the newest docker version (v17.03.0-ce), as the structure of that file has changed and the credentials are no longer be stored there. They are stored in a more secure system-native way (i.e. the keychain on macOS) using the docker credential helpers.

These changes have already rolled out on the beta channel of Docker for Mac. When you use docker login to store your credentials, they are put into the keychain. The config.json file looks like this:

$ cat ~/.docker/config.json
{
	"auths": {
		"https://index.docker.io/v1/": {}
	},
	"credsStore": "osxkeychain"
}

So that presents a problem for Remote API users like us. We can no longer read the credentials from a file. I don't know exactly how to interact with the credential helpers, whether they have their own API or if we need to use system calls to execute the binary. It is possible that there is already some easy solution and I just don't know what it is.

I want to open this ticket to discuss the coming changes and brainstorm solutions.

The text was updated successfully, but these errors were encountered:

mattnworb · 2017-03-02T15:42:18Z

Good catch, and thanks for opening the issue up.

mattnworb · 2017-05-16T11:42:41Z

seems like the API for the credential helpers is defined at the bottom of https://github.com/docker/docker-credential-helpers#development - a client is support to execute the process passing certain values in the first argument

NicoTexas · 2017-06-01T02:35:54Z

This is true also on Windows as Docker login is storing in wincred.

{ "auths": { "docker-dev.domain.com": {} }, "credsStore": "wincred" }

mattnworb · 2017-06-01T18:22:14Z

@NicoTexas yea there is no support in docker-client for credsStore or helpers yet.

I think it would be possible to add support for the credential helpers to this library, since the API seems pretty well established, although it would feel weird that this library might end up execing random processes on the user's host.

beihaifeiwu · 2017-07-03T05:39:07Z

I got the fllow exception on windows after docker upgrade:

[INFO] --- dockerfile-maven-plugin:1.3.0:build (default) @ nagrand-ws ---
[INFO] Building Docker context F:\Source\nagrand\nagrand-ws
[INFO]
[INFO] Image will be built as 10.0.25.21:5000/nagrand-ws:2.1-SNAPSHOT
[INFO]
[WARNING] An attempt failed, will retry 1 more times
org.apache.maven.plugin.MojoExecutionException: Could not build image
        at com.spotify.plugin.dockerfile.BuildMojo.buildImage(BuildMojo.java:164)
        at com.spotify.plugin.dockerfile.BuildMojo.execute(BuildMojo.java:95)
        at com.spotify.plugin.dockerfile.AbstractDockerMojo.tryExecute(AbstractDockerMojo.java:219)
        at com.spotify.plugin.dockerfile.AbstractDockerMojo.execute(AbstractDockerMojo.java:208)
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
        at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
        at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:309)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:194)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:107)
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:993)
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:345)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:191)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: com.spotify.docker.client.exceptions.DockerException: com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.JsonMappingException: Can not construct instance of com.spotify.docker.client.messages.RegistryAuth: no String-argument constructor/factory method to deserialize from String value ('wincred')
 at [Source: N/A; line: -1, column: -1] (through reference chain: java.util.LinkedHashMap["credsStore"])
        at com.spotify.docker.client.auth.ConfigFileRegistryAuthSupplier.authForBuild(ConfigFileRegistryAuthSupplier.java:96)
        at com.spotify.docker.client.auth.MultiRegistryAuthSupplier.authForBuild(MultiRegistryAuthSupplier.java:77)
        at com.spotify.docker.client.DefaultDockerClient.build(DefaultDockerClient.java:1371)
        at com.spotify.docker.client.DefaultDockerClient.build(DefaultDockerClient.java:1348)
        at com.spotify.plugin.dockerfile.BuildMojo.buildImage(BuildMojo.java:157)
        ... 25 more
Caused by: com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.JsonMappingException: Can not construct instance of com.spotify.docker.client.messages.RegistryAuth: no String-argument constructor/factory method to deserialize from String value ('wincred')
 at [Source: N/A; line: -1, column: -1] (through reference chain: java.util.LinkedHashMap["credsStore"])
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:270)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.DeserializationContext.instantiationException(DeserializationContext.java:1456)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1012)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.deser.ValueInstantiator._createFromStringFallbacks(ValueInstantiator.java:370)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromString(StdValueInstantiator.java:315)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromString(BeanDeserializerBase.java:1283)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:159)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:150)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:517)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:362)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:27)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1187)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:314)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:148)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3770)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2099)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.ObjectMapper.treeToValue(ObjectMapper.java:2596)
        at com.spotify.docker.client.DockerConfigReader.parseDockerConfig(DockerConfigReader.java:94)
        at com.spotify.docker.client.DockerConfigReader.fromConfig(DockerConfigReader.java:48)
        at com.spotify.docker.client.auth.ConfigFileRegistryAuthSupplier.authForBuild(ConfigFileRegistryAuthSupplier.java:94)
        ... 29 more
[INFO] Building Docker context F:\Source\nagrand\nagrand-ws

Is there any workaround about this issue ?

jordijansen · 2017-07-03T07:03:34Z

+1
I'm getting the same error as @beihaifeiwu

After ugrapding to 17.06.0-ce. I'm now unable to build any image using the spotify-dockerfile plugin.

jordijansen · 2017-07-06T13:50:52Z

@beihaifeiwu if you don't use the credStore you can remove it from your config.json (C:\Users<user>.docker\config.json). Remove the credStore key from the JSON object. You then at least can build images again. This is just a temporary workaround.

ranef · 2017-07-25T14:50:18Z

I faced the same problem and tried the workaround suggested of removing the credStore key from the config.json. In my environment this was the only entry in the json file and when I removed it I got an exception about the file being empty.
The only thing that solved the problem in my case was removing the entire file

ljnelson · 2017-08-15T19:40:18Z

It seems that if you remove the ~/.docker/config.json file, then the plugin will fall back to looking in your Maven's settings information. That's good. But then unless you jump through other hoops your docker command line won't work, right? Isn't this a bug, not an enhancement?

mattnworb · 2017-08-15T20:07:50Z

@ljnelson I am not sure if the semantics makes a difference, this is something that was introduced in docker relatively recently (~6 months) which docker-client does not yet have support for. We would welcome any PRs to add support for credsStore, but have not had a chance to do it ourselves yet.

ljnelson · 2017-08-15T21:27:36Z

I see; thanks for the information!

ranef · 2017-08-16T08:57:48Z

The docker command line works fine without the config.json file
One thing I noticed which is pretty annoying is that config.json file is recreated each time the docker service is started (e.g. every windows restart) so I have to keep deleting it each time.
Is there any dummy entries I can put in this file that will eliminate the credStore error but will prevent docker from recreating it?

jschneid-nmdp · 2017-10-16T23:59:45Z

When using the useMavenSettingsForAuth plugin configuration option, it appears dockerfile-maven-plugin version 1.3.6 requires the Docker server password in the Maven settings file to be stored as cleartext, not "encrypted" via mvn --encrypt-password.

stromnet · 2018-02-12T12:33:42Z

FYI: As described earlier, OS X installation adds credsStore to config file causing some operations to fall, for example .pull.

Our temporary workaround (we don't need auth since it's internal repos) is to create the client with .registryAuthSupplier(new MultiRegistryAuthSupplier(Collections.emptyList()))

smythie86 · 2018-04-03T18:15:48Z

I believe since my PR was merged above this issue can be closed.

alibbbian · 2021-08-05T08:50:41Z

I got the fllow exception on windows after docker upgrade:

[INFO] --- dockerfile-maven-plugin:1.3.0:build (default) @ nagrand-ws ---
[INFO] Building Docker context F:\Source\nagrand\nagrand-ws
[INFO]
[INFO] Image will be built as 10.0.25.21:5000/nagrand-ws:2.1-SNAPSHOT
[INFO]
[WARNING] An attempt failed, will retry 1 more times
org.apache.maven.plugin.MojoExecutionException: Could not build image
        at com.spotify.plugin.dockerfile.BuildMojo.buildImage(BuildMojo.java:164)
        at com.spotify.plugin.dockerfile.BuildMojo.execute(BuildMojo.java:95)
        at com.spotify.plugin.dockerfile.AbstractDockerMojo.tryExecute(AbstractDockerMojo.java:219)
        at com.spotify.plugin.dockerfile.AbstractDockerMojo.execute(AbstractDockerMojo.java:208)
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
        at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
        at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:309)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:194)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:107)
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:993)
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:345)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:191)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: com.spotify.docker.client.exceptions.DockerException: com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.JsonMappingException: Can not construct instance of com.spotify.docker.client.messages.RegistryAuth: no String-argument constructor/factory method to deserialize from String value ('wincred')
 at [Source: N/A; line: -1, column: -1] (through reference chain: java.util.LinkedHashMap["credsStore"])
        at com.spotify.docker.client.auth.ConfigFileRegistryAuthSupplier.authForBuild(ConfigFileRegistryAuthSupplier.java:96)
        at com.spotify.docker.client.auth.MultiRegistryAuthSupplier.authForBuild(MultiRegistryAuthSupplier.java:77)
        at com.spotify.docker.client.DefaultDockerClient.build(DefaultDockerClient.java:1371)
        at com.spotify.docker.client.DefaultDockerClient.build(DefaultDockerClient.java:1348)
        at com.spotify.plugin.dockerfile.BuildMojo.buildImage(BuildMojo.java:157)
        ... 25 more
Caused by: com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.JsonMappingException: Can not construct instance of com.spotify.docker.client.messages.RegistryAuth: no String-argument constructor/factory method to deserialize from String value ('wincred')
 at [Source: N/A; line: -1, column: -1] (through reference chain: java.util.LinkedHashMap["credsStore"])
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:270)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.DeserializationContext.instantiationException(DeserializationContext.java:1456)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1012)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.deser.ValueInstantiator._createFromStringFallbacks(ValueInstantiator.java:370)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromString(StdValueInstantiator.java:315)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromString(BeanDeserializerBase.java:1283)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:159)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:150)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:517)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:362)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:27)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1187)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:314)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:148)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3770)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2099)
        at com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.ObjectMapper.treeToValue(ObjectMapper.java:2596)
        at com.spotify.docker.client.DockerConfigReader.parseDockerConfig(DockerConfigReader.java:94)
        at com.spotify.docker.client.DockerConfigReader.fromConfig(DockerConfigReader.java:48)
        at com.spotify.docker.client.auth.ConfigFileRegistryAuthSupplier.authForBuild(ConfigFileRegistryAuthSupplier.java:94)
        ... 29 more
[INFO] Building Docker context F:\Source\nagrand\nagrand-ws

Is there any workaround about this issue ?

delete the file ./docker/config.json，but The file appears again after restart docker.

davidxia added help wanted question labels Mar 2, 2017

johnflavin mentioned this issue Mar 15, 2017

pulling an image from private repository always needs authentication #682

Closed

umcodemonkey mentioned this issue Apr 21, 2017

Can't push to private Docker repo (ECR) from Mac spotify/docker-maven-plugin#321

Closed

mattnworb changed the title ~~RegistryAuth is broken in new docker version~~ support for credsStore or credential-helper in new docker versions Jun 6, 2017

This was referenced Jun 6, 2017

Pushing with authentication doesn't work with credsStore osxkeychain spotify/docker-maven-plugin#328

Closed

Support Docker external credential stores spotify/dockerfile-maven#24

Closed

declension mentioned this issue Jun 14, 2017

Warn when parsing local config that refers to unsupported external credentials stores #797

Closed

davidxia added the enhancement label Jun 30, 2017

davidxia mentioned this issue Jun 30, 2017

NullpointerException after Docker upgrade spotify/dockerfile-maven#33

Closed

dbeaumont mentioned this issue Sep 14, 2017

unable to push to docker hub from plugin although i can push using docker push spotify/dockerfile-maven#51

Closed

smythie86 mentioned this issue Dec 8, 2017

Add support for credsStore in new docker versions #945

Merged

mattnworb closed this as completed Apr 3, 2018

atomotic mentioned this issue Aug 7, 2018

error on mvn package dlcs/elucidate-server#89

Open

ankitsri11 mentioned this issue Oct 12, 2018

Unable to run pipeline using docker-elastic-agent gocd-contrib/docker-elastic-agents-plugin#59

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support for credsStore or credential-helper in new docker versions #657

support for credsStore or credential-helper in new docker versions #657

johnflavin commented Mar 2, 2017

mattnworb commented Mar 2, 2017

mattnworb commented May 16, 2017 •

edited

Loading

NicoTexas commented Jun 1, 2017 •

edited

Loading

mattnworb commented Jun 1, 2017

beihaifeiwu commented Jul 3, 2017

jordijansen commented Jul 3, 2017

jordijansen commented Jul 6, 2017 •

edited

Loading

ranef commented Jul 25, 2017

ljnelson commented Aug 15, 2017

mattnworb commented Aug 15, 2017

ljnelson commented Aug 15, 2017

ranef commented Aug 16, 2017

jschneid-nmdp commented Oct 16, 2017

stromnet commented Feb 12, 2018 •

edited

Loading

smythie86 commented Apr 3, 2018

alibbbian commented Aug 5, 2021 •

edited

Loading

support for credsStore or credential-helper in new docker versions #657

support for credsStore or credential-helper in new docker versions #657

Comments

johnflavin commented Mar 2, 2017

mattnworb commented Mar 2, 2017

mattnworb commented May 16, 2017 • edited Loading

NicoTexas commented Jun 1, 2017 • edited Loading

mattnworb commented Jun 1, 2017

beihaifeiwu commented Jul 3, 2017

jordijansen commented Jul 3, 2017

jordijansen commented Jul 6, 2017 • edited Loading

ranef commented Jul 25, 2017

ljnelson commented Aug 15, 2017

mattnworb commented Aug 15, 2017

ljnelson commented Aug 15, 2017

ranef commented Aug 16, 2017

jschneid-nmdp commented Oct 16, 2017

stromnet commented Feb 12, 2018 • edited Loading

smythie86 commented Apr 3, 2018

alibbbian commented Aug 5, 2021 • edited Loading

mattnworb commented May 16, 2017 •

edited

Loading

NicoTexas commented Jun 1, 2017 •

edited

Loading

jordijansen commented Jul 6, 2017 •

edited

Loading

stromnet commented Feb 12, 2018 •

edited

Loading

alibbbian commented Aug 5, 2021 •

edited

Loading