Allow Maven properties only for configuraion

[jstrachan]

TL;DR; it'd be nice to add strings like "docker.from", "docker.registry" to the build configuration class.
e.g. on all these @parameter values:
https://github.com/rhuss/docker-maven-plugin/blob/master/src/main/java/org/jolokia/docker/maven/config/BuildImageConfiguration.java#L15

Also allowing maven properties to be used to define env vars & ports would be nice using docker.env.NAME=value and docker.port.container.NAME=8080 as properties.

Rather long background on this....
So on the fabric8 project we're reusing your awesome maven plugin for generating & pushing docker images, we're also generating kubernetes json files too:
http://fabric8.io/v2/mavenPlugin.html#generating-the-json

then a parallel project called Jube which implements a pure kubernetes implementation without docker (for folks running java middleware on non-linux or non-docker based operating systems) - which makes docker-like images (which are just zips with shell scripts so can be run on any platform that has a JVM):
http://fabric8.io/jube/mavenPlugin.html#building-your-image-zip

we've found there's often lots of common stuff between these 3 maven plugins (docker / fabric8 / jube) for building images or generating kubernetes json. Things like the base image name, the image name, the env vars, the ports.

Using maven properties can help keep configuration DRY; since we can inherit stuff and put configuration into base projects (as its often folks have multiple images with the same base; or often expose similar ports or env vars).
e.g. this parent project defines a bunch of ActiveMQ stuff so defines the image name (based on maven aritfact) along with exposing the jolokia contianer port and defines the docker.baseImage.
https://github.com/jstrachan/quickstarts/blob/changes/apps/pom.xml#L38

(Note I'm just about to migrate all these from 0.9.x :)

So what would be nice is to be able to use maven properties to configure the build configuration.
e.g. on all these @parameter values:
https://github.com/rhuss/docker-maven-plugin/blob/master/src/main/java/org/jolokia/docker/maven/config/BuildImageConfiguration.java#L15

it'd be nice to have a name, like "docker.from", "docker.registry" so we can share configuration between parent/child poms - and share configuration between maven plugins too.

One thing we found with the generation of kubernetes json and jube image zips; things like env vars and ports tend to be additive; you often have base things and wish to add new ports; or override env vars etc. So again using maven properties turned out to be a nice way to do it. e.g. any pom.xml can expose an extra container port via adding

    <docker.port.container.jolokia>8778</docker.port.container.jolokia>

(BTW we used the '.container' suffix as kubernetes (and docker really) has the idea of internal container ports and external host ports; while the docker images don't yet allow you to specify the latter, certainly in kubernetes we can).

So for env vars and ports we found that just adding a little bit of code that if the user has not supplied a List/Map of ports/env vars, we look in the maven properties and build the list/map from those. e.g.

Here's the code:
https://github.com/jstrachan/fabric8/blob/changes/fabric8-maven-plugin/src/main/java/io/fabric8/maven/JsonMojo.java#L373-373

we're currently using "fabric8.env.FOO" as the naming convention; but I'd prefer it if we could all use "docker.env.FOO" really for all 3 maven plugins; then users can just configure things once in a nice DRY way.
e.g.
https://github.com/jstrachan/quickstarts/blob/changes/apps/fabric8-mq-autoscaler/pom.xml#L37-37

we use the same approach to defining labels in kubernetes json too; it works very nicely in maven as maven properties are much more composable.

I can supply a PR if you like; its really just adding a few strings to some @parameter values; and adding a factory method on the env vars / ports if there is nothing configured to look at the maven properties and create the List/Map from them. I figured I'd explain my thoughts first to see what you think

[jstrachan]

ah - this might not be possible with the nested DTOs used to configure images/builds. e.g. I tried adding...

public class ImageConfiguration implements StartOrderResolver.Resolvable {

    @Parameter(required = true, property = "docker.image")
    private String name;

and it seems to be ignored ;)

[jstrachan]

at least the XML in user projects could use ${docker.image}, ${docker.from}, ${docker.registry} I guess; we could still do the env var / ports stuff though? Then the maven plugin can be defined once in a base project and then configured using maven properties in child projects

[rhuss]

Good points, James. Currently I'm at Devoxx, hopefully have some time this evening and maybe come up with a good solution for your use case. Have still to read through your comments completely first, though ;-)

[jstrachan]

BTW I went with just lots of copy/pasting for now adding in lots of ${docker.foo} variables everywhere
https://github.com/fabric8io/quickstarts/blob/master/quickstarts/java/camel-spring/pom.xml#L153

its only for env and ports couldn't copy/paste the same blob of XML into all projects. It'd be really nice for this to be the default though, so there's less to type/maintain

[rhuss]

Just thought a bit about your suggestion I think it would be indeed awesome to have some easy way to share configuration properties among the various plugins.

However I doubt a bit that the properties-road is the way to go, because:

• Sharing properties subtly tie the plugins together because of shared naming of props (i.e. if the kube plugin changes naming, the other plugins must follow). Might not be so dangerous, but if a problem occurs here, that will be quite hard to detect.
• The nature of the props are not flat (envs, ports are maps) and a flat property list doesn't really fit. Of course, one can find a way around this (with naming conventions as you suggest), but personally I would find to have a hierarchically configuration format (XML, JSON, YAML) would feel better.

I think supporting properties as you suggest is fine, but we should put the responsibility to one plugin, with the other plugins supporting the syntax. Let's say fabric8 is the leading plugin, then one could say the docker-maven-plugin has support for the fabric8 plugin (for a specific version). So its clear, that they are separate.

The, it would be nice to have a universal, probably external configuration format (which then can be referenced from the plugins' configuration) describing this. There are already some (fig, kubernetes, soon: groups.yml when Docker supports intrinsic composition).

Another question for me is: For the fabric8 case, do you always have a single image or should multiple images be configured via properties ? If a single image is sufficient, I think I could have a nice default with properties which simply also only configure a single image (or add an extra image to the list of configured images in the docker-maven-plugin's config).

For globally applicable properties (registry, etc.) I will introduce properties, too (Currently the registry stuff still needs some polishing).

'will start to implement the property default for a single image soon ...

[rhuss]

External configuration

In order to avoid implicit dependencies on property names I suggest to create an external link to an 'external' configuration. This external configuration then can specify one or more image configuration (with run and build parts). Examples for external configurations are:

• Properties from the fabric8 Plugin or even direct access to the plugin's configuration (if this is feasible from within Maven).
• Reference to a fig.yml (or later for groups.yml when Docker groups are released) for extracting the plugin's configuration.

My suggestion is to include this as part of the regular configuration, e.g.

<configuration>
 <images>
   <image>
     <reference>
       <!-- Lookup the known properties evaluated by the fabric8 plugin -->
       <type>fabric8</type>
     </reference>
   </image>
   <image>
     <reference>
       <!-- Use a Fig definition and add one or more images to the list -->
       <type>fig</type>
       <definition>src/main/resources/fig.yml</definition>
     </reference>
   </image>
   <!-- "Normal" image definitions -->
   .....
 </images>
</configuration>

Then the docker-maven-plugin has a list of plugins (keyed by type) which know how to create image definitions for these external systems. I like the idea of having plugins for a plugin ;-) Also this would directly support my use case of using fig compositions files directly for this plugin.

What do you think about this idea ?

We could go even farther: Considered we have such a plugin infrastructure, we could use Maven's DI (via Plexus) to get these config plugin from outside of this docker plugin. That way (if this is possible, still have to lookup some Plexus magic) the fabric8 plugin itself could maintain its configuration handling (and property lookup) so that there is even less dependency. The only drawback would be that the fabric8 Plugin then would have some code dependencies on this docker-maven-plugin in order to implement the interface

[rhuss]

I implemented a first version for this feature for referenced configuration. The snapshot 0.10.5-SNAPSHOT has been pushed to Maven central.

You can try it with the following configuration:

  <profile>
      <id>props</id>
      <properties>
        <docker.name>jolokia/${project.artifactId}:${project.version}</docker.name>
        <docker.alias>service</docker.alias>
        <docker.from>${image}</docker.from>
        <docker.assemblyDescriptor>src/main/docker-assembly.xml</docker.assemblyDescriptor>
        <docker.env.CATALINA_OPTS>-Xmx32m</docker.env.CATALINA_OPTS>
        <docker.env.JOLOKIA_OPTS>1</docker.env.JOLOKIA_OPTS>
        <docker.ports.jolokia.port>8080</docker.ports.jolokia.port>
        <docker.wait.url>http://localhost:${jolokia.port}/jolokia</docker.wait.url>
        <docker.time>10000</docker.time>
      </properties>
      <build>
        <plugins>
          <plugin>
            <groupId>org.jolokia</groupId>
            <artifactId>docker-maven-plugin</artifactId>
            <configuration combine.self="override">
              <images>
                <image>
                  <reference>
                    <type>props</type>
                    <prefix>docker</prefix>
                  </reference>
                </image>
              </images>
            </configuration>
          </plugin>
        </plugins>
      </build>
    </profile>

Is that approximately that what you need ? In the native configuration ports and env configuration are separately definable which in this simple format is not the case (the same is both for building and running images).

Maybe you could give it a try ? You can find the code (along with the example) in branch 42-external-config. The example is in samples/data-jolokia-demo and can be switched on with the profile props.

[jstrachan]

Oh that looks awesome!!! :) Many thanks!

Providing a configurable prefix sounds like a nice way to do things; then folks can share properties; or make them unique between properties if required etc.

I agree with your other comments too. It does seem maven properties are a nice compositional way to configure things. What I don't like about the external configuration file thing is; if you're going to copy/paste a blob of properties/xml/json from project to project, you may as well just copy the XML inside the plugin XML's <configuration> ;). Having an external config file just seems to make things more brittle and in many cases doesn't have any reuse ability. (The difference is for things like assembly.xml files which are often just parameterised with the current project artifact/version).

i.e. the thing I like about maven properties is that if we have a project which defines, say, a bunch of different images which all share the same base image and env vars; we can keep things DRY and put those into a parent pom.xml; then just override/add new properties when required - rather than copying/pasting a config file into every project.

Another idea we had (Jimmi Dyson) was to have an uber mvn plugin that invoked all the other plugins directly passing in the configuration; though that does seem quite brittle and we need to keep re-releasing the uber plugin as any other plugin changes. So I'm liking the loose coupling approach via maven properties so far - seems the least sucky way to do it at least ;).

I wish also there was a way to define a BOM like thing for maven plugins; so we could have standard configurations for things like the docker-maven-plugin / fabric8-maven-plugin / jube-maven-plugin. So far the closest I've seen in this direction is maven tiles but I've not had time to play with it yet.

[rhuss]

I agreed that we might reuse in the configuration could be better, and I also wished there would be better (or any ;-) support for composition of poms (even support for standard XML includes would be a step in the right direction).

As mentioned above on the roadmap is support for Fig configuration and the forthcoming docker composition files. Maybe that could be a common base for our configuration needs ? (though not sure whether this syntax supports all features required).

I just released 0.10.5 with the property support as discussed (with some minor changes in the port mapping syntax since the last checking), so I'm going to close this issue for now. Feel free to reopen in case.