Investigate plugin model for update handler

[ryanbreen]

We now have support for treating Java properties files as a special case, but I'm generally uneasy with this: I would rather keep g2c thin and general purpose. We should investigate moving such strategies into a plugins layer.

[skyscooby]

I think it oversteps the bounds of the tool entirely.. if you want your keys and values a certain way in consul.. translate them onto your filesystem in a kv structure before committing them to your repository and let git2consul do it's job well..... for example write your own script/tool to transform whatever files you have.. properties / yaml / json.. into a filesystem structure that matches how git2consul translates files to keys.. perhaps release these in the utils toolbox.. making git2consul a huge translation engine doesn't seem to follow the keep it simple pattern..

[timstilwell]

there are many registry solutions available.
building the most elegant doesn't matter all that much if it has a short shelf life because another solution is determined to be more satisfactory.

if containers are expected to read config files hidden from modification then change injection must come through consul.

you're holding up implementation in the marketplace and the market window is closing. don't take too long to decide.

[ryanbreen]

Whether we implement this is up to @cleung2010 and @maclennann. My personal feeling is that we're running a real risk of encumbering git2consul with too many exceptional cases (special case handling for different config file types, for example). There seems to be a demand for this use case, but I would prefer to remove it from the core codebase.

[calvn]

git2consul's initial intention was for uploading entire files to the KV so that configuration files can be more easily propagated around consul-registered machines. However, as this project gained some traction, it has become clear that the community is using it for much more than that. We do not want to divert too far away from git2consul's use case, but we believe that git2consul should provide more flexibility than just placing entire files to the KV and we have showed some backing towards this direction by adding JSON and java.properties support.

After some discussion, we've decided to go with a plugin model to accommodate for various cases/formats that git2consul might get to accept.

The initial change will be to move existing supported formats (i.e. JSON and java.properties) to said plugin model, which then can be used as examples for future plugin additions.

[skyscooby]

How will the plugin interface work? Will the plugins be registered based on file extension type? Will they be limited to one handler per extension? Perhaps a stack of filters (like rack) and allow the plugin to decide if it wants to act on a particular file?

What will drive the configuration of the individual plugins? One source repository may expect a certain json behavior while another expects a different one.. Will this be module json-1 vs json-2 specified in the main configuration or will the user be expected to run two separate instances of git2consul with different versions of the json plugin.. and also certain files should be able to be flagged within the repository to actually be treated as files vs 'converted' to k/v's.. (ie one consuming application may actually want that json file as it was in the repository.. the possibilities are endless.. here be dragons.

The definition of the git2consul data model that is used internally could be the mediator.. and have two types of plugins.. ingest plugins and publish plugins.

Ingest plugins like json, java properties, yaml etc etc.. are not overly configurable but simply convert source file formats into the internal git2consul model. Perhaps there is a standard 'super-extension' as a way to bypass the translation on certain files...for example: myapp.json.verbatim or something

Publish plugins can tailor data to accommodate the end consumer of the values.. (such as Spring Cloud, Ribbon, etc). Maybe you can publish the same source data many times under different root paths using the various publish plugins..?

Just food for thought.. very interested to see what you come up with.

[calvn]

Those are great questions, thanks for bringing them up! I can't immediately provide an answer to all of them, but here are a few that I thought I could address right now:

One source repository may expect a certain json behavior while another expects a different one

If we move into a plugin model, the way that git2consul will determine how to parse/push the data to the KV will be repo-based. That is, the configuration for that particular repo will be declared inside its repo object in the repos array.

.. and also certain files should be able to be flagged within the repository to actually be treated as files vs 'converted' to k/v's

I am not sure if we want to go down this rabbit hole, since, as you mentioned, this can end up in endless configuration possibilities.

Another point to consider is whether we should have a particular repository's configuration live in the git2consul configuration, or in the repo itself (something to the extent of app.json) which gets read and loaded to git2consul after the initial pull.