ansible-config

[bcoca]

Proposal: ansible-config

Author: Brian Coca @bcoca

Date: 2016/10/-1

• Status: New
• Proposal type: core design
• Targeted Release: future release
• Estimated time to implement: 3 weeks

Draft implementation: ansible/ansible#12797

Motivation

The current configuration system is limited to a single file and previously hardcoded entries in constants.py. We want to allow plays, plugins and even roles to add config settings when they need them.

Problems

• plugins don't have standardized config entries and adding new/custom plugins requires hardcoding many settings
• Configs are not mergable, sometimes you want to be able to just override a couple of entries and not the whole thing.
• Role that require certain configuration settings are not easily shareable
• There is no current way to 'dump' the config and see the current settings
• Lack of documentation, many config options are left out of the docs and even if added not every way to set an option is detailed.

Solution proposal

• New config.yml that allows for structured config in the familiar YAML format
• New ansible-config command line to allow:
  • dump of current config settings
  • update/write a config entry
  • show available options (ini entry, yaml, env var, etc)
• Definition data for configs also in YAML format, instead of hardcoding into config.py, easy to extend by adding new plugin.yml file as 'config definition'.
• new config setting 'mergable' to allow the current config file to merge with the next priority target one
• new meta/ dir adjacent to play that can host a config.yml specific

Other options

• new meta/config.yml for roles, same purpose as play one
• new setting to toggle reading role's config.yml read_config_from_role = no|yes
• meta/<plugin_type>/<plugin_name>.yml to add config data definition file (config would be in preexisting config.yml)

Config Data definition

• Internal name of setting
• ini entry name
• yaml var name
• environment var
• default value
• inventory var (if applicable)
• description
• type

Example config file:

ansible_cow_selection: random
deprecation_warnings: false
plugins:
  callbacks:
    mail:
       smpt_host: smtp.example.com

Example config definition file:

main:
  ANSIBLE_COW_SELECTION:
      desc: Allows choosing specific cowsay stencils, use 'random' to cycle through them
      ini:
        section: defaults
        key: cow_selection
      env: ANSIBLE_COW_SELECTION
      type: string
      default: null
   DEPRECATION_WARNINGS:
      desc: Controls the display of deprecation warnings.
      ini:
         section: defaults
         key: deprecation_warnings
      yaml:  deprecation_warnings
      env: ANSIBLE_DEPRECATION_WARNINGS
      default:  True
      type: boolean

...

plugins:
  callbacks:
      mail:
        SMTP_HOST:
           desc: Host to use send email via SMTP.
           yaml: smtp_host
           env: SMTPHOST
           default: localhost

[jhawkesworth]

This would be great, but I'm curious how it would interoperate with vault.
Would it be possible to vault some, but not all of the vars? It would be good to be able to just vault say passwords for some external service without hiding the hostname.

[kustodian]

I really like the idea and the flexibility it gives you, the only thing which I'm not sure I'm the fan of is using YAML, because from your example this ini format:

# set which cowsay stencil you'd like to use by default. When set to 'random',
# a random stencil will be selected for each task. The selection will be filtered
# against the `cow_whitelist` option below.
#cow_selection = default

is replaced with:

  ANSIBLE_COW_SELECTION:
      desc: Allows choosing specific cowsay stencils, use 'random' to cycle through them
      ini:
        section: defaults
        key: cow_selection
      env: ANSIBLE_COW_SELECTION
      type: string
      default: null

The ini file is far more easier to read/edit and understand. Of course YAML gives you additional info, but is all of it important except the environment variable which can be used, but you can put the variable in the comment of the ini file as well.

This YAML is awesome when you want to add some new option (as you added smtp_host), but for overwriting it isn't the best one, would be cool if for overwriting a value you could just write:

main:
  defaults:
    cow_selection: 'defaults'

The reason why I'm saying this, is that most often you would like to override some ini value for a project/role/play, but the above option feels too complicated for that.

[bcoca]

@jhawkesworth that is already been added in 2.2

@kustodian that exmaple is the 'data definition' not the config file, the config file looks like this:

cow_selection: random

[kustodian]

@bcoca that config file looks great :)

[hyperized]

@bcoca YAML formatting, always +1. This proposal allows for proper meta data (not loosely filled comment fields) and being declarative for everyone to read and understand in the whole otherwise YAML oriented Ansible setup.

[willthames]

How about runtime configuration too?

ansible-playbook --config callback_whitelist=oneoff_callback playbook.yml

or

ansible-playbook --config @meta/callback-override.yml playbook.yml

As an alternative to:

ANSIBLE_CALLBACK_WHITELIST=oneoff_callback ansible-playbook playbook.yml

[bcoca]

@willthames we currently have an ANSIBLE_CONFIG option, which with the proposed 'mergable' toggles would act the same. I'm not against adding a --config to do same.

[gundalow]

Would we be able to auto generate a new page on docs.ansible.com from this?
I wonder if in RST there is a way to mark something as an option, so any references to (say) DEPRECATION_WARNINGS would link to the relevant section in the new docs page

[bcoca]

@gundalow yes, all that is part of the plan .. just not sure when it will be ready

[dmsimard]

@bcoca just making sure... Am I mistaken or it looks like the plugin portion of the spec hasn't been implemented yet ? For example there is the notion of plugins in ConfigData but seemingly nothing that loads actual plugins and their configuration into _plugins.

I ask because I might want to get involved into this for ARA.

[bcoca]

it is a work in progress, I expect most features to be done in time for 2.4, but this project will probably extend into 2.5

[mrWinston]

Is there any update on this issue? It didn't ship with 2.6 ( as far as I'm concerned ). Is it still on the roadmap?

We're currently facing an extreme amount of config duplication across our different ansible projects which would be elegantly solved by outsourcing most of the settings into a common ansible.cfg and only overwriting those that are needed.

[bcoca]

it is waiting for other projects to be finalized as they can influence how this would take final shape, so I am still planing on finishing all the features noted above, but this is a 'multi release' project.

[davividal]

@bcoca how can I be involved in this? I would like to help.

[kladiv]

+1

[lcnittl]

I guess there is still no news on this? :( (Or am I missing something? Current docs, 2.9, states it is still the "first file behavior")

[rguhr]

Maybe this simple shell workaround can help some people.
It makes it possible to use multiple configuration files in a limited way.

https://gitlab.com/snippets/1851171

[AlanCoding]

We can close ansible/ansible#61778 in favor of this. I believe it's covered by the agenda here.

[bcoca]

Adding plugin support ansible/ansible#49627

[tony-sol]

Is it still impossible to read and use multiple config files?

E.g. - i wanna use ANSIBLE_CONFIG="{$XDG_CONFIG_HOME}/ansible defined globally (just to keep my HOME clean) AND ansible.cfg in cwd.

Or maybe is there a way to also read ${ANSIBLE_HOME}/ansible.cfg file, like fallback if $PWD/ansible.cfg doesn't exist?

[minsis]

I'm looking to get more involved in the ansible community and I'd like to take a crack at parts of this, specifically config precedence with the standard ansible.cfg files:

• ANSIBLE_* - environment variables
• ansible.cfg - in playbook directory
• ~/.ansible.cfg - in user's home directory
• /etc/ansible/ansible.cfg - on system's filesystem

[bcoca]

@minsis i'm not sure what you want to change, but the existing precedence is already as you list. My proposal would not affect that, it would only affect .yml configuration files as we also require a indicator for the setting to be mergable/nonmergable to allow for mandated/incompatible settings detection.

[minsis]

@bcoca To clarify I mean the merging of the config files.

[bcoca]

that is what i suspected and that is why my answer also includes why the current 'ini-ish' .cfg files are not a target for that.

[dberardo-com]

Is this feature of including multiple .cfg files inside the main one still not available?

[minsis]

@bcoca why would it only be limited to yaml files only and not support backward compatibility? Inventory files does this.

Also is there any ongoing discussions on which variables would be protected ones?

[bcoca]

@minsis because in yaml i can do things like !mergable or !fixed , there is no facility to do such things in ini formats

[minsis]

@bcoca i guess I'm not quite understanding the feature you're talking about. Can you link some docs on what you mean?

Overall I still think it's a mistake not to support backwards compatibility. On day one of removing ini files is going to break a lot of people's automation including my own. We deploy a standard cfg file to servers to point to our private automation hub.

If the overall goal is to make everything yaml then maybe a depreciation warning for a few versions to allow people time to migrate.

[bcoca]

Where is it mentioned that the ini files are going away? And if they were going to be removed we already have a 4 version deprecation cycle so that would give people plenty of time.

The reason the merge would only support YAML has to do with different security contexts. For example, in a context where you have strict regulations/policy on what is permitted the /etc/ansible/config.yml might have a few things you cannot override due to security/compliance/policy

setting_not_to_be_overriden: !nomerge thevalue

while others are permitted to be merged

random_thing: !overridable "value that can be overriden by other config file"

Also having a generic setting for the whole file:

merge_policy:  forbidden/authoritative/overridable

It is not possible to have this type of fine grained control with the ini format.

This would allow the Administrator to set policy over settings and not have the playbook author/user bypass anything that is globally set due to security/regulation/policy.