Trying to reference existing pillars in new pillars (both first-level and subdirectories/init.sls) leads to unexpected behaviour

[johnnybubonic]

Description of Issue/Question

The documentation is not clear, and there are numerous issues on this repo about this.

What is the canonical, elegant way to access a pillar from a pillar?

I have an ext_pillar set up, and I have ext_pillar_first set to True in my master config.

I then need to further assign pillars based on what the pillars that are returned from ext_pillars are.

I've found that I can do this in a completely hacky and not at all proper way via opts['pillar']['ext_pillar_name'] (and such as a result from that dict) - in the pillar_roots top file with no issue.

But I need to consistently be able to access that same data in recursed pillars (e.g. accessing ext_pillar from <pillar_root>/something/init.sls, or even <pillar_root>/not_top.sls which is called from <pillar_root>/top.sls). When I try, it's a gamble if it works (presumably a cache? Though I have pillar cache disabled) or if I get this:

jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'pillar'

In full:

Traceback (most recent call last):
  File "/usr/lib/python3.4/site-packages/salt/utils/templates.py", line 393, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3.4/site-packages/jinja2/environment.py", line 989, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3.4/site-packages/jinja2/environment.py", line 754, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3.4/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 1, in top-level template code
  File "/usr/lib/python3.4/site-packages/jinja2/environment.py", line 389, in getitem
    return obj[argument]
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'pillar'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.4/site-packages/salt/utils/templates.py", line 170, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/python3.4/site-packages/salt/utils/templates.py", line 403, in render_jinja_tmpl
    buf=tmplstr)
salt.exceptions.SaltRenderError: Jinja variable 'dict object' has no attribute 'pillar'

Please, please, please, for the love of my sanity - why can't I reliably access existing pillars in new pillars?

pillars.get('some:key') does not work. salt['pillars.get']('some:key') does not work.

The only thing that DOES work (albeit half the time) is the opts[''] hack. Grains, as expected, work fine but you can't enforce those from the master.

Salt Version:
           Salt: 2018.3.3
 
Dependency Versions:
           cffi: 1.11.5
       cherrypy: unknown
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: 0.26.8
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.17
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.26.4
         Python: 3.4.9 (default, Aug 14 2018, 21:28:57)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.2
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7.6.1810 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-957.1.3.el7.x86_64
         system: Linux
        version: CentOS Linux 7.6.1810 Core

[Ch3LL]

looks like there was this issue: #6955

which was closed with the inclusion of saltclass which solved this problem.

another issue here: #4244 (comment) is a workaround to include other pillar data..

I think you are right though we need to define this more within the docs.

ping @saltstack/team-core it seems based off the issues I read, referencing pillar in other pillar files is not entirely supported without using workarounds. Is this true on your end as well? if so I think we should document this with the workarounds, until we can add this capability.

[johnnybubonic]

Thank you SO much. I was going crazy, here.

I'll check out saltclass! I briefly was reading about it but couldn't really use it because my data source is in MySQL stored in formats that aren't exactly Salt-friendly (e.g. UUIDs are in BINARY(16), IPv4/IPv6 addresses are in VARBINARY(4) and VARBINARY(16) respectively, etc. - so I can't directly use the mysql module because there are necessary transformations). While I could probably do this with a "py" template, an ext_pillar seemed more appropriate/"clean".

I technically don't even need to inherit pillars from parent pillars, though it'd be nice - I was just operating under the assumption that because my ext_pillar_first is True, that they'd be available to later pillar_roots but it would seem this ONLY affects merge strategy. Is this correct?

Pardon my insanity last night; to clarify the specific use case, there are some pillars that will be unique to every minion, but others that are shared for every minion (or most minions, with an overridden exception here and there) but you want to be able to enforce them from/keep private to the master. Roles and role-based strategy is an example of this; you don't want to trust a role to a grain because a machine shouldn't be able to decide or change its role, the saltmaster should. This is especially true of organizations with a global presence where some of the machines in remote locations may be liable to... well, political motivations outside the organization's influence.

An alternative, at least in MY use case, would be to have a new type of fact/object, master-enforced grains ("rocks", I guess they could be called?) that would act as grains scope-wise (e.g. available to pillars) but would be enforced by the master and could not be changed by a minion. That way it's an ideal compromise between the policy enforcement of pillars while still being able to reference them in pillars - but this largely seems a redundancy of pillars, so the ROI of implementing such a new type would probably not be worth it. Pillars-in-pillars would definitely be beneficial to a very large number of users, though, I'd wager.

Thanks again for looking into this! Sorry for the novel! I wanted to clarify my (admittedly exhausted) initial post on my reasoning behind this.

[johnnybubonic]

Also throwing here for reference since it's where I got the opts['pillar'] thing from in the first place and it took me forever to find it again: #37905 (comment) .

[max-arnold]

@johnnybubonic I'd avoid saltclass unless you want to deal with the open issues. The person who contributed it (@olivier-mauras) does not respond to mentions and has no activity on Github since June 2018 (life happens).

If you want to go this route, it is worth looking at reclass (Salt ships with builtin reclass adapters for both state and pillar trees). However, this pillar-heavy approach will lead to master performance problems at a certain scale (personally, I used it up to 80-100 nodes, folks at Mirantis reported that it works just fine for hundreds of nodes, and the real limit is unknown).

Also, there are other ext_pillar backends that try to solve the pillar inheritance problem in different ways (one example is varstack_pillar).

Generally speaking, you need to consider the following factors:

1. The scale of your infrastructure
2. The number of unique classes of server configurations (are they uniform, or there are lots of snowflakes?)
3. How dynamic is your infrastructure (disposable nodes, autoscaling, etc.)?
4. Security

For security, you would want to avoid assigning secrets to nodes based on untrusted data (anything that comes from a minion: grains, fileserver cache, etc. - the only trusted thing is grains.id). So these are best kept in a pillar.

For a highly dynamic infrastructure, reclass/saltclass is a bad fit because you need to create one file per node (although reclass has a workaround).

If the scale is small and you have many snowflakes, then reclass is the right fit. My advice would be to avoid being too DRY at first (it is very hard to design the right data model, and it is quite likely that you'll end up with highly coupled states). Expect to refactor your state tree once or twice.

For high scale (thousands of nodes), the common advice is to minimize pillar usage (use it only to for sensitive data, or retrieve your secrets using the Vault sdb backend). All other values should be stored in json (or yaml) files in the salt:// file server and processed on minions using Jinja or custom modules.

Hope this helps.

[johnnybubonic]

helps a bit, yeah; part of my issue is trying to maintain DRY while also wondering how the thing's going to scale - not by model, as i'm intentionally trying to design that in, but the resource usage for master(s). i just mostly wish there were a way to dynamically and conditionally assign pillars based on other pillars in the top file. i've gotten around it for now by implementing pillar matching in the pillar top, and that works well enough:

{%- set saltpath = opts.pillar_roots[saltenv] -%}
base:
  'ext_pillar_name:type:foo':
    {%- set invtype = 'foo' %}
    - match: pillar
    - foo
  'ext_pillar:type:bar':
    {%- set invtype = 'bar' %}
    - match: pillar
    - bar
{%- for curpath in saltpath %}
{%- if salt['file.file_exists']('{0}/{1}/{2}.sls'.format(curpath, invtype, grains.id)) %}
  '{{ grains.id }}':
    - {{ invtype }}.{{ grains.id }}
{%- endif %}
{%- endfor %}

and this lets me get done what i need to get done, but it... feels hacky. it was the most reliable way i could get it working, though.

[johnnybubonic]

on that note, was ignore_missing removed from pillars at some point? i keep seeing mentions of it but i didn't find it in the documentation.

[max-arnold]

It was recently documented and is available in develop docs: https://docs.saltstack.com/en/develop/topics/pillar/

See #46833

[johnnybubonic]

OH. it's in a FUTURE release! that explains it. thanks! that'll simplify some things!

[siliconsheep]

I seem to have the same problem still / again on 2019.2. ext_pillar_first is set to True but targetting using the external pillar's data in the pillar top file doesn't work:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/master.py", line 1838, in run_func
    ret = getattr(self, func)(load)
  File "/usr/lib/python2.7/dist-packages/salt/master.py", line 1543, in _pillar
    data = pillar.compile_pillar()
  File "/usr/lib/python2.7/dist-packages/salt/pillar/__init__.py", line 1005, in compile_pillar
    matches = self.top_matches(top)
  File "/usr/lib/python2.7/dist-packages/salt/pillar/__init__.py", line 675, in top_matches
    self.opts.get('nodegroups', {}),
  File "/usr/lib/python2.7/dist-packages/salt/matchers/confirm_top.py", line 36, in confirm_top
    return m(match)
  File "/usr/lib/python2.7/dist-packages/salt/matchers/pillar_match.py", line 24, in match
    __opts__['pillar'], tgt, delimiter=delimiter
KeyError: u'pillar'

@johnnybubonic 's last comment suggests this somehow should work, but even deleting minion cache doesn't seem to solve it

[johnnybubonic]

@siliconsheep i think it's still only in the develop branch, i don't think it's in a release yet

[siliconsheep]

Oh, I assumed by the looks of #51403 you managed to find a way to do exactly the thing I'm trying to accomplish or were you using the develop branch then as well?

Currently, I'm using pillarstack as a workaround, but would be nice to know if the original plan will work in the future as well :)

[johnnybubonic]

@siliconsheep sorry for the delay - no, i had to completely redesign my approach; it's currently significantly less efficient/performant as a result, but it works at least. i never was able to get the above working reliably.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[johnnybubonic]

keep open; we still want pillar referencing within pillars. :)

[stale]

Thank you for updating this issue. It is no longer marked as stale.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[johnnybubonic]

stalebot, i JUST marked this as unstale yesterday. wat r u doin

[stale]

Thank you for updating this issue. It is no longer marked as stale.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[johnnybubonic]

still valid

[stale]

Thank you for updating this issue. It is no longer marked as stale.

[Armadill0]

Funny I stumbled upon this when a friend of mine tried ext_pillar_first: True the first time today.

We found another "workaround" for this if you just use the pillar dictionary: {{ pillar['key']['subkey'] }} I was wondering why nobody else mentioned this here and in the other referenced issues I've read and thought that might help at least some of you.

I was using this for years very successfully with highly complex pillar data (from a custom Netbox module and gitstack) because pillar and grains dictionaries are mentioned in a lot of Salt docs. But the "bug" or missing featue didn't hit me because I always ran ext_pillar after pillar_roots. 🙂

[dampersand]

You guys are the real heroes. Add my name to the list of people that want something similar.

https://www.reddit.com/r/saltstack/comments/k0h0tr/using_pillars_to_assign_pillars_in_the_pillar/

[dgengtek]

This is such a weird behavior. Is this a bug or feature?

in pillar_roots

{{opts['pillar']['key']}} works

{{pillar['key']}} works

{{pillar.get('key')}} does not work

{{salt.pillar.get('key')}} does not work

using a custom execution module which internally tries to reference the pillar via __salt__.pillar.get('key') or whichever of the forms above does not work at all

Being able to access previously set pillars from ext_pillar_first: True should work because why should it not if it is rendered before pillar_roots? I am able to grab data set by previous ext_pillars in a subsequent ext_pillar as well.

[sharkbruhaha]

bump

[tacerus]

In case of IPAM or cloud provider solutions being used as ext_pillar's, it would be very beneficial being able to conditionally structure a GitFS based pillar based off such data.

One idea would be a setting similar to ext_pillar_first: True but specific to a certain pillar - such as first_ext_pillar: netbox - which defines an ext_pillar to be loaded before every other pillar.

It seems solutions such as pillarstack could help with solving this outside of Salt's built in pillar merging functionality - however they are, in my opinion, not well documented. The project documentation on pillarstack mostly provides basic examples on including local pillar files, and community provided sample configurations do not hint as to how existing ext_pillars could be integrated. Does someone happen to have advice on merging GitFS and Netbox using pillarstack?