[BUG] Can't get custom grains sync at provisioning (via salt-cloud)

[cavepopo]

Description
When provisioning windows VM I'm trying to get a custom grains sync'ed somehow but cannot get it to work.

Setup
I've been trying to get using :

• salt-cloud config file, hitting this
• orchestration state + reactor :

# Reactor entry :
- 'salt/cloud/*/deploy_windows':
    - '/srv/reactors/windows_first_start.sls'

# in /srv/reactors/windows_first_start.sls : 
sync_grains:
  local.saltutil.sync_grains:
    - tgt: {{ data['name'] }}

windows_orch:
  runner.state.orchestrate:
    - mods: orch.windows
    - pillar:
        vm_name: {{ data.name }}

# Orchestration states, in orch.windows : 
{% set vm_name = salt.pillar.get('vm_name') %}

wait_waht_for:
  cmd.run:
    - name: sleep 20
 
##### Sync_all to make custom modules available
#####+ https://github.com/saltstack/salt/issues/54452
#
load_my_custom_module:
  module.run:
    - name: saltutil.sync_all
    - refresh: True

orchestrate_windows_first_start:
  salt.state:
    - tgt: '{{ vm_name }}'
    - sls:
      - common.windows

Please be as specific as possible and give set-up details.

• VM as QEMU hosted on proxmox

Steps to Reproduce the behavior
provision windows QEMU on proxmox host and try to get custom grains available after provisioning

Expected behavior
Getting custom grains available when running orchestration states straight after the provisioning

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

######### Master's version :

Salt Version:
          Salt: 3004.1
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: Not Installed
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: 2.0.5
     gitpython: 2.1.11
        Jinja2: 2.10
       libgit2: 1.1.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.5.6
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.20
      pycrypto: 2.6.1
  pycryptodome: 3.6.1
        pygit2: 1.6.1
        Python: 3.7.3 (default, Jan 22 2021, 20:04:44)
  python-gnupg: Not Installed
        PyYAML: 3.13
         PyZMQ: 17.1.2
         smmap: 2.0.5
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.1
 
System Versions:
          dist: debian 10 buster
        locale: utf-8
       machine: x86_64
       release: 5.4.162-1-pve
        system: Linux
       version: Debian GNU/Linux 10 buster

######### Minion's version :

Salt Version:
          Salt: 3004.1

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.7
     gitpython: Not Installed
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: 1.1.4
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.20
      pycrypto: Not Installed
  pycryptodome: 3.10.1
        pygit2: Not Installed
        Python: 3.8.8 (tags/v3.8.8:024d805, Feb 19 2021, 13:08:11) [MSC v.1928 32 bit (Intel)]
  python-gnupg: 0.4.7
        PyYAML: 5.4.1
         PyZMQ: 19.0.0
         smmap: 4.0.0
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.2

System Versions:
          dist:
        locale: cp1252
       machine: AMD64
       release: 2021
        system: Windows
       version: 2021 10.0.19041 SP0 Multiprocessor Free

Additional context
Logs from the master while provisioning, I can see the sync_all module is ran but nothing is found to be sync'ed :

                    "__id__": "load_my_custom_module",
                    "__run_num__": 1,
                    "__sls__": "orch.windows",
                    "changes": {
                        "ret": {
                            "beacons": [],
                            "clouds": [],
                            "engines": [],
                            "executors": [],
                            "grains": [],
                            "log_handlers": [],
                            "matchers": [],
                            "modules": [],
                            "output": [],
                            "pillar": [],
                            "proxymodules": [],
                            "renderers": [],
                            "returners": [],
                            "sdb": [],
                            "serializers": [],
                            "states": [],
                            "thorium": [],
                            "utils": []
                        }
                    },
                    "comment": "Module function saltutil.sync_all executed",
                    "duration": 1732.087,
                    "name": "saltutil.sync_all",
                    "result": true,
                    "start_time": "07:45:24.661448"
                },

Custom grains is available to the master :

root@salt-master-01:~# salt-run fileserver.file_list  
- _grains/presentusersgrain.py
[...]

Right after the provisioning I can see there is no present_users grains using grains.ls ...

And I can even sync after the vm is actually provisioned :

root@salt-master-01:~# salt gitlab-runner-win-01  saltutil.sync_grains
gitlab-runner-win-01:
    - grains.presentusersgrain

And see it then :

root@salt-master-01:~# salt gitlab-runner-win-01 grains.ls
gitlab-runner-win-01:
    - biosversion
    - cpu_model
    - cpuarch
    - cwd
    - disks
    - domain
    - efi
    - efi-secure-boot
    - fqdn
    - fqdn_ip4
    - fqdn_ip6
    - fqdns
    - gpus
    - groupname
    - host
    - hwaddr_interfaces
    - id
    - init
    - ip4_interfaces
    - ip6_interfaces
    - ip_interfaces
    - ipv4
    - ipv6
    - kernel
    - kernelrelease
    - kernelversion
    - locale_info
    - localhost
    - manufacturer
    - master
    - mem_total
    - motherboard
    - nodename
    - num_cpus
    - num_gpus
    - os
    - os_family
    - osfinger
    - osfullname
    - osmanufacturer
    - osrelease
    - osrelease_info
    - osservicepack
    - osversion
    - path
    - pending_reboot
    - pid
    - present_users
    - productname
    - ps
[...]

[nicholasmhughes]

It looks like your sync_all is being run at the orchestration level, so it's not targeting vm_name.

Does something like this work?

load_my_custom_module:
  salt.function:
    - tgt: '{{ vm_name }}'
    - name: saltutil.sync_all
    - kwarg:
        refresh: True

[cavepopo]

@nicholasmhughes your are right, it's been too long for me, I lost my salt foo, your solution fix it, thanks !

I keep that one open as this is only a workaround, the main goal here is to get the cloud config option to work as expected.

[nicholasmhughes]

Looking at the code and the documentation, it seems like sync_after_install is not meant to be a boolean... but instead a word representing the completion of a saltutil.sync_ module function showing what you want to sync (all, beacons, etc.)

Can you try setting sync_after_install: all in your cloud config?

If that works, the follow-ups I see are:

• a comprehensive list of saltutil.sync_ functions should be added to the code and documentation
• a non-falsey value that doesn't match the list should probably just assume all

[nicholasmhughes]

@cavepopo , can you take a look at the previous comment and let me know if all as the value fixes the behavior?

[nicholasmhughes]

@cavepopo , have you been able to test the suggestion above?

[cavepopo]

@cavepopo , have you been able to test the suggestion above?

Not yet, I am still using the workaround, but I think I already tried the "all" fix but I'll try tomorrow to be sure and to give a correct feedback

[cavepopo]

Hi @nicholasmhughes ,

Unfortunately this (sync_after_install: all) does not work, here is some info from the event bus :

[INFO    ] salt-cloud starting
[INFO    ] Applying map from '/etc/salt/gitlab-runner.map'.
[INFO    ] Calculating dependencies for gitlab-runner
[INFO    ] Creating Cloud VM gitlab-runner
[INFO    ] Configuring cloned VM
[INFO    ] Rendering deploy script: /usr/lib/python3/dist-packages/salt/cloud/deploy/bootstrap-salt.sh
[INFO    ] Connecting to SMB Tree \\<ip>\ADMIN$
[INFO    ] Creating open to PAExec file with delete on close flags
[INFO    ] Disconnecting from SMB Tree \\<ip>\ADMIN$
[INFO    ] Get connection smbprotocol
[INFO    ] Connecting to SMB Tree \\<ip>\ADMIN$
[INFO    ] Creating open to PAExec file with delete on close flags
[INFO    ] Disconnecting from SMB Tree \\<ip>\ADMIN$
[INFO    ] Connecting to SMB Tree \\<ip>\ADMIN$
[INFO    ] Creating open to PAExec file with delete on close flags
[INFO    ] Disconnecting from SMB Tree \\<ip>\ADMIN$
[INFO    ] Connecting to SMB Tree \\<ip>\ADMIN$
[INFO    ] Creating open to PAExec file with delete on close flags
[INFO    ] Disconnecting from SMB Tree \\<ip>\ADMIN$
[INFO    ] Salt installed on gitlab-runner
[INFO    ] Created Cloud VM 'gitlab-runner'
[ERROR   ] There was a query error: 'ImmutableDict' object has no attribute 'update'

[nicholasmhughes]

@cavepopo , I think PR #62056 might fix the issue. It looks like the code was attempting to update DEFAULT_MASTER_OPTS directly... which is an ImmutableDict. If you'd like to try to patch it in place and test, please let me know the results.