dellos10_config does not process jinja2 templates from "src"

[cultcom]

SUMMARY

I try to configure a switch with a given template and all what is sent to the switch is the literal name of the file pointed to as "src".

ISSUE TYPE

• Bug Report

COMPONENT NAME

https://github.com/ansible-collections/dellemc.os10/blame/master/plugins/modules/os10_config.py#L221

ANSIBLE VERSION

ansible [core 2.11.5] 
  config file = /home/xxx/workdir/playbooks/dell-OS10/ansible.cfg
  configured module search path = ['/home/xxx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/xxx/.local/lib/python3.8/site-packages/ansible
  ansible collection location = /home/xxx/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/xxx/.local/bin/ansible
  python version = 3.8.10 (default, Jun  2 2021, 10:49:15) [GCC 9.4.0]
  jinja version = 2.10.1
  libyaml = True
(master) xxx@vewok ~/workdir/playbooks/dell-OS10 $ pip freeze | grep ansible
ansible==4.5.0
ansible-core==2.11.5
ansible-pylibssh==0.2.0

COLLECTION VERSION

dellemc.os10      1.1.1                                   

CONFIGURATION

ANSIBLE_NOCOWS(/home/xxx/workdir/playbooks/dell-OS10/ansible.cfg) = True
INTERPRETER_PYTHON(/home/xxx/workdir/playbooks/dell-OS10/ansible.cfg) = /usr/bin/python3

OS / ENVIRONMENT

Controller:
No LSB modules are available.
Distributor ID: Linuxmint
Description: Linux Mint 20.2
Release: 20.2
Codename: uma

Switch:
Dell EMC Networking OS10 Enterprise
Copyright (c) 1999-2021 by Dell Inc. All Rights Reserved.
OS Version: 10.5.2.7
Build Version: 10.5.2.7.374
Build Time: 2021-07-28T04:48:16+0000
System Type: S5248F-ON
Architecture: x86_64
Up Time: 4 days 02:00:54

STEPS TO REPRODUCE

---
- hosts:
    - spine
    - leaf
  gather_facts: False

  tasks:
    - name: "Provision basic settings"
      dellemc.os10.os10_config:
        src: templates/00-hrz-basics.j2

ansible-playbook -i inventory/core playbooks/basic.yml -l core-leaf-c-02

PLAY [spine,leaf] *********************************************************************************************************************

TASK [Provision basic settings] *******************************************************************************************************

fatal: [core-leaf-c-02]: FAILED! => {"changed": false, "command": "templates/00-hrz-basics.j2", "msg": "templates/00-hrz-basics.j2\r\n% Error: Unrecognized command.\r\n\u0007core-leaf-c-02(config)# ", "rc": -32603}

PLAY RECAP ****************************************************************************************************************************
core-leaf-c-02             : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

EXPECTED RESULTS

I expect the jinja2 template to be evaluated and sent as commands to the switch.

ACTUAL RESULTS

The literal filename configured as "src" is sent as a command to the switch.

[lpereira1]

Im having the same issue here

[zordanda]

same issue, was working with the old ansible.builtin.dellos10

[javeedf]

Could you please check if this helps?
Run your playbook as below,

ANSIBLE_NETWORK_GROUP_MODULES=os10 ansible-playbook -i inventory/core playbooks/basic.yml -l core-leaf-c-02

between, is the roles working fine?

[cultcom]

No, sorry, I get the same error.

[zordanda]

same result for me.

fatal: [PLAT-SW-1]: FAILED! => {"changed": false, "command": "templates/plat_system.j2", "msg": "plat_system.j2\r\n% Error: Unrecognized command.\r\n\u0007PLAT-SW-1(config)# ", "rc": -32603}

roles (dellemc.os10.os10_system in this case) work fine though.

[javeedf]

@zordanda Could you please paste your directory structure?

[zordanda]

just some additional info, i have a role that extends the functionalities of dellemc.os10.os10_system.
i put this new role under ~/.ansible/roles/mysystemrole

this is the structure of the role dir:

├── defaults
│   └── main.yml
├── handlers
│   └── main.yml
├── LICENSE
├── README.md
├── tasks
│   └── main.yml
├── templates
│   └── plat_system.j2
└── vars
    └── main.yml

the directory where the playbook is located contains

├── 1_L3Switch_conf_playbook.yaml
├── ansible.cfg
├── group_vars
│   └── L3Switch.yaml
├── hosts.yaml
├── host_vars
│   ├── PLAT-SW-1.yaml
│   ├── PLAT-SW-2.yaml
├── secrets.yaml

[javeedf]

@zordanda if you are creating a new role, then your playbook should look like this,

---
- hosts: L3Switch
  gather_facts: False
  connection: network_cli
  collections:
    - dellemc.os10  
  roles:
    - mysystemrole

Roles: mysystemrole/tasks/main.yaml file should look as below, no need to mention, templates/plat_system.j2

 - name: "Provisioning plat Systems config"
   dellemc.os10.os10_config:
      src: plat_system.j2
   register: output

if this doesn't work, if possible please share your code repo

[zordanda]

That exactly how my playbook looks like. And I removed the "templates/" part of the path from src in mysystemrole/tasks/main.yaml but I still get the same error.
BTW thanks for your help.

[zordanda]

I tried copying the dellemc.os10.os10_system role from ~/.ansible/collections/ansible_collections/dellemc/os10/roles/os10_system to .ansible/roles/mysystem

and using the original role in the playbook works, while using the renamed one does not.

[javeedf]

@zordanda I think we are missing the meta/main.yaml file

[zordanda]

Same when using the os10_config module directly in the playbook, with the template in the same folder as the playbook.

playbook:

---
- hosts: L3Switch
  gather_facts: False
  connection: network_cli
  tasks:
  - name: "Provisioning system configuration for os10"
    dellemc.os10.os10_config:
      src: os10_system.j2
    register: output 

output:

fatal: [PLAT-SW-1]: FAILED! => {
    "changed": false,
    "command": "os10_system.j2",
    "invocation": {
        "module_args": {
            "after": null,
            "backup": false,
            "backup_options": null,
            "before": null,
            "config": null,
            "lines": null,
            "match": "line",
            "parents": null,
            "provider": null,
            "replace": "line",
            "save": false,
            "src": "os10_system.j2",
            "update": "merge"
        }
    },
    "msg": "os10_system.j2\r\n% Error: Unrecognized command.\r\n\u0007PLAT-SW-1(config)# ",
    "rc": -32603
}

however, it works if I include the "collections" in the playbook as follows:
playbook:

---
- hosts: L3Switch
  gather_facts: False
  connection: network_cli
  collections:
  - dellemc.os10
  tasks:
  - name: "Provisioning system configuration for os10"
    os10_config:
      src: os10_system.j2
    register: output 

output:

ok: [PLAT-SW-1] => {
    "changed": false,
    "invocation": {
        "module_args": {
            "after": null,
            "backup": false,
            "backup_options": null,
            "before": null,
            "config": null,
            "lines": null,
            "match": "line",
            "parents": null,
            "provider": null,
            "replace": "line",
            "save": false,
            "src": "hostname PLAT-SW-1\n",
            "update": "merge"
        }
    },
    "saved": false
}

[zordanda]

@zordanda I think we are missing the meta/main.yaml file

@javeedf can you please elaborate on that?

Am I missing a file or is it something missing in the dellemc.os10.os10_config module?

[javeedf]

@zordanda We need to create a directory like this. not sure whether this help or not.

https://github.com/ansible-collections/dellemc.os10/tree/master/roles/os10_system/meta

[cultcom]

I tried playing around with the different hints from this thread and can tell now that it works with the template when I set the "collections" and use the short name for the task like this:

---
- hosts: dellos10
  gather_facts: False
  connection: network_cli
  collections:
    - dellemc.os10

  tasks:

  - os10_config:
      src: vlan.j2
    tags:
      - never
      - os10_config

Using the full path module name "dellemc.os10.os10_config" for the task breaks running the jinja template.

[zordanda]

@cultcom yes, that is also what I observed.

@javeedf You are right, adding a meta/main.yaml file into the role folder with content:

---
collections:
- dellemc.os10

and then using the module with just os10_config in the task/main.yaml of the role (not the full dellemc.os10.os10_config) solved the problem.

Thank you everyone for your help!!!

[javeedf]

if we are good, shall we close this issue?

[cultcom]

Well, to me it sounds more like a workaround than a solution?

[zordanda]

This is a documented way to use collections in playbooks and roles, as I just found out.

see: https://docs.ansible.com/ansible/latest/user_guide/collections_using.html#using-collections-in-a-playbook

However, It is true that it should be possible to use the FQCN.

[Darzaguhl]

I am having the same problem using Ansible tower.

---
- name: Check logging and NTP config
  hosts: test
  gather_facts: False
  connection: network_cli
  collections:
    - dellemc.os10
    - ansible.netcommon
 roles:
   - os10_logging
   - os10_ntp

The roles are under playbooks/collections/ansible-collections/dellemc/os10/roles
The roles use a jinja2 template to generate config from variables. These roles are native to the collection.

I have been debugging all day and can not get the "src" field to be evaluated other than literally in the os10_config part of these roles.

There is another part in these roles, that is optional, that generate from the same jinja2 template using the "template" module and that works. But when it comes to os10_config it does not work.

I am running out of ideas of what this can be. Any ideas?

@javeedf I also defined collections in playbook and meta main.yaml

└── playbooks
    ├── Check-NTP-and-Logging.yaml
    └── collections
        └── ansible_collections
            └── dellemc.os10
                ├── LICENSE
                ├── README.md
                ├── changelogs
                │   ├── CHANGELOG.rst
                │   ├── changelog.yaml
                │   ├── config.yaml
                │   └── fragments
                ├── docs
                │   └── roles.rst
                ├── galaxy.yml
                ├── meta
                │   └── runtime.yml
                ├── playbooks
                │   ├── clos_fabric_ebgp
                │   ├── delltesting
                │   └── vxlan_evpn
                ├── plugins
                │   ├── action
                │   ├── cliconf
                │   ├── doc_fragments
                │   ├── module_utils
                │   ├── modules
                │   └── terminal
                ├── requirements.txt
                ├── roles
                │   ├── os10_aaa
                │   ├── os10_acl
                │   ├── os10_bfd
                │   ├── os10_bgp
                │   ├── os10_copy_config
                │   ├── os10_dns
                │   ├── os10_ecmp
                │   ├── os10_fabric_summary
                │   ├── os10_flow_monitor
                │   ├── os10_image_upgrade
                │   ├── os10_interface
                │   ├── os10_lag
                │   ├── os10_lldp
                │   ├── os10_logging
                │   ├── os10_network_validation
                │   ├── os10_ntp
                │   ├── os10_prefix_list
                │   ├── os10_qos
                │   ├── os10_raguard
                │   ├── os10_route_map
                │   ├── os10_snmp
                │   ├── os10_system
                │   ├── os10_template
                │   ├── os10_uplink
                │   ├── os10_users
                │   ├── os10_vlan
                │   ├── os10_vlt
                │   ├── os10_vrf
                │   ├── os10_vrrp
                │   ├── os10_vxlan
                │   └── os10_xstp
                └── tests
                    ├── integration
                    ├── sanity
                    └── unit

os10_logging
├── LICENSE
├── README.md
├── defaults
│   └── main.yml
├── handlers
│   └── main.yml
├── meta
│   └── main.yml
├── tasks
│   └── main.yml
├── templates
│   └── os10_logging.j2
├── tests
│   ├── inventory.yaml
│   ├── main.os10.yaml
│   └── test.yaml
└── vars
    └── main.yml

tasks/main.yml 
---
# tasks file for os10
 - name: "Generating logging configuration for os10"
   template:
      src: os10_logging.j2
      dest: "{{ build_dir }}/logging10_{{ hostname }}.conf.part"
   when: (ansible_network_os is defined and ansible_network_os == "dellemc.os10.os10") and ((os10_cfg_generate | default('False')) | bool)
#   notify: save config os10
   register: generate_output

 - name: "Provisioning logging configuration for os10"
   os10_config:
      src: os10_logging.j2
   when: (ansible_network_os is defined and ansible_network_os == "dellemc.os10.os10")
#   notify: save config os10
   register: output

[lesinigo]

Hello everyone, I've been bitten by the same problem and done some research, here are my findings for everyone that came so far as to find this issue.

As far as I can tell, it is related to this module using common networking code from ansible.netcommon.

The os10_config plugin from this collection (see plugins/modules/os10_config.py) is calling NetworkConfig().load() from ansible.netcommon passing to it the contents of src - I guess expecting that it would use that as a filename for a Jinja template to load and parse.

But as far as I can tell the current version of ansible.netcommon is just loading the contents of the .load() parameter as configuration lines, not as a filename to open (see netcommon.plugins.module_utils.network.common.config.NetworkConfig, check out .load() and .parse())

It seems to me that NetworkConfig has always been like that at least from late 2019, maybe its behavior was different in the past, maybe the dellos10 code is just making a wrong assumption or maybe I'm just missing something.

For those writing their own roles it should probably be enough to use lines or to pass the same lines as a whole text content to src - but I have not tested this. For those - like me - that are trying to use the "official" Dell-Networking roles from Dell EMC are probably out of luck.

[Darzaguhl]

Well if that were true @lesinigo then why did @zorlanda and @cultcom get it to work?
But it does make sense....
Also NetworkConfig has this:

    def loadfp(self, fp):
        with open(fp) as f:
            return self.load(f.read())

Not a python expert but this method at least load a file. Not sure how that will work with a jinja2 template.

As of now though none of the os10 config based roles work since they all seem to use a jinja2 template.

[lesinigo]

Well if that were true @lesinigo then why did @zorlanda and @cultcom get it to work? But it does make sense....

I had the same question in mind and sincerely don't know the answer... my first thought was that NetworkConfig().load() did change behavior some time in the past but after a brief look through git history it doesn't seem to be that case.

I'm actually more concerned with the future of this code rather than the past because I have to decide what solution to adopt to manage some specific aspects of a OS10 deployment (the addition and removal of VLANs from some ports on systems that are also moving production traffic).

Also NetworkConfig has this:

    def loadfp(self, fp):
        with open(fp) as f:
            return self.load(f.read())

Not a python expert but this method at least load a file. Not sure how that will work with a jinja2 template.
As of now though none of the os10 config based roles work since they all seem to use a jinja2 template.

As far as I can tell from a quick look that's correct: NetworkConfig.loadfp() expects a text file whose contents will be directly parsed as configuration lines without any interpolation or template rendering - any templating should be handled by the caller.

[lakedh]

@prasadapr can you take a look on that issue too?