[Bug] Module zos_mvs_raw errors with long multi-line quoted string in content field

[PedroTreeev]

SUMMARY

When using module zos_mvs_raw with program IDCAMS in the parameter CONTENT inserting a string the quote is interpreted as content and escaped, getting the message "IDC3211I KEYWORD '\"' IS IMPROPER"

COMPONENT NAME

zos_mvs_raw

ANSIBLE VERSION

Ansible version output: 
ansible [core 2.13.1]
  config file = None
  configured module search path = ['/Users/pedrotrevino/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/homebrew/Cellar/ansible/6.0.0/libexec/lib/python3.10/site-packages/ansible
  ansible collection location = /Users/pedrotrevino/.ansible/collections:/usr/share/ansible/collections
  executable location = /opt/homebrew/bin/ansible
  python version = 3.10.5 (main, Jun 23 2022, 17:14:57) [Clang 13.1.6 (clang-1316.0.21.2.5)]
  jinja version = 3.1.2
  libyaml = True

SPECIFY ANSIBLE COLLECTION VERSION

version": "1.4.0-beta.1

SPECIFY THE Z OPEN AUTOMATION UTILITIES VERSION

zoau version 1.1.1

ENVIRONMENT

z/OS 2.4
IBM Enterprise Python 3.8.2

STEPS TO REPRODUCE

In the definition of clusters, on the CONTENT paramter, using large strings

  - name: Define cluster 1
    zos_mvs_raw:
      program_name: idcams
      auth: yes
      dds:
        - dd_output:
            dd_name: sysprint
            return_content:
              type: text
        - dd_input:
            dd_name: sysin
            content:
              "  DEFINE CLUSTER(                                -
                                NAME({{ DFS_AUTH_LIB_HLQ1 }}.{{ DFS_AUTH_LIB_HLQ2 }}.RECON1) -
                                CYL(10 10)  -
                                FREESPACE(20 20)                -
                                INDEXED                         -
                                KEYS(32 0)                      -
                                NOERASE                         -
                                NONSPANNED                      -
                                NOREUSE                         -
                                SHAREOPTIONS(3 3)               -
                                SPEED                           -
                                UNORDERED                       -
                                RECORDSIZE(4086 32600)          -
                {% if DFS_SMS_ENVIRONMENT %}
                                STORAGECLASS({{DFS_SMS_CLASS}})                 -
                {% endif %}
                {% if not DFS_DS_VOLUME1 == None  %}
                                VOLUMES({{ DFS_DS_VOLUME1 }})                 -
                {% endif %}
                                UNIQUE                          -
                                )"

When you try to execute the playbooks, it will fail on the Define clusters task with the next message "IDC3211I KEYWORD '\"' IS IMPROPER"

EXPECTED RESULTS

Is expected that CONTENT read the entire block of JCL code without any problem, allowing the definition of the cluster that will lead to the creation of 2 files with extension .DATA and .INDEX

ACTUAL RESULTS

The execution of the playbook continues but without define the clusters, it just returns "IDC3211I KEYWORD '\"' IS IMPROPER" and code 12

Playbook output:
changed: [zosendpoint] => {
    "backups": [],
    "changed": true,
    "dd_names": [
        {
            "byte_count": 1089,
            "content": [
                "\fIDCAMS  SYSTEM SERVICES                                           TIME: 14:49:07        08/15/22     PAGE      1",
                "",
                "        ",
                "\"  DEFINE CLUSTER(                                -",
                "                  NAME(IMSTESTL.IMS1.RECON1) -",
                "                  CYL(10 10)  -",
                "                  FREESPACE(20 20)                -",
                "                  INDEXED                         -",
                "                  KEYS(32 0)                      -",
                "                  NOERASE                         -",
                "                  NONSPANNED                      -",
                "                  NOREUSE                         -",
                "                  SHAREOPTIONS(3 3)               -",
                "                  SPEED                           -",
                "                  UNORDERED                       -",
                "                  RECORDSIZE(4086 32600)          -",
                "                      VOLUMES(USRV01)                 -",
                "                    UNIQUE                          -",
                "                  )\"",
                "",
                "IDC3211I KEYWORD '\"' IS IMPROPER",
                "",
                "IDC3202I ABOVE TEXT BYPASSED UNTIL NEXT COMMAND. CONDITION CODE IS 12",
                "",
                "        ",
                "",
                "IDC0002I IDCAMS PROCESSING COMPLETE. MAXIMUM CONDITION CODE WAS 12",
                ""
            ],
            "dd_name": "sysprint",
            "name": "OMVSADM.P6777661.T0727611.C0000000",
            "record_count": 28
        }
    ],
    "invocation": {
        "module_args": {
            "auth": true,
            "dds": [
                {
                    "dd_concat": null,
                    "dd_data_set": null,
                    "dd_dummy": null,
                    "dd_input": null,
                    "dd_output": {
                        "dd_name": "sysprint",
                        "return_content": {
                            "response_encoding": "iso8859-1",
                            "src_encoding": "ibm-1047",
                            "type": "text"
                        }
                    },
                    "dd_unix": null,
                    "dd_vio": null
                },
                {
                    "dd_concat": null,
                    "dd_data_set": null,
                    "dd_dummy": null,
                    "dd_input": {
                        "content": "\"  DEFINE CLUSTER(                                -\n                  NAME(IMSTESTL.IMS1.RECON1) -\n                  CYL(10 10)  -\n                  FREESPACE(20 20)                -\n                  INDEXED                         -\n                  KEYS(32 0)                      -\n                  NOERASE                         -\n                  NONSPANNED                      -\n                  NOREUSE                         -\n                  SHAREOPTIONS(3 3)               -\n                  SPEED                           -\n                  UNORDERED                       -\n                  RECORDSIZE(4086 32600)          -\n                      VOLUMES(USRV01)                 -\n                    UNIQUE                          -\n                  )\"\n",
                        "dd_name": "sysin",
                        "return_content": null
                    },
                    "dd_output": null,
                    "dd_unix": null,
                    "dd_vio": null
                }
            ],
            "parm": null,
            "program_name": "idcams",
            "verbose": false
        }
    },
    "ret_code": {
        "code": 12
    }
}
Read vars_file 'vars_zvm/ims-dbdc.yml'

CONFIGURATION

Ansible.cfg

'ansible-config dump --only-changed' output:
DEFAULT_FORKS(/Users/pedrotrevino/git/github/playbooks/ansible.cfg) = 25
DEFAULT_MODULE_PATH(env: ANSIBLE_LIBRARY) = ['/Users/pedrotrevino/.ansible/collections/ansible_collections/ibm/ibm_zos_core/plugins/module>

Inventory

Inventory content:
zvm:
  hosts:
    zvm:
      ansible_host: ec33023a.vmec.svl.ibm.com
      ansible_user: omvsadm
      ansible_python_interpreter: /python3/usr/lpp/IBM/cyp/v3r8/pyz/bin/python3

Vars

'group_vars' or 'host_vars' content:
        "system_environment": {
            "LANG": "C",
            "LIBPATH": "/zoau/v1.1.1-ptf1/lib:/python3/usr/lpp/IBM/cyp/v3r8/pyz/bin/python3/lib:/lib:/usr/lib:.",
            "PATH": "/zoau/v1.1.1-ptf1/bin:/python3/usr/lpp/IBM/cyp/v3r8/pyz/bin/python3/bin:/bin:/var/bin:/usr/lpp/java/java180/J8.0_64/bin:/usr/sbin:.",
            "PYTHONPATH": "/zoau/v1.1.1-ptf1/lib",
            "STEPLIB": "IMSBLD.I15RTSMM.COMRESL1",
            "ZOAU_HOME": "/zoau/v1.1.1-ptf1",
            "_BPXK_AUTOCVT": "ON",
            "_CEE_RUNOPTS": "FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)",
            "_TAG_REDIR_ERR": "txt",
            "_TAG_REDIR_IN": "txt",
            "_TAG_REDIR_OUT": "txt"
        }

SCREENSHOTS

[fernandofloresg]

Hi @PedroTreeev I executed the sample playbook and the error was different than the described :

TASK [Define cluster 1] ***************************************************************************************************************************************** fatal: [zvm]: FAILED! => {"backups": [], "changed": false, "dd_names": [], "msg": "DatasetWriteError('An error occurred during write of data set \"OMVSADM.P0332190.T0999343.C0000000\". RC=1. ')", "ret_code": {"code": 8}}

Could you please review and share the correct playbook.

[fernandofloresg]

I was able to reproduce the issue, using :

- name: Define cluster 1
      zos_mvs_raw:
        program_name: idcams
        auth: yes
        dds:
          - dd_output:
              dd_name: sysprint
              return_content:
                type: text
          - dd_input:
              dd_name: sysin
              content: |
                "DEFINE CLUSTER(                                -
                                  NAME(OMVSADM.TESTMVS.RECON1) -
                                  CYL(10 10)  -
                                  FREESPACE(20 20)                -
                                  INDEXED                         -
                                  KEYS(32 0)                      -
                                  NOERASE                         -
                                  NONSPANNED                      -
                                  NOREUSE                         -
                                  SHAREOPTIONS(3 3)               -
                                  SPEED                           -
                                  UNORDERED                       -
                                  RECORDSIZE(4086 32600)          -
                                  VOLUMES({{ DFS_DS_VOLUME1 }})   -
                                  UNIQUE                          -
                                  )"
      register: result

[ddimatos]

@fernandofloresg I was comparing your recreation (playbook above) to the issue #373 and could not see much of a difference other than the content in #373 started with a leading space.

I used your solution from #373 to add the block style indicator and used @PedroTreeev's playbook and it ran fine, when I removed the block style indicator i got the below error:

fatal: [zvm]: FAILED! => {"backups": [], "changed": false, "dd_names": [], "msg": "DatasetWriteError('An error occurred during write of data set \"OMVSADM.P0332010.T0799238.C0000000\". RC=1. ')", "ret_code": {"code": 8}}

For me all this works so long as there is a block style indicator, see my attached playbook.

zos_raw_2.yml.txt

Could you confirm what I saw and if we both see success then we can rely on a doc update to clarify things and we can close this

[ddimatos]

@PedroTreeev
Here are some tests I created to explore different ways of providing SYSIN content. Your issue is closest to bullet (5) , see (1) - (3) for alternatives while we address the other bullets.

---
- hosts: zvm
  collections:
    - ibm.ibm_zos_core
  gather_facts: false
  environment: "{{ environment_vars }}"

  tasks:
    # --------------------------------------------------------------------------
    # (1) SYSIN multi-line input as a list of strings with 1 line per list item
    # --------------------------------------------------------------------------
    - name: Define cluster
      zos_mvs_raw:
        program_name: idcams
        auth: yes
        dds:
          - dd_output:
              dd_name: sysprint
              return_content:
                type: text
          - dd_input:
              dd_name: sysin
              content:
                - "  DEFINE CLUSTER -"
                - "            (NAME(ANSIBLE.TEST.VSAM) -"
                - "            CYL(10 10)  -"
                - "            FREESPACE(20 20) -"
                - "            INDEXED -"
                - "            KEYS(32 0) -"
                - "            NOERASE -"
                - "            NONSPANNED -"
                - "            NOREUSE -"
                - "            SHAREOPTIONS(3 3) -"
                - "            SPEED -"
                - "            UNORDERED -"
                - "            RECORDSIZE(4086 32600) -"
                - "            VOLUMES(222222) -"
                - "            UNIQUE -"
                - "            )"
      register: result

    - name: Result
      debug:
         msg: "{{result}}"

    - name: Delete cluster
      zos_mvs_raw:
        program_name: idcams
        auth: yes
        dds:
          - dd_output:
              dd_name: sysprint
              return_content:
                type: text
          - dd_input:
              dd_name: sysin
              content:
                - "  DELETE ANSIBLE.TEST.VSAM CLUSTER -"
                - "            PURGE"
      register: result

    - name: show  response
      debug:
        msg: "{{ result }}"

    # --------------------------------------------------------------------------
    # (2) SYSIN multi-line as a multi-line string
    # Using block style with block style indicator with indentation of `1`,
    # for some reason any other value than 1 will not work, no value truncates
    # the first char from "DEFINE" to "EFINE", greator than 1 errors. Looking
    # into this but the example as is will work below and whats nice is if you
    # have quotes it will preserve them in the sysin.
    # --------------------------------------------------------------------------
    - name: Define cluster
      zos_mvs_raw:
        program_name: idcams
        auth: yes
        dds:
          - dd_output:
              dd_name: sysprint
              return_content:
                type: text
          - dd_input:
              dd_name: sysin
              content: |1
                DEFINE CLUSTER -
                          (NAME(ANSIBLE.TEST.VSAM) -
                          CYL(10 10)  -
                          FREESPACE(20 20) -
                          INDEXED -
                          KEYS(32 0) -
                          NOERASE -
                          NONSPANNED -
                          NOREUSE -
                          SHAREOPTIONS(3 3) -
                          SPEED -
                          UNORDERED -
                          RECORDSIZE(4086 32600) -
                          VOLUMES(222222) -
                          UNIQUE)
      register: result

    - name: Result
      debug:
         msg: "{{result}}"

    - name: Delete cluster
      zos_mvs_raw:
        program_name: idcams
        auth: yes
        dds:
          - dd_output:
              dd_name: sysprint
              return_content:
                type: text
          - dd_input:
              dd_name: sysin
              content:
                - "  DELETE ANSIBLE.TEST.VSAM CLUSTER -"
                - "            PURGE"
      register: result

    - name: show  response
      debug:
        msg: "{{ result }}"

    # --------------------------------------------------------------------------
    # (3) SYSIN multi-line as a multi-line string
    # Using a block style indicator without a indentation indicator causes
    # truncation of the first char thus "DEFINE CLUSTER" becomes "EFINE CLUSTER",
    # if you don't want to use a indentation indicator then you must prefix
    # "DEFINE" with a char that can be discarded, eg "DEFINE CLUSTER" should be
    # "DDEFINE CLUSTER", otherwise you will get "IDC3219I VERB NAME 'EFINE' UNKNOWN",
    # --------------------------------------------------------------------------
    - name: Define cluster
      zos_mvs_raw:
        program_name: idcams
        auth: yes
        dds:
          - dd_output:
              dd_name: sysprint
              return_content:
                type: text
          - dd_input:
              dd_name: sysin
              content: |
                  DDEFINE CLUSTER -
                    (NAME(IMSTESTL.IMS1.RECON4) -
                    CYL(10 10) -
                    FREESPACE(20 20) -
                    INDEXED -
                    KEYS(32 0) -
                    NOERASE -
                    NONSPANNED -
                    NOREUSE -
                    SHAREOPTIONS(3 3) -
                    SPEED -
                    UNORDERED -
                    RECORDSIZE(4086 32600) -
                    VOLUMES(222222) -
                    UNIQUE)"
      register: result

    - name: Result
      debug:
         msg: "{{result}}"

    - name: Delete cluster
      zos_mvs_raw:
        program_name: idcams
        auth: yes
        dds:
          - dd_output:
              dd_name: sysprint
              return_content:
                type: text
          - dd_input:
              dd_name: sysin
              content:
                - "  DELETE ANSIBLE.TEST.VSAM CLUSTER -"
                - "            PURGE"
      register: result

    - name: Result
      debug:
         msg: "{{result}}"

    # --------------------------------------------------------------------------
    # (4) SYSIN multi-line as a multi-line string
    # This does NOT work because with a multi-line string all spaces are preserved
    # and when the SYSIN content is put into a temporary data set by ZOAU that
    # with LRECL of 80 the data (string) ends up being 1384 chars long which is
    # greater than 80.
    # eg:
    #   "  DEFINE CLUSTER -            (NAME(IMSTESTL.IMS1.RECON4) -            CYL(10 10) -            FREESPACE(20 20) -            INDEXED -            KEYS(32 0) -            NOERASE -            NONSPANNED -            NOREUSE -            SHAREOPTIONS(3 3) -            SPEED -            UNORDERED -            RECORDSIZE(4086 32600) -            VOLUMES(222222) -            UNIQUE)"
    # I also tried using the contintuation marker "+" which removes preceding
    # spaces but it did not work, line was too long.
    # Ansible could in the future pass its own data set reflecting the correct
    # LRECL else ZOAU could as well cater to this.
    # --------------------------------------------------------------------------
    - name: Define cluster 
      zos_mvs_raw:
        program_name: idcams
        auth: yes
        dds:
          - dd_output:
              dd_name: sysprint
              return_content:
                type: text
          - dd_input:
              dd_name: sysin
              content:
                "  DEFINE CLUSTER -
                            (NAME(IMSTESTL.IMS1.RECON4) -
                            CYL(10 10) -
                            FREESPACE(20 20) -
                            INDEXED -
                            KEYS(32 0) -
                            NOERASE -
                            NONSPANNED -
                            NOREUSE -
                            SHAREOPTIONS(3 3) -
                            SPEED -
                            UNORDERED -
                            RECORDSIZE(4086 32600) -
                            VOLUMES(222222) -
                            UNIQUE)"
      ignore_errors: yes
      register: result

    - name: Result
      debug:
         msg: "{{result}}"

    # --------------------------------------------------------------------------
    # (5) SYSIN multi-line as a multi-line string
    # This does NOT work with a bock style indicator, some odd escaping happens
    # making the sysin content invalid with an error of:
    # "IDC3211I KEYWORD '\"' IS IMPROPER", which appears to be that YAML is
    # escaping the the double quote.
    # --------------------------------------------------------------------------
    - name: Define cluster
      zos_mvs_raw:
        program_name: idcams
        auth: yes
        dds:
          - dd_output:
              dd_name: sysprint
              return_content:
                type: text
          - dd_input:
              dd_name: sysin
              content: |
                "  DEFINE CLUSTER -
                            (NAME(IMSTESTL.IMS1.RECON4) -
                            CYL(10 10) -
                            FREESPACE(20 20) -
                            INDEXED -
                            KEYS(32 0) -
                            NOERASE -
                            NONSPANNED -
                            NOREUSE -
                            SHAREOPTIONS(3 3) -
                            SPEED -
                            UNORDERED -
                            RECORDSIZE(4086 32600) -
                            VOLUMES(222222) -
                            UNIQUE)"
      ignore_errors: yes
      register: result

    - name: Result
      debug:
         msg: "{{result}}"

    # --------------------------------------------------------------------------
    # (6) SYSIN multi-line input as a list with 1 line per list item
    # This is NOT supported
    # Documentation explains this:
    #   Multi-line input can be provided as a multi-line string or a list of
    #   strings with 1 line per list item.
    #   If a multi-line string is provided make sure to use the proper literal
    #   block style indicator '|'
    # --------------------------------------------------------------------------
    - name: Define cluster
      zos_mvs_raw:
        program_name: idcams
        auth: yes
        dds:
          - dd_output:
              dd_name: sysprint
              return_content:
                type: text
          - dd_input:
              dd_name: sysin
              content:
                -  DEFINE CLUSTER -
                -            (NAME(ANSIBLE.TEST.VSAM) -
                -            CYL(10 10)  -
                -            FREESPACE(20 20) -
                -            INDEXED -
                -            KEYS(32 0) -
                -            NOERASE -
                -            NONSPANNED -
                -            NOREUSE -
                -            SHAREOPTIONS(3 3) -
                -            SPEED -
                -            UNORDERED -
                -            RECORDSIZE(4086 32600) -
                -            VOLUMES(222222) -
                -            UNIQUE -
                -            )
      ignore_errors: yes
      register: result

    - name: Result
      debug:
         msg: "{{result}}"