Merge branch 'ntc_update' into 'master'

Support unknown protocol in arista_eos_show_interfaces_description (networktocode#1540)

See merge request bsh-developers/ntc-templates!48

.github/workflows/ci.yml

@@ -82,7 +82,7 @@ jobs:
     strategy:
       fail-fast: true
       matrix:
-        python-version: ["3.7", "3.8", "3.9", "3.10"]
+        python-version: ["3.8", "3.9", "3.10"]
     runs-on: "ubuntu-20.04"
     env:
       PYTHON_VER: "${{ matrix.python-version }}"
@@ -118,7 +118,7 @@ jobs:
     strategy:
       fail-fast: true
       matrix:
-        python-version: ["3.7"]
+        python-version: ["3.10"]
     env:
       PYTHON_VER: "${{ matrix.python-version }}"
     steps:
@@ -154,7 +154,7 @@ jobs:
     strategy:
       fail-fast: true
       matrix:
-        python-version: ["3.7", "3.8", "3.9", "3.10"]
+        python-version: ["3.8", "3.9", "3.10"]
     runs-on: "ubuntu-20.04"
     env:
       PYTHON_VER: "${{ matrix.python-version }}"

Dockerfile

@@ -14,9 +14,12 @@ RUN apt-get update && \
 
 RUN pip install --upgrade pip
 
-RUN curl -sSL https://install.python-poetry.org -o /tmp/install-poetry.py && \
-    python /tmp/install-poetry.py --version 1.2.0 && \
-    rm -f /tmp/install-poetry.py
+# Install Poetry manually via its installer script;
+# We might be using an older version of ntc-templates that includes an older version of Poetry
+# and CI and local development may have a newer version of Poetry
+# Since this is only used for development and we don't ship this container, pinning Poetry back is not expressly necessary
+# We also don't need virtual environments in container
+RUN which poetry || curl -sSL https://install.python-poetry.org | python3 - 
 
 # Add poetry install location to the $PATH
 ENV PATH="${PATH}:/root/.local/bin"

docs/dev/data_model.md

@@ -0,0 +1,44 @@
+# Goals
+
+- The table below becomes an evolving reference of common capture group names
+- Normalize common capture groups across templates (including cross-vendor)
+
+# Reasoning
+
+Normalized capture group names enable more predictable structured data across templates where similar data exists.
+
+Example: Imagine the extra programming logic needed to consume structured data when capture groups could have two or more names across several templates or several vendors! Yikes! This is why we seek to normalize.
+
+# Considerations
+
+- Some capture groups are a single value and others are lists of values despite containing similar data as one another
+    - Example: single IP address vs list of IP addresses
+- Normalizing and using standard capture group names will take time, persistence, and patience from the community
+- Other than new templates, these normalization changes modify the expected template output and are considered "breaking changes" to be included in version releases
+
+# Common Capture Groups
+
+| Capture Group      | Usage Description |
+|--------------------|:------------------|
+| `BIA`              | use this if the template already has MAC_ADDRESS in use for the active MAC address |
+| `DESCRIPTION`      | often used for port or interface descriptions |
+| `GATEWAY`          | gateway address for a subnet |
+| `INTERFACE`        | full word instead of IFACE, INTF, INTFC, etc |
+| `IP_ADDRESS`       | for a single IP address, often IPv4 |
+| `IP_ADDRESSES`     | for lists of IPv4 addresses, but in the case of some templates there may be a mix of IP protocol versions |
+| `IP_HELPER`        | for lists DHCP IP helper addresses |
+| `IP_VERSION`       | Internet Protocol (IP) version in the case of multiple versions appearing in output (use where necessary) |
+| `IPV6_ADDRESS`     | for a single IPv6 address |
+| `IPV6_ADDRESSES`   | for lists of IPv6 addresses |
+| `IPV6_GATEWAY`     | for IPv6 gateway address |
+| `LOCAL_IP_ADDRESS` | local IP address in the case of First Hop Redundancy Protocols (FHRP)
+| `MAC_ADDRESS`      | instead of MAC or MACADDR |
+| `MGMT_IP_ADDRESS`  | instead of MGMT_IP or MGMT_ADDRESS or MANAGEMENT_IP or REMOTE_MANAGEMENT_ADDRESS |
+| `NEIGHBOR_ID`      | for router IDs remote to the system being parsed |
+| `NETMASK`          | for IPv4 dotted quad masks |
+| `NETWORK`          | for network numbers or subnet address (without the mask or prefix/slash notation); in place of ROUTE |
+| `PREFIX_LENGTH`    | instead of PREFIX or CIDR for the numbers of a slash notation or CIDR mask |
+| `PROTOCOL`         | instead of PROTO |
+| `ROUTER_ID`        | for local router IDs (local to the device being parsed) |
+| `VLAN_ID`          | instead of VLAN or TAG |
+| `VLAN_NAME`        | VLAN name or description |

docs/dev/dev_parser.md

@@ -42,6 +42,12 @@ Start
   ^${TIME}\s+${TIMEZONE}\s+${DAYWEEK}\s+${DAY}/${MONTH}/${YEAR} -> Record
   ^. -> Error
 ```
+
+!!! tip
+    Use [standard capture group names](https://github.com/networktocode/ntc-templates/blob/master/docs/dev/data_model.md) where possible to achieve a level of normalization.
+
+    If templates you are creating or editing have capture groups that might be in common with others, please check other templates for similar capture groups as there might already be one in use. If you have a suggestion for a standard capture group name, suggest the change(s) to Network to Code for discussion and consideration.
+
 ### States
 
 If the raw output has a heading, the `Start` state should match on the column headings and then transition to another state that will match the device's output table with the capture groups. This helps ensure the regex patterns for the capture groups are attempting to match the correct information, and allows templates to easily add additional States for tables that have different headings. 
@@ -276,4 +282,4 @@ Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)
 LOCAL - Running command yamllint .
 
 [... skipping remaining output for brevity ...]
-```
+```

docs/requirements.txt

@@ -1,8 +1,5 @@
-mkdocs==1.3.1
-# Material for MkDocs theme
-mkdocs-material==8.3.9
-# Render custom markdown for version added/changed/remove notes
+mkdocs==1.5.2
+mkdocs-material==9.1.15
 mkdocs-version-annotations==1.0.0
-# Automatic documentation from sources, for MkDocs
-mkdocstrings==0.19
-mkdocstrings-python==0.7.1
+mkdocstrings-python==1.5.2
+mkdocstrings==0.22.0

ntc_templates/parse.py

@@ -56,7 +56,9 @@ def parse_output(platform=None, command=None, data=None):
         cli_table.ParseCmd(data, attrs)
         structured_data = _clitable_to_dict(cli_table)
     except clitable.CliTableError as err:
-        raise Exception(f'Unable to parse command "{command}" on platform {platform} - {str(err)}') from err
+        raise Exception(  # pylint: disable=broad-exception-raised
+            f'Unable to parse command "{command}" on platform {platform} - {str(err)}'
+        ) from err
         # Invalid or Missing template
         # module.fail_json(msg='parsing error', error=str(e))
         # rather than fail, fallback to return raw text

ntc_templates/templates/arista_eos_show_interfaces_description.textfsm

@@ -1,7 +1,7 @@
 Value PORT (\S+)
 Value STATUS (up|down|admin\s+down)
 Value PROTOCOL (up|down|lowerlayerdown|notpresent|unknown)
-Value DESCRIP (\S.*?)
+Value DESCRIPTION (\S.*?)
 
 Start
   ^Interface\s+Status\s+Protocol\s+Description\s*$$ -> Begin
@@ -12,6 +12,6 @@ Start
   ^. -> Error
 
 Begin
-  ^${PORT}\s+${STATUS}\s+${PROTOCOL}(?:\s+${DESCRIP})?\s*$$ -> Record
+  ^${PORT}\s+${STATUS}\s+${PROTOCOL}(?:\s+${DESCRIPTION})?\s*$$ -> Record
   ^\s*$$
   ^. -> Error

ntc_templates/templates/aruba_aoscx_show_interface.textfsm

@@ -1,4 +1,4 @@
-Value INTERFACE (\S+)
+Value Required INTERFACE (\S+)
 Value LINK_STATUS (\w+)
 Value LINK_ADMIN (\w+)
 Value LINK_STATE_INFO (\.*)
@@ -8,6 +8,7 @@ Value HW_TYPE (\w+)
 Value MAC_ADDRESS (\S+)
 Value MTU (\d+)
 Value IF_TYPE (\S+)
+Value IP_ADDRESS (\S+)
 Value DUPLEX (\w+)
 Value QOS_TRUST (\S+)
 Value SPEED (\S+\s\S+) 
@@ -16,6 +17,9 @@ Value FLOW_CONTROL (\S+)
 Value ERROR_CONTROL (\S+)
 Value VLAN_MODE (\S+)
 Value VLAN_ACCESS (\S+)
+Value VLAN_NATIVE (\S+)
+Value List VLAN_TRUNK ([^,]+)
+Value List AGGREGATED_INTERFACES (\S+)
 Value RX_TOTAL_PACKETS (\d+)
 Value RX_TOTAL_BYTES (\d+)
 Value RX_UNICAST_PACKETS (\d+)
@@ -31,28 +35,58 @@ Value TX_MCAST_PACKETS (\d+)
 Value TX_BROADCAST_PACKETS (\d+)
 Value TX_ERRORS (\d+)
 Value TX_DROPPED (\d+)
-Value TX_CRC_FCS (\d+)
+Value TX_CRC_FCS (\S+)
 Value TX_COLLISION (\d+)
 
 Start
-  ^\s*Interface\s*${INTERFACE}\s*is\s*${LINK_STATUS}
+  ^\s*(Interface|Aggregate)\s+\S+\s+is -> Continue.Record
+  ^\s*(Interface|Aggregate)\s+${INTERFACE}\s+is\s+${LINK_STATUS}
   ^\s*Admin\s*state\s*is\s*${LINK_ADMIN}
-  ^\s*Link\s*transitions:\s*${LINK_TRANSITIONS}
-  ^\s*Description:\s${INTERFACE_DESC}
-  ^\s*State\s*information:\s*${LINK_STATE_INFO}
-  ^\s*Hardware:\s*${HW_TYPE},?\s*MAC\s*Address:\s+${MAC_ADDRESS}
+  ^\s*Link\s*transitions\s*:\s*${LINK_TRANSITIONS}
+  ^\s*Description\s*:\s*${INTERFACE_DESC}
+  ^\s*State\s*information\s*:\s*${LINK_STATE_INFO}
+  ^\s*Hardware\s*:\s*${HW_TYPE},?\s*MAC\s*Address:\s+${MAC_ADDRESS}
+  ^\s*MAC\s+Address\s*:\s*${MAC_ADDRESS}
+  ^\s*IPv4\s+address\s+${IP_ADDRESS}
   ^\s*MTU\s*${MTU}
   ^\s*Type\s*${IF_TYPE}
   ^\s*${DUPLEX}-duplex
   ^\s*qos\s*trust\s*${QOS_TRUST}
-  ^\s*Speed\s*${SPEED}
+  ^\s*Speed\s*:*\s*${SPEED}
   ^\s*Auto-negotiation\s*is\s*${AUTO_NEG}
-  ^\s*Flow-control: ${FLOW_CONTROL}
-  ^\s*Error-control: ${ERROR_CONTROL}
-  ^\s*VLAN Mode: ${VLAN_MODE}
-  ^\s*Access VLAN: ${VLAN_ACCESS}
+  ^\s*Flow-control\s*:\s*${FLOW_CONTROL}
+  ^\s*Error-control\s*:\s*${ERROR_CONTROL}
+  ^\s*VLAN Mode\s*:\s*${VLAN_MODE}
+  ^\s*Access VLAN\s*:\s*${VLAN_ACCESS}
+  ^\s*Native VLAN\s*:\s*${VLAN_NATIVE}
+  ^\s*Allowed VLAN List\s*:\s+${VLAN_TRUNK},* -> Continue
+  ^\s*Allowed VLAN List\s*:\s+(?:[^,]+,){1}${VLAN_TRUNK},* -> Continue
+  ^\s*Allowed VLAN List\s*:\s+(?:[^,]+,){2}${VLAN_TRUNK},* -> Continue
+  ^\s*Allowed VLAN List\s*:\s+(?:[^,]+,){3}${VLAN_TRUNK},* -> Continue
+  ^\s*Allowed VLAN List\s*:\s+(?:[^,]+,){4}${VLAN_TRUNK},* -> Continue
+  ^\s*Allowed VLAN List\s*:\s+(?:[^,]+,){5}${VLAN_TRUNK},* -> Continue
+  ^\s*Allowed VLAN List\s*:
+  ^\s*Link state
+  ^\s*Aggregated-interfaces\s*:\s+${AGGREGATED_INTERFACES}\s* -> Continue
+  ^\s*Aggregated-interfaces\s*:\s+(?:\S+\s+){1}${AGGREGATED_INTERFACES}\s* -> Continue
+  ^\s*Aggregated-interfaces\s*:\s+(?:\S+\s+){2}${AGGREGATED_INTERFACES}\s* -> Continue
+  ^\s*Aggregated-interfaces\s*:\s+(?:\S+\s+){3}${AGGREGATED_INTERFACES}\s* -> Continue
+  ^\s*Aggregated-interfaces\s*:\s+(?:\S+\s+){4}${AGGREGATED_INTERFACES}\s* -> Continue
+  ^\s*Aggregated-interfaces\s*:\s+(?:\S+\s+){5}${AGGREGATED_INTERFACES}\s* -> Continue
+  ^\s*Aggregated-interfaces\s*:\s+(?:\S+\s+){6}${AGGREGATED_INTERFACES}\s* -> Continue
+  ^\s*Aggregated-interfaces\s*:\s+(?:\S+\s+){7}${AGGREGATED_INTERFACES}\s* -> Continue
+  ^\s*Aggregated-interfaces\s*:
+  ^\s*Aggregation-key
+  ^\s*Aggregate\s+mode
+  ^\s*Persona
+  ^\s*rate-limit\s+broadcast
+  ^\s*L3\s+Counters
+  ^\s*MDI mode
+  ^\s*Rate collection interval
+  ^\s*Energy-Efficient
   ^\s*Rx -> Rx
-  ^\s*Description:
+  ^\s*Rate\s+RX\s+TX\s+Total -> Rate
+  ^\s*Statistic\s+RX\s+TX\s+Total -> Statistic
   ^. -> Error
 
 Rx
@@ -73,5 +107,36 @@ Tx
   ^\s*${TX_ERRORS} errors\s*${TX_DROPPED} dropped
   ^\s*${TX_CRC_FCS} CRC/FCS
   ^\s*${TX_COLLISION}\s*collision
-  ^\s*$$ -> Record Start
-  ^. -> Error
+  ^\s*$$ -> Start
+  ^. -> Error
+
+Rate
+  ^\s*----
+  ^\s*Mbits\s+/\s+sec
+  ^\s*KPkts\s+/\s+sec
+  ^\s+Unicast
+  ^\s+Multicast
+  ^\s+Broadcast
+  ^\s+Utilization
+  ^\s*$$ -> Statistic
+  ^. -> Error
+
+Statistic
+  ^\s*Statistic\s+RX\s+TX\s+Total
+  ^\s*----
+  ^\s*Packets\s+${RX_TOTAL_PACKETS}\s+${TX_TOTAL_PACKETS}\s+\d+
+  ^\s+Unicast\s+${RX_UNICAST_PACKETS}\s+${TX_UNICAST_PACKETS}\s+\d+
+  ^\s+Multicast\s+${RX_MCAST_PACKETS}\s+${TX_MCAST_PACKETS}\s+\d+
+  ^\s+Broadcast\s+${RX_BROADCAST_PACKETS}\s+${TX_BROADCAST_PACKETS}\s+\d+
+  ^\s*Bytes\s+${RX_TOTAL_BYTES}\s+${TX_TOTAL_BYTES}\s+\d+
+  ^\s*Jumbos
+  ^\s*Pause\s+Frames
+  ^\s*Dropped\s+${RX_DROPPED}\s+${TX_DROPPED}\s+\d+
+  ^\s*Errors\s+${RX_ERRORS}\s+${TX_ERRORS}\s+\d+
+  ^\s+CRC/FCS\s+${RX_CRC_FCS}\s+${TX_CRC_FCS}\s+\d+
+  ^\s+Collision\s+\S+\s+${TX_COLLISION}\s+\d+
+  ^\s*Runts
+  ^\s*Giants
+  ^\s*L3\s+Packets
+  ^\s*$$ -> Start
+  ^. -> Error

ntc_templates/templates/aruba_aoscx_show_lldp_neighbors-info_detail.textfsm

@@ -12,7 +12,9 @@ Value PORT_DESC (\S+)
 Start
   ^Port\s*:\s*${LOCAL_PORT}
   ^Neighbor\s*Chassis-Name\s*:\s*${CHASSIS_NAME}
+  ^Neighbor\s+System-Name\s*:\s*${CHASSIS_NAME}
   ^Neighbor\s*Chassis-Description\s*:\s*${CHASSIS_DESC}
+  ^Neighbor\s+System-Description\s*:\s*${CHASSIS_DESC}
   ^Neighbor\s*Chassis-ID\s*:\s*${CHASSIS_ID}
   ^Neighbor\s*Management-Address\s*:\s*${MGMT_ADDR}
   ^Chassis\s*Capabilities\s*Available\s*:\s*${CAPABILITIES}
@@ -24,9 +26,12 @@ Start
   ^TTL\.*
   ^P[D|SE]\.*
   ^Power\.*
+  ^MED\s+capabilities
   ^\s+ -> Next
   ^\S+#\.* -> Next
   ^LLDP\.*
+  ^Link\s+aggregation 
+  ^Aggregation\s+port
   ^Total\.*
   ^=+
   ^. -> Error

ntc_templates/templates/aruba_aoscx_show_vlan.textfsm

@@ -0,0 +1,34 @@
+Value Required VLAN_ID (\d+)
+Value VLAN_NAME (\S+)
+Value STATUS (\S+)
+Value REASON (\S+)
+Value TYPE (\S+)
+Value List INTERFACES ([^,]+)
+
+Start
+  ^---
+  ^VLAN\s+Name\s+Status -> VLANS
+  ^. -> Error
+
+VLANS
+  ^---
+  ^\d+ -> Continue.Record
+  # Lines starting with VLAN ID
+  ^${VLAN_ID}\s+${VLAN_NAME}\s+${STATUS}\s+${REASON}\s+${TYPE}\s*$$
+  ^${VLAN_ID}\s+${VLAN_NAME}\s+${STATUS}\s+${REASON}\s+${TYPE}\s+${INTERFACES},* -> Continue
+  ^\d+\s+(?:\S+\s+){4}(?:[^,]+,){1}${INTERFACES},* -> Continue
+  ^\d+\s+(?:\S+\s+){4}(?:[^,]+,){2}${INTERFACES},* -> Continue
+  ^\d+\s+(?:\S+\s+){4}(?:[^,]+,){3}${INTERFACES},* -> Continue
+  ^\d+\s+(?:\S+\s+){4}(?:[^,]+,){4}${INTERFACES},* -> Continue
+  ^\d+\s+(?:\S+\s+){4}(?:[^,]+,){5}${INTERFACES},* -> Continue
+  # Lines starting with multiple spaces
+  ^\s{6,}${INTERFACES},* -> Continue
+  ^\s{6,}(?:[^,]+,){1}${INTERFACES},* -> Continue
+  ^\s{6,}(?:[^,]+,){2}${INTERFACES},* -> Continue
+  ^\s{6,}(?:[^,]+,){3}${INTERFACES},* -> Continue
+  ^\s{6,}(?:[^,]+,){4}${INTERFACES},* -> Continue
+  ^\s{6,}(?:[^,]+,){5}${INTERFACES},* -> Continue
+  # Dropping lines
+  ^\d+\s+\S+\s+\S+\s+\S+\s+\S+\s*
+  ^\s{6,}
+  ^.+ -> Error

ntc_templates/templates/cisco_asa_ping.textfsm

@@ -2,18 +2,19 @@ Value Required SENT_QTY (\d+)
 Value Required SENT_TYPE (.*)
 Value Required DESTINATION (\S+)
 Value Required TIMEOUT (\d+)
-Value Required RESPONSE_STREAM ([\.\!]+)
+Value Required RESPONSE_STREAM ([\.\!/Q/U]+)
 Value Required SUCCESS_PCT (\d+)
 Value Required SUCCESS_QTY (\d+)
-Value Required RTT_MIN (\d+)
-Value Required RTT_AVG (\d+)
-Value Required RTT_MAX (\d+)
+Value RTT_MIN (\d+)
+Value RTT_AVG (\d+)
+Value RTT_MAX (\d+)
 
 
 Start
   ^Type\s+escape\s+sequence\s+to\s+abort.
   ^Sending\s+${SENT_QTY},\s+${SENT_TYPE}\s+to\s+${DESTINATION},\s+timeout\s+is\s+${TIMEOUT}\s+seconds:
   ^${RESPONSE_STREAM}
-  ^Success\s+rate\s+is\s+${SUCCESS_PCT}\s+percent\s+\(${SUCCESS_QTY}/\d+\),\s+round-trip\s+min/avg/max\s+=\s+${RTT_MIN}/${RTT_AVG}/${RTT_MAX}\s+ms
+  ^Success\s+rate\s+is\s+${SUCCESS_PCT}\s+percent\s+\(${SUCCESS_QTY}\/\d+\)(?:,\s+round-trip\s+min/avg/max\s+\=\s+)?(?:${RTT_MIN}/${RTT_AVG}/${RTT_MAX})?(?:\sms)?
   ^\s*$$
-  ^. -> Error
+  # Error out if raw data does not match any above rules.
+  ^.* -> Error "Could not parse line:"