Updated fields rebase (#353)

* Update regex to parse IPv6 (#334) Update default listener log path * Add HAProxy Plugin (#335) * Add haproxy plugin * Add supported platforms and min stanza version * PR Feedback fixes * Rename frontend_name to frontend_name_transport in regex * for all move operations, check if field is nil before moving. "set log type to haproxy and haproxy.error (not .http / .tcp)" * typoe: nill --> nil * typo, log_format: http --> default Co-authored-by: jsirianni <[email protected]> * Allow DBID to be empty & Correct case matching (#331) * Allow DBID to be empty & Correct case matching The DBID field is able to be empty on some versions of Oracle DB The multiline regex was looking for `Audit File`, but logs have `Audit file` * Switch to line end for multiline with double newline pattern * Fix plugin failure when using inline truncate check * Switch back to a regex parse for record splitting Co-authored-by: jsirianni <[email protected]> * Release 0.0.79 (#336) * 0.0.79 changelog * dbid oracle pr * fix release date * move frontend port to resources (#338) * Add more checks to reduce errors (#337) * Add more checks to reduce errors * Add ac_lite_ap_parser change to changelog for ubiquiti * 0.0.80 changelog Co-authored-by: jsirianni <[email protected]> * update regex to handle {} brackets before http request info (#342) * update regex to handle {} brackets before http request info * haproxy http default log format fix * make change backwards compatible * Adjust parsing further based on more detailed oracle db audit logs (#343) * release 0.0.82 * CI Testing: End to End Tests (#345) * end to end nginx testing * fix format * fix format * use sudo to compare against files from container mount * sleep so stanza can parse, kill stanza when done * try cloning log library * use token to clone lob lib * fix repo name * fix expect and output paths * handle both nginx formats * add apache_http workflow * dump container log * dump container log * 10 second sleep * use jq with diff to prevent formatting issues * sudo * cannot use sudo with redirection to diff, so just format with jq before using diff * Switch back to diff, something else is going on.. * pause and cat raw output before comparing * sudo * fix paths * sort before compare * redirect output * sort and cat * use jtool for comparing json files * use jtool for comparing json files * chmod it * haproxy workflow * add oracledb workflow * single test case for oracledb * mount plugins dir * stop and then get stanza logs * sleep 20 seconds instead of 10, sometimes 10 is not enough * fix log dirg * install jtool in its own step * fix mount * split oracle up. start with alert logs * oracle audit log * upgrade jtool and use skip timestamp for haproxy and oracle * upgrade jtoo * upgrade jtool * pause, stop, logs * listener log, oracle * Handle second {} in http log entry if present (#346) Co-authored-by: jsirianni <[email protected]> * Add tcpudp plugin (#341) * Add tcpudp plugin * Add tcpudp schema and tests * Split into two plugins udp and tcp * Add schema files for tests * Update plugins/tcp.yaml * Update plugins/tcp.yaml * Update plugins/udp.yaml * Update plugins/udp.yaml * Update test/configs/tcp/invalid/invalid_listen_port.yaml * Update plugins/udp.yaml Co-authored-by: jsirianni <[email protected]> Co-authored-by: Joseph Sirianni <[email protected]> * tcp / udp: move to message field (#347) * move to message field * \n * Add common event format plugin (#328) * Add common event format plugin * use key value parser for parsing extensions field * Promote fields to labels and resources * Promote fields to labels and resources * Update changelog * Remove key value parser * Add promote device_vendor and device_version to resources Co-authored-by: jsirianni <[email protected]> Co-authored-by: jsirianni <[email protected]> * release 0.0.83 * Add http plugin (#352) * Add http plugin * Update log_type label to http * remove duplicate param * tcp --> http * typo * token_header --> auth_header * small refactor * upgrade stanza 1.2.9 Co-authored-by: jsirianni <[email protected]> * Update Titles and uwsgi field name (#350) Co-authored-by: Joseph Sirianni <[email protected]> * Update cisco_meraki plugin to use key_value_parser (#349) * Update cisco_meraki plugin to use key_value_parser instead of custom regex * use stanza 1.2.9 Co-authored-by: jsirianni <[email protected]> * Create Sonicwall log parser plugin (#340) * Create Sonicwall log parser plugin * Add pri field severity_parser * Rename msg field to message * Add parameter location to support setting timezone * Update to use udp_input and add extra tests * Use stanza 1.2.7 for tests * update stanza and get go.sum * Update plugins/sonicwall.yaml Co-authored-by: Joseph Sirianni <[email protected]> * Fix using wrong parameter if you defined listen_port Co-authored-by: jsirianni <[email protected]> Co-authored-by: Joseph Sirianni <[email protected]> * release-0.0.84 * fix ci link * remove start_at reference * remove start_at test for sonicwall, not needed * enable new operators * fix start_at for w3c tests due to delete_at_end being added * Add cisco_catalyst plugin (#351) * Add cisco_catalyst plugin * Add severity field group to regex. Update parse from field for severity and regex. * Remove parse_to message in udp_input and parse_from message in regex_parser * fix ci link * remove start_at reference * remove start_at test for sonicwall, not needed * enable new operators * fix start_at for w3c tests due to delete_at_end being added Co-authored-by: Joseph Sirianni <[email protected]> * remove tests for now, not compatable with otel branch * remove tests for now, not compatable with otel branch * port cisco catalyst to otel * fix cef * fix haproxy * port http * port sonicwall * fix haproxy * Update plugins/cisco_catalyst.yaml Co-authored-by: Keith Schmitt <[email protected]> * rebase oracledb * try and fix severities that were missed Co-authored-by: Dylan Myers <[email protected]> Co-authored-by: EricWHolt <[email protected]> Co-authored-by: Keith Schmitt <[email protected]> Co-authored-by: schmikei <[email protected]>
observIQ · Oct 5, 2021 · c8df84e · c8df84e
1 parent 3366387
commit c8df84e
Show file tree

Hide file tree

Showing 10 changed files with 431 additions and 45 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -5,12 +5,30 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.0.85] 2021-10-04
 
-## [0.0.83] Unreleased
+### Added
+
+- Added `cisco_catalyst` plugin ([351](https://github.com/observIQ/stanza-plugins/pull/351))
+
+## [0.0.84] 2021-10-04
+
+### Added
+
+- Added `sonicwall` plugin ([PR340](https://github.com/observIQ/stanza-plugins/pull/340))
+
+### Changed
+
+- cisco_meraki: Remove custom regex parsers and use key_value_parser instead. ([PR349](https://github.com/observIQ/stanza-plugins/pull/349))
+- Update `codeigniter`, `common_event_format`, and `uwsgi`
+  - Remove Log Parser from title.
+  - Update uwsgi field `headers` to `headers_count`
+
+## [0.0.83] 2021-09-29
 
 ### Added
 
-- Add `tcp` and `udp` plugin ([PR341](https://github.com/observIQ/stanza-plugins/pull/341))
+- Added `tcp` and `udp` plugin ([PR341](https://github.com/observIQ/stanza-plugins/pull/341))
 - Added `common_event_format` plugin ([328](https://github.com/observIQ/stanza-plugins/pull/328))
 
 ### Fixed

diff --git a/README.md b/README.md
@@ -2,7 +2,7 @@
 
 stanza-plugins contains plugins for the [Stanza Log Agent](https://github.com/observIQ/stanza)
 
-[![Status](https://github.com/observIQ/stanza-plugins/workflows/Test/badge.svg)](https://github.com/observIQ/stanza-plugins/Test)
+[![Test](https://github.com/observIQ/stanza-plugins/actions/workflows/validate.yml/badge.svg)](https://github.com/observIQ/stanza-plugins/actions/workflows/validate.yml)
 
 ## Release Process
 

diff --git a/plugins/cisco_catalyst.yaml b/plugins/cisco_catalyst.yaml
@@ -0,0 +1,57 @@
+version: 0.0.1
+title: Cisco Catalyst
+description: Log parser for Cisco Catalyst
+supported_platforms:
+  - linux
+  - windows
+  - macos
+min_stanza_version: 1.2.7
+parameters:
+  - name: listen_port
+    label: Listen Port
+    description: A port which the agent will listen for udp messages
+    type: int
+    default: 514
+  - name: listen_ip
+    label: Listen IP
+    description: A UDP ip address of the form `<ip>`
+    type: string
+    default: "0.0.0.0"
+    advanced_config: true
+  - name: add_labels
+    label: Labels
+    description: Adds net.transport, net.peer.ip, net.peer.port, net.host.ip and net.host.port labels.
+    type: bool
+    default: true
+    advanced_config: true
+# Set Defaults
+# {{$listen_port := default 514 .listen_port}}
+# {{$listen_ip := default "0.0.0.0" .listen_ip}}
+# {{$start_at := default "end" .start_at}}
+# {{$add_labels := default true .add_labels}}
+pipeline:
+  - type: udp_input
+    listen_address: '{{ $listen_ip }}:{{ $listen_port }}'
+    attributes:
+      log_type: 'cisco_catalyst'
+      plugin_id: {{ .id }}
+    add_attributes: {{$add_labels}}
+
+  - type: regex_parser
+    regex: '^(?P<sequence_number>\d+):\s+(?P<timestamp>\w{3}\s+\d{1,2}\s+\d{2}:\d{2}:\d{2}\s+\w{3}):\s+%(?P<facility>[^-]+)-(?P<severity>\d+)-(?P<mnemonic>[^:]+):\s*(?P<message>.*)'
+    timestamp:
+      parse_from: $body.timestamp
+      layout_type: gotime
+      layout: 'Jan _2 15:04:05 MST'
+    severity:
+      parse_from: $body.severity
+      mapping:
+        fatal: 0
+        error3: 1
+        error2: 2
+        error: 3
+        warn2: 4
+        warn: 5
+        info: 6
+        debug: 7
+    output: {{ .output }}
diff --git a/plugins/cisco_meraki.yaml b/plugins/cisco_meraki.yaml
@@ -2,6 +2,11 @@
 version: 1.0.0
 title: Cisco Meraki
 description: Log parser for Cisco Meraki
+min_stanza_version: 1.2.7
+supported_platforms:
+  - linux
+  - windows
+  - macos
 parameters:
   - name: listen_port
     label: Listen Port
@@ -42,12 +47,12 @@ pipeline:
     type: router
     default: catch_all_parser
     routes:
-      - expr: '$body matches ".*=[\\s\\S]*"'
+      - expr: '$body matches "^<[^>]+>[\\d]+\\s+[\\d\\.]+\\s+[^\\s]+\\s+[^\\s]+\\s+[\\s\\S]*"'
         output: meraki_parser
 
   - id: meraki_parser
     type: regex_parser
-    regex: '^<(?P<priority>[^>]+)>(?P<version>[\d]+)\s*(?P<timestamp>[^\s]*)\s*(?P<hostname>[^\s]*)\s*(?P<app_name>[^\s]*)\s*(?P<message>[\s\S]*)'
+    regex: '^<(?P<priority>[^>]+)>(?P<version>[\d]+)\s*(?P<timestamp>[\d\.]*)\s*(?P<hostname>[^\s]*)\s*(?P<app_name>[^\s]*)\s*(?P<message>[\s\S]*)'
     timestamp:
       parse_from: timestamp
       layout_type: epoch
@@ -56,6 +61,7 @@ pipeline:
 
   # Severity Parser using piority field to get the severity
   - type: severity_parser
+    if: '$body.priority != nil'
     parse_from: $body.priority
     mapping:
       fatal2: [0,8,16,24,32,40,48,56,64,72,80,88,96,104,112,120,128,136,144,152,160,168,176,184]
@@ -66,39 +72,15 @@ pipeline:
       info2: [5,13,21,29,37,45,53,61,69,77,85,93,101,109,117,125,133,141,149,157,165,173,181,189]
       info: [6,14,22,30,38,46,54,62,70,78,86,94,102,110,118,126,134,142,150,158,166,174,182,190]
       debug: [7,15,23,31,39,47,55,63,71,79,87,95,103,111,119,127,135,143,151,159,167,175,183,191]
-    output: message_router
+    output: key_value_parser
 
-  # Route messages to different regex's for further parsing. If a regex match is not found then don't parse message.
-  - id: message_router
-    type: router
-    default: {{.output}}
-    routes:
-      - expr: '$body.message matches "^src=.*\\s*dst=.*\\s*protocol=.*\\s*sport=.*\\s*dport=.*\\s*translated_src_ip=.*\\s*translated_port=.*\\s*"'
-        output: message_1_parser
-      - expr: '$body.message matches "^src=.*\\s*dst=.*\\s*mac=.*\\s*protocol=.*\\s*sport=.*\\s*dport=.*"'
-        output: message_2_parser
-      - expr: '$body.message matches "^src=.*\\s*dst=.*\\s*mac=.*\\s*user=.*\\s*"'
-        output: message_3_parser
-
-  - id: message_1_parser
-    type: regex_parser
-    parse_from: $body.message
-    regex: '^src=(?P<src>[^\s]*)\s*dst=(?P<dst>[^\s]*)\s*protocol=(?P<protocol>[^\s]*)\s*sport=(?P<sport>[^\s]*)\s*dport=(?P<dport>[^\s]*)\s*translated_src_ip=(?P<translated_src_ip>[^\s]*)\s*translated_port=(?P<translated_port>[^\s]*)\s*(?P<message>[\s\S]*)'
-    output: {{.output}}
-
-  - id: message_2_parser
-    type: regex_parser
-    parse_from: $body.message
-    regex: '^src=(?P<src>[^\s]*)\s*dst=(?P<dst>[^\s]*)\s*mac=(?P<mac>[^\s]*)\s*protocol=(?P<protocol>[^\s]*)\s*sport=(?P<sport>[^\s]*)\s*dport=(?P<dport>[^\s]*)\s*(?P<message>[\s\S]*)'
-    output: {{.output}}
-
-  - id: message_3_parser
-    type: regex_parser
-    parse_from: $body.message
-    regex: '^src=(?P<src>[^\s]*)\s*dst=(?P<dst>[^\s]*)\s*mac=(?P<mac>[^\s]*)\s*user=(?P<user>[^\s]*)\s*(?P<message>[\s\S]*)'
+  - type: key_value_parser
+    if: '$body.message != nil'
+    parse_from: message
     output: {{.output}}
 
   - id: catch_all_parser
+    if: '$body matches "^<[\\d]+>"'
     type: regex_parser
     regex: '^<(?P<priority>[^>]+)>(\s*)?(?P<message>[\s\S]*)'
     output: severity_parser
diff --git a/plugins/codeigniter.yaml b/plugins/codeigniter.yaml
@@ -1,5 +1,5 @@
 version: 1.0.0
-title: CodeIgniter Log Parser
+title: CodeIgniter
 description: Log parser for CodeIgniter formatted logs
 parameters:
   - name: log_path

diff --git a/plugins/common_event_format.yaml b/plugins/common_event_format.yaml
@@ -1,6 +1,6 @@
 # Plugin Info
 version: 1.0.0
-title: Common Event Format Log Parser
+title: Common Event Format
 description: File Input Common Event Format Parser
 min_stanza_version: 1.2.7
 supported_platforms:
@@ -163,15 +163,15 @@ pipeline:
         - min: 1
           max: 3
         - low
-      warning:
+      warn2:
         - min: 4
           max: 6
         - medium
       error:
         - min: 7
           max: 8
         - high
-      critical:
+      error2:
         - min: 9
           max: 10
         - very-high

diff --git a/plugins/haproxy.yaml b/plugins/haproxy.yaml
@@ -133,8 +133,8 @@ pipeline:
     preset: none
     mapping:
       info: 2xx
-      notice: 3xx
-      warning: 4xx
+      warn: 3xx
+      warn2: 4xx
       error: 5xx
     output: uri_parser
 
@@ -165,7 +165,7 @@ pipeline:
 # protocol
 # protocol_version
 # referer
-  # Promote fields to labels
+  # Promote fields to attributes
   - id: frontend_ssl_version_move
     type: move
     if: '$body.frontend_ssl_version != nil'
@@ -258,8 +258,8 @@ pipeline:
     if: '$body.severity != nil'
     parse_from: $body.severity
     mapping:
-      critical: crit
-      emergency: emerg
+      error2: crit
+      fatal: emerg
       error: err
     output: {{ .output }}