[Logstash] Add ability to set condition for logs and metrics.

[BenB196]

Enhancement

What does this PR do?

1. Add the ability to set the condition option to both Logstash Logs and Metrics inputs

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• condition is a top level of both inputs as I couldn't think of a reason where a user would want conditions to be different based on what is being collected within an input, and this greatly simplifies things for users.

Before Upgrade

inputs:
  - id: logfile-logstash-b78cdf23-6f82-456f-95a6-92729c338a05
    name: logstash-1
    revision: 1
    type: logfile
    use_output: default
    meta:
      package:
        name: logstash
        version: 2.2.2-preview1
    data_stream:
      namespace: default
    package_policy_id: b78cdf23-6f82-456f-95a6-92729c338a05
    streams:
      - id: logfile-logstash.log-b78cdf23-6f82-456f-95a6-92729c338a05
        data_stream:
          dataset: logstash.log
          type: logs
        paths:
          - /var/log/logstash/logstash-json*.log
        exclude_files:
          - .gz$
        multiline:
          pattern: '^((\[[0-9]{4}-[0-9]{2}-[0-9]{2}[^\]]+\])|({.+}))'
          negate: true
          match: after
        processors:
          - add_locale.when.not.regexp.message: '^{'
          - add_fields:
              target: ''
              fields:
                ecs.version: 1.10.0
      - id: logfile-logstash.slowlog-b78cdf23-6f82-456f-95a6-92729c338a05
        data_stream:
          dataset: logstash.slowlog
          type: logs
        paths:
          - /var/log/logstash/logstash-slowlog-json*.log
        exclude_files:
          - .gz$
        processors:
          - add_locale.when.not.regexp.message: '^{'
          - add_fields:
              target: ''
              fields:
                ecs.version: 1.10.0
  - id: logstash/metrics-logstash-b78cdf23-6f82-456f-95a6-92729c338a05
    name: logstash-1
    revision: 1
    type: logstash/metrics
    use_output: default
    meta:
      package:
        name: logstash
        version: 2.2.2-preview1
    data_stream:
      namespace: default
    package_policy_id: b78cdf23-6f82-456f-95a6-92729c338a05
    streams:
      - id: >-
          logstash/metrics-logstash.stack_monitoring.node-b78cdf23-6f82-456f-95a6-92729c338a05
        data_stream:
          dataset: logstash.stack_monitoring.node
          type: metrics
        metricsets:
          - node
        hosts:
          - 'http://localhost:9600'
        period: 10s
        ssl: null
      - id: >-
          logstash/metrics-logstash.stack_monitoring.node_stats-b78cdf23-6f82-456f-95a6-92729c338a05
        data_stream:
          dataset: logstash.stack_monitoring.node_stats
          type: metrics
        metricsets:
          - node_stats
        hosts:
          - 'http://localhost:9600'
        period: 10s
        ssl: null

After Upgrade (No changes)

inputs:
  - id: logfile-logstash-b78cdf23-6f82-456f-95a6-92729c338a05
    name: logstash-1
    revision: 2
    type: logfile
    use_output: default
    meta:
      package:
        name: logstash
        version: 2.3.0-preview1
    data_stream:
      namespace: default
    package_policy_id: b78cdf23-6f82-456f-95a6-92729c338a05
    streams:
      - id: logfile-logstash.log-b78cdf23-6f82-456f-95a6-92729c338a05
        data_stream:
          dataset: logstash.log
          type: logs
        exclude_files:
          - .gz$
        paths:
          - /var/log/logstash/logstash-json*.log
        multiline:
          negate: true
          pattern: '^((\[[0-9]{4}-[0-9]{2}-[0-9]{2}[^\]]+\])|({.+}))'
          match: after
        processors:
          - add_locale.when.not.regexp.message: '^{'
          - add_fields:
              fields:
                ecs.version: 1.10.0
              target: ''
      - id: logfile-logstash.slowlog-b78cdf23-6f82-456f-95a6-92729c338a05
        data_stream:
          dataset: logstash.slowlog
          type: logs
        exclude_files:
          - .gz$
        paths:
          - /var/log/logstash/logstash-slowlog-json*.log
        processors:
          - add_locale.when.not.regexp.message: '^{'
          - add_fields:
              fields:
                ecs.version: 1.10.0
              target: ''
  - id: logstash/metrics-logstash-b78cdf23-6f82-456f-95a6-92729c338a05
    name: logstash-1
    revision: 2
    type: logstash/metrics
    use_output: default
    meta:
      package:
        name: logstash
        version: 2.3.0-preview1
    data_stream:
      namespace: default
    package_policy_id: b78cdf23-6f82-456f-95a6-92729c338a05
    streams:
      - id: >-
          logstash/metrics-logstash.stack_monitoring.node-b78cdf23-6f82-456f-95a6-92729c338a05
        data_stream:
          dataset: logstash.stack_monitoring.node
          type: metrics
        period: 10s
        hosts:
          - 'http://localhost:9600'
        metricsets:
          - node
        ssl: null
      - id: >-
          logstash/metrics-logstash.stack_monitoring.node_stats-b78cdf23-6f82-456f-95a6-92729c338a05
        data_stream:
          dataset: logstash.stack_monitoring.node_stats
          type: metrics
        period: 10s
        hosts:
          - 'http://localhost:9600'
        metricsets:
          - node_stats
        ssl: null

After Upgrade (Conditions Added)

inputs:
  - id: logfile-logstash-b78cdf23-6f82-456f-95a6-92729c338a05
    name: logstash-1
    revision: 3
    type: logfile
    use_output: default
    meta:
      package:
        name: logstash
        version: 2.3.0-preview1
    data_stream:
      namespace: default
    package_policy_id: b78cdf23-6f82-456f-95a6-92729c338a05
    streams:
      - id: logfile-logstash.log-b78cdf23-6f82-456f-95a6-92729c338a05
        data_stream:
          dataset: logstash.log
          type: logs
        exclude_files:
          - .gz$
        condition: '${kubernetes.label} == ''logstash'''
        paths:
          - /var/log/logstash/logstash-json*.log
        multiline:
          negate: true
          pattern: '^((\[[0-9]{4}-[0-9]{2}-[0-9]{2}[^\]]+\])|({.+}))'
          match: after
        processors:
          - add_locale.when.not.regexp.message: '^{'
          - add_fields:
              fields:
                ecs.version: 1.10.0
              target: ''
      - id: logfile-logstash.slowlog-b78cdf23-6f82-456f-95a6-92729c338a05
        data_stream:
          dataset: logstash.slowlog
          type: logs
        exclude_files:
          - .gz$
        condition: '${kubernetes.label} == ''logstash'''
        paths:
          - /var/log/logstash/logstash-slowlog-json*.log
        processors:
          - add_locale.when.not.regexp.message: '^{'
          - add_fields:
              fields:
                ecs.version: 1.10.0
              target: ''
  - id: logstash/metrics-logstash-b78cdf23-6f82-456f-95a6-92729c338a05
    name: logstash-1
    revision: 3
    type: logstash/metrics
    use_output: default
    meta:
      package:
        name: logstash
        version: 2.3.0-preview1
    data_stream:
      namespace: default
    package_policy_id: b78cdf23-6f82-456f-95a6-92729c338a05
    streams:
      - id: >-
          logstash/metrics-logstash.stack_monitoring.node-b78cdf23-6f82-456f-95a6-92729c338a05
        data_stream:
          dataset: logstash.stack_monitoring.node
          type: metrics
        period: 10s
        condition: '${kubernetes.label} == ''logstash'''
        hosts:
          - 'http://localhost:9600'
        metricsets:
          - node
        ssl: null
      - id: >-
          logstash/metrics-logstash.stack_monitoring.node_stats-b78cdf23-6f82-456f-95a6-92729c338a05
        data_stream:
          dataset: logstash.stack_monitoring.node_stats
          type: metrics
        period: 10s
        condition: '${kubernetes.label} == ''logstash'''
        hosts:
          - 'http://localhost:9600'
        metricsets:
          - node_stats
        ssl: null

How to test this PR locally

1. Install current integration version
2. Upgrade integration to this PR
3. Add/Remove condition values.

Related issues

• Relates [Elasticsearch] Add condition and leader election to Elasticsearch integration #5353, [Kibana] Add condition configuration for Kibana logs and metrics #5359

Screenshots

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-03-20T17:13:50.200+0000

• Duration: 17 min 43 sec

Test stats 🧪

Test	Results
Failed	0
Passed	25
Skipped	0
Total	25

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[klacabane]

LGTM, thanks for the change

[klacabane]

/test

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (2/2)	💚
Files	100.0% (6/6)	💚 3.216
Classes	100.0% (6/6)	💚 3.216
Methods	100.0% (46/46)	💚 8.913
Lines	87.558% (190/217)	👎 -4.497
Conditionals	100.0% (0/0)	💚

[elasticmachine]

Package logstash - 2.3.0-preview1 containing this change is available at https://epr.elastic.co/search?package=logstash