[Kibana] Add `condition` configuration for Kibana logs and metrics #5359

BenB196 · 2023-02-22T17:56:50Z

Enhancement

What does this PR do?

Add the ability to set the condition option to both Kibana Logs and Metrics inputs

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

condition is a top level of both inputs as I couldn't think of a reason where a user would want conditions to be different based on what is being collected within an input, and this greatly simplifies things for users.
The examples below have version numbers that don't directly match this PR, it was the only way I could get this integration to show up in Kibana for testing purposes.

Before Upgrade

inputs:
  - id: logfile-kibana-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
    name: kibana-1
    revision: 1
    type: logfile
    use_output: default
    meta:
      package:
        name: kibana
        version: 2.3.0
    data_stream:
      namespace: default
    package_policy_id: 8db829a3-4ae0-4bad-8851-4b652c5c0bf0
    streams:
      - id: logfile-kibana.audit-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
        data_stream:
          dataset: kibana.audit
          type: logs
        paths:
          - /var/log/kibana/*_audit.json
        exclude_files:
          - .gz$
      - id: logfile-kibana.log-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
        data_stream:
          dataset: kibana.log
          type: logs
        paths:
          - /var/log/kibana/kibana.stdout
        exclude_files:
          - .gz$
  - id: kibana/metrics-kibana-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
    name: kibana-1
    revision: 1
    type: kibana/metrics
    use_output: default
    meta:
      package:
        name: kibana
        version: 2.3.0
    data_stream:
      namespace: default
    package_policy_id: 8db829a3-4ae0-4bad-8851-4b652c5c0bf0
    streams:
      - id: >-
          kibana/metrics-kibana.stack_monitoring.cluster_actions-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
        data_stream:
          dataset: kibana.stack_monitoring.cluster_actions
          type: metrics
        metricsets:
          - cluster_actions
        hosts:
          - 'http://localhost:5601'
        period: 10s
        ssl: null
      - id: >-
          kibana/metrics-kibana.stack_monitoring.cluster_rules-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
        data_stream:
          dataset: kibana.stack_monitoring.cluster_rules
          type: metrics
        metricsets:
          - cluster_rules
        hosts:
          - 'http://localhost:5601'
        period: 10s
        ssl: null
      - id: >-
          kibana/metrics-kibana.stack_monitoring.node_actions-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
        data_stream:
          dataset: kibana.stack_monitoring.node_actions
          type: metrics
        metricsets:
          - node_actions
        hosts:
          - 'http://localhost:5601'
        period: 10s
        ssl: null
      - id: >-
          kibana/metrics-kibana.stack_monitoring.node_rules-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
        data_stream:
          dataset: kibana.stack_monitoring.node_rules
          type: metrics
        metricsets:
          - node_rules
        hosts:
          - 'http://localhost:5601'
        period: 10s
        ssl: null
      - id: >-
          kibana/metrics-kibana.stack_monitoring.stats-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
        data_stream:
          dataset: kibana.stack_monitoring.stats
          type: metrics
        metricsets:
          - stats
        hosts:
          - 'http://localhost:5601'
        period: 10s
        ssl: null
      - id: >-
          kibana/metrics-kibana.stack_monitoring.status-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
        data_stream:
          dataset: kibana.stack_monitoring.status
          type: metrics
        metricsets:
          - status
        hosts:
          - 'http://localhost:5601'
        period: 10s
        ssl: null

After Upgrade (No changes)

inputs:
  - id: logfile-kibana-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
    name: kibana-1
    revision: 2
    type: logfile
    use_output: default
    meta:
      package:
        name: kibana
        version: 2.4.0
    data_stream:
      namespace: default
    package_policy_id: 8db829a3-4ae0-4bad-8851-4b652c5c0bf0
    streams:
      - id: logfile-kibana.audit-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
        data_stream:
          dataset: kibana.audit
          type: logs
        exclude_files:
          - .gz$
        paths:
          - /var/log/kibana/*_audit.json
      - id: logfile-kibana.log-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
        data_stream:
          dataset: kibana.log
          type: logs
        exclude_files:
          - .gz$
        paths:
          - /var/log/kibana/kibana.stdout
  - id: kibana/metrics-kibana-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
    name: kibana-1
    revision: 2
    type: kibana/metrics
    use_output: default
    meta:
      package:
        name: kibana
        version: 2.4.0
    data_stream:
      namespace: default
    package_policy_id: 8db829a3-4ae0-4bad-8851-4b652c5c0bf0
    streams:
      - id: >-
          kibana/metrics-kibana.stack_monitoring.cluster_actions-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
        data_stream:
          dataset: kibana.stack_monitoring.cluster_actions
          type: metrics
        period: 10s
        hosts:
          - 'http://localhost:5601'
        metricsets:
          - cluster_actions
        ssl: null
      - id: >-
          kibana/metrics-kibana.stack_monitoring.cluster_rules-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
        data_stream:
          dataset: kibana.stack_monitoring.cluster_rules
          type: metrics
        period: 10s
        hosts:
          - 'http://localhost:5601'
        metricsets:
          - cluster_rules
        ssl: null
      - id: >-
          kibana/metrics-kibana.stack_monitoring.node_actions-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
        data_stream:
          dataset: kibana.stack_monitoring.node_actions
          type: metrics
        period: 10s
        hosts:
          - 'http://localhost:5601'
        metricsets:
          - node_actions
        ssl: null
      - id: >-
          kibana/metrics-kibana.stack_monitoring.node_rules-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
        data_stream:
          dataset: kibana.stack_monitoring.node_rules
          type: metrics
        period: 10s
        hosts:
          - 'http://localhost:5601'
        metricsets:
          - node_rules
        ssl: null
      - id: >-
          kibana/metrics-kibana.stack_monitoring.stats-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
        data_stream:
          dataset: kibana.stack_monitoring.stats
          type: metrics
        period: 10s
        hosts:
          - 'http://localhost:5601'
        metricsets:
          - stats
        ssl: null
      - id: >-
          kibana/metrics-kibana.stack_monitoring.status-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
        data_stream:
          dataset: kibana.stack_monitoring.status
          type: metrics
        period: 10s
        hosts:
          - 'http://localhost:5601'
        metricsets:
          - status
        ssl: null

After Upgrade (Conditions Added)

inputs:
  - id: logfile-kibana-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
    name: kibana-1
    revision: 3
    type: logfile
    use_output: default
    meta:
      package:
        name: kibana
        version: 2.4.0
    data_stream:
      namespace: default
    package_policy_id: 8db829a3-4ae0-4bad-8851-4b652c5c0bf0
    streams:
      - id: logfile-kibana.audit-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
        data_stream:
          dataset: kibana.audit
          type: logs
        exclude_files:
          - .gz$
        condition: k8s.labels == 'abc'
        paths:
          - /var/log/kibana/*_audit.json
      - id: logfile-kibana.log-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
        data_stream:
          dataset: kibana.log
          type: logs
        exclude_files:
          - .gz$
        condition: k8s.labels == 'abc'
        paths:
          - /var/log/kibana/kibana.stdout
  - id: kibana/metrics-kibana-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
    name: kibana-1
    revision: 3
    type: kibana/metrics
    use_output: default
    meta:
      package:
        name: kibana
        version: 2.4.0
    data_stream:
      namespace: default
    package_policy_id: 8db829a3-4ae0-4bad-8851-4b652c5c0bf0
    streams:
      - id: >-
          kibana/metrics-kibana.stack_monitoring.cluster_actions-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
        data_stream:
          dataset: kibana.stack_monitoring.cluster_actions
          type: metrics
        period: 10s
        condition: k8s.labels == 'abc'
        hosts:
          - 'http://localhost:5601'
        metricsets:
          - cluster_actions
        ssl: null
      - id: >-
          kibana/metrics-kibana.stack_monitoring.cluster_rules-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
        data_stream:
          dataset: kibana.stack_monitoring.cluster_rules
          type: metrics
        period: 10s
        condition: k8s.labels == 'abc'
        hosts:
          - 'http://localhost:5601'
        metricsets:
          - cluster_rules
        ssl: null
      - id: >-
          kibana/metrics-kibana.stack_monitoring.node_actions-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
        data_stream:
          dataset: kibana.stack_monitoring.node_actions
          type: metrics
        period: 10s
        condition: k8s.labels == 'abc'
        hosts:
          - 'http://localhost:5601'
        metricsets:
          - node_actions
        ssl: null
      - id: >-
          kibana/metrics-kibana.stack_monitoring.node_rules-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
        data_stream:
          dataset: kibana.stack_monitoring.node_rules
          type: metrics
        period: 10s
        condition: k8s.labels == 'abc'
        hosts:
          - 'http://localhost:5601'
        metricsets:
          - node_rules
        ssl: null
      - id: >-
          kibana/metrics-kibana.stack_monitoring.stats-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
        data_stream:
          dataset: kibana.stack_monitoring.stats
          type: metrics
        period: 10s
        condition: k8s.labels == 'abc'
        hosts:
          - 'http://localhost:5601'
        metricsets:
          - stats
        ssl: null
      - id: >-
          kibana/metrics-kibana.stack_monitoring.status-8db829a3-4ae0-4bad-8851-4b652c5c0bf0
        data_stream:
          dataset: kibana.stack_monitoring.status
          type: metrics
        period: 10s
        condition: k8s.labels == 'abc'
        hosts:
          - 'http://localhost:5601'
        metricsets:
          - status
        ssl: null

How to test this PR locally

Install current integration version
Upgrade integration to this PR
Add/Remove condition values.

Related issues

Relates [Elasticsearch] Add condition and leader election to Elasticsearch integration #5353

Screenshots

elasticmachine · 2023-02-22T18:00:12Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-03-06T17:14:25.446+0000
Duration: 29 min 57 sec

Test stats 🧪

Test	Results
Failed	0
Passed	28
Skipped	0
Total	28

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

packages/kibana/manifest.yml

klacabane · 2023-03-06T16:04:26Z

/test

elasticmachine · 2023-03-06T16:33:58Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`2/2`)	💚
Files	100.0% (`4/4`)	💚
Classes	100.0% (`4/4`)	💚
Methods	94.872% (`37/39`)	👍 69.872
Lines	87.234% (`82/94`)	👎 -12.766
Conditionals	100.0% (`0/0`)	💚

Co-authored-by: Kevin Lacabane <[email protected]>

klacabane

LGTM, thank you for the change

klacabane · 2023-03-06T17:14:12Z

/test

elasticmachine · 2023-03-07T14:37:26Z

Package kibana - 2.3.0-preview1 containing this change is available at https://epr.elastic.co/search?package=kibana

…lastic#5359) * Add `condition` configuration for logs and metrics * Update changelog.yml * Update packages/kibana/manifest.yml Co-authored-by: Kevin Lacabane <[email protected]> * Update packages/kibana/manifest.yml Co-authored-by: Kevin Lacabane <[email protected]> --------- Co-authored-by: Kevin Lacabane <[email protected]>

Add condition configuration for logs and metrics

fe801d4

BenB196 requested a review from a team as a code owner February 22, 2023 17:56

Update changelog.yml

de60ab3

BenB196 changed the title ~~Add condition configuration for logs and metrics~~ Add condition configuration for Kibana logs and metrics Feb 22, 2023

BenB196 changed the title ~~Add condition configuration for Kibana logs and metrics~~ [Kibana] Add condition configuration for Kibana logs and metrics Feb 22, 2023

klacabane reviewed Mar 6, 2023

View reviewed changes

packages/kibana/manifest.yml Outdated Show resolved Hide resolved

klacabane reviewed Mar 6, 2023

View reviewed changes

packages/kibana/manifest.yml Outdated Show resolved Hide resolved

klacabane assigned BenB196 Mar 6, 2023

klacabane added Integration:kibana Kibana Team:Infra Monitoring UI - DEPRECATED Label for the Infrastructure Monitoring UI team. - DEPRECATED - Use Team:obs-ux-infra_services >enhancement labels Mar 6, 2023

BenB196 and others added 2 commits March 6, 2023 11:45

Update packages/kibana/manifest.yml

a303f1b

Co-authored-by: Kevin Lacabane <[email protected]>

Update packages/kibana/manifest.yml

0594e6d

Co-authored-by: Kevin Lacabane <[email protected]>

klacabane approved these changes Mar 6, 2023

View reviewed changes

klacabane merged commit 69aa2ac into elastic:main Mar 7, 2023

BenB196 mentioned this pull request Mar 16, 2023

[Logstash] Add ability to set condition for logs and metrics. #5557

Merged

4 tasks

BenB196 deleted the add-kibana-condition branch November 29, 2023 12:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Kibana] Add `condition` configuration for Kibana logs and metrics #5359

[Kibana] Add `condition` configuration for Kibana logs and metrics #5359

BenB196 commented Feb 22, 2023 •

edited

Loading

elasticmachine commented Feb 22, 2023 •

edited

Loading

Build stats

Test stats 🧪

klacabane commented Mar 6, 2023

elasticmachine commented Mar 6, 2023

klacabane left a comment

klacabane commented Mar 6, 2023

elasticmachine commented Mar 7, 2023

[Kibana] Add condition configuration for Kibana logs and metrics #5359

[Kibana] Add condition configuration for Kibana logs and metrics #5359

Conversation

BenB196 commented Feb 22, 2023 • edited Loading

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Feb 22, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

klacabane commented Mar 6, 2023

elasticmachine commented Mar 6, 2023

🌐 Coverage report

klacabane left a comment

Choose a reason for hiding this comment

klacabane commented Mar 6, 2023

elasticmachine commented Mar 7, 2023

[Kibana] Add `condition` configuration for Kibana logs and metrics #5359

[Kibana] Add `condition` configuration for Kibana logs and metrics #5359

BenB196 commented Feb 22, 2023 •

edited

Loading

elasticmachine commented Feb 22, 2023 •

edited

Loading