[Elasticsearch] Add condition and leader election to Elasticsearch integration #5353

BenB196 · 2023-02-22T01:42:47Z

Enhancement

What does this PR do?

Add the ability to set condition to both the Logs and Metrics sections
Add the ability to set leader election on the Metrics section

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

condition is a top level of both inputs as I couldn't think of a reason where a user would want conditions to be different based on what is being collected within an input, and this greatly simplifies things for users.
leaderelection was added to the metrics input, because if you want to use the cluster scope, you'd want only one Elastic Agent performing the collection
condition and leaderelection are both under the advanced sections, as I'm not sure how many people would actually use this (I don't know how many users actually run Elasticsearch on Kubernetes)

Integration Policy Before Upgrade

inputs:
  - id: logfile-elasticsearch-3bed89b0-3bff-40e6-95c2-43f6d203d7db
    name: elasticsearch-1
    revision: 1
    type: logfile
    use_output: default
    meta:
      package:
        name: elasticsearch
        version: 1.2.0
    data_stream:
      namespace: default
    package_policy_id: 3bed89b0-3bff-40e6-95c2-43f6d203d7db
    streams:
      - id: logfile-elasticsearch.audit-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.audit
          type: logs
        paths:
          - /var/log/elasticsearch/*_audit.json
        exclude_files:
          - .gz$
        processors:
          - add_locale: null
          - add_fields:
              target: ''
              fields:
                ecs.version: 1.10.0
          - decode_json_fields:
              fields:
                - message
              target: _json
          - rename:
              fields:
                - from: _json.request.body
                  to: _request
              ignore_missing: true
          - drop_fields:
              fields:
                - _json
          - detect_mime_type:
              field: _request
              target: http.request.mime_type
          - drop_fields:
              fields:
                - _request
              ignore_missing: true
      - id: logfile-elasticsearch.deprecation-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.deprecation
          type: logs
        paths:
          - /var/log/elasticsearch/*_deprecation.json
        exclude_files:
          - .gz$
          - _slowlog.log$
          - _access.log$
      - id: logfile-elasticsearch.gc-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.gc
          type: logs
        paths:
          - '/var/log/elasticsearch/gc.log.[0-9]*'
          - /var/log/elasticsearch/gc.log
        exclude_files:
          - .gz$
        exclude_lines:
          - '^(OpenJDK|Java HotSpot).* Server VM '
          - '^CommandLine flags: '
          - '^Memory: '
          - '^{'
        multiline:
          pattern: '^(\[?[0-9]{4}-[0-9]{2}-[0-9]{2}|{)'
          negate: true
          match: after
        processors:
          - add_fields:
              target: ''
              fields:
                ecs.version: 1.10.0
      - id: logfile-elasticsearch.server-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.server
          type: logs
        paths:
          - /var/log/elasticsearch/*_server.json
        exclude_files:
          - .gz$
          - _slowlog.log$
          - _access.log$
          - _deprecation.log$
      - id: logfile-elasticsearch.slowlog-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.slowlog
          type: logs
        paths:
          - /var/log/elasticsearch/*_index_search_slowlog.json
          - /var/log/elasticsearch/*_index_indexing_slowlog.json
        exclude_files:
          - .gz$
  - id: elasticsearch/metrics-elasticsearch-3bed89b0-3bff-40e6-95c2-43f6d203d7db
    name: elasticsearch-1
    revision: 1
    type: elasticsearch/metrics
    use_output: default
    meta:
      package:
        name: elasticsearch
        version: 1.2.0
    data_stream:
      namespace: default
    package_policy_id: 3bed89b0-3bff-40e6-95c2-43f6d203d7db
    streams:
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.ccr-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.ccr
          type: metrics
        metricsets:
          - ccr
        hosts:
          - 'http://localhost:9200'
        scope: node
        period: 10s
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.cluster_stats-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.cluster_stats
          type: metrics
        metricsets:
          - cluster_stats
        hosts:
          - 'http://localhost:9200'
        scope: node
        period: 10s
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.enrich-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.enrich
          type: metrics
        metricsets:
          - enrich
        hosts:
          - 'http://localhost:9200'
        scope: node
        period: 10s
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.index-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.index
          type: metrics
        metricsets:
          - index
        hosts:
          - 'http://localhost:9200'
        scope: node
        period: 10s
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.index_recovery-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.index_recovery
          type: metrics
        metricsets:
          - index_recovery
        hosts:
          - 'http://localhost:9200'
        scope: node
        period: 10s
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.index_summary-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.index_summary
          type: metrics
        metricsets:
          - index_summary
        hosts:
          - 'http://localhost:9200'
        scope: node
        period: 10s
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.ml_job-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.ml_job
          type: metrics
        metricsets:
          - ml_job
        hosts:
          - 'http://localhost:9200'
        scope: node
        period: 10s
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.node-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.node
          type: metrics
        metricsets:
          - node
        hosts:
          - 'http://localhost:9200'
        scope: node
        period: 10s
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.node_stats-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.node_stats
          type: metrics
        metricsets:
          - node_stats
        hosts:
          - 'http://localhost:9200'
        scope: node
        period: 10s
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.pending_tasks-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.pending_tasks
          type: metrics
        metricsets:
          - pending_tasks
        hosts:
          - 'http://localhost:9200'
        scope: node
        period: 10s
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.shard-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.shard
          type: metrics
        metricsets:
          - shard
        hosts:
          - 'http://localhost:9200'
        scope: node
        period: 10s
        ssl: null

Integration Policy After Upgrade

inputs:
  - id: logfile-elasticsearch-3bed89b0-3bff-40e6-95c2-43f6d203d7db
    name: elasticsearch-1
    revision: 2
    type: logfile
    use_output: default
    meta:
      package:
        name: elasticsearch
        version: 1.3.0
    data_stream:
      namespace: default
    package_policy_id: 3bed89b0-3bff-40e6-95c2-43f6d203d7db
    streams:
      - id: logfile-elasticsearch.audit-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.audit
          type: logs
        exclude_files:
          - .gz$
        paths:
          - /var/log/elasticsearch/*_audit.json
        processors:
          - add_locale: null
          - add_fields:
              fields:
                ecs.version: 1.10.0
              target: ''
          - decode_json_fields:
              fields:
                - message
              target: _json
          - rename:
              ignore_missing: true
              fields:
                - from: _json.request.body
                  to: _request
          - drop_fields:
              fields:
                - _json
          - detect_mime_type:
              field: _request
              target: http.request.mime_type
          - drop_fields:
              ignore_missing: true
              fields:
                - _request
      - id: logfile-elasticsearch.deprecation-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.deprecation
          type: logs
        exclude_files:
          - .gz$
          - _slowlog.log$
          - _access.log$
        paths:
          - /var/log/elasticsearch/*_deprecation.json
      - id: logfile-elasticsearch.gc-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.gc
          type: logs
        exclude_files:
          - .gz$
        paths:
          - '/var/log/elasticsearch/gc.log.[0-9]*'
          - /var/log/elasticsearch/gc.log
        multiline:
          negate: true
          pattern: '^(\[?[0-9]{4}-[0-9]{2}-[0-9]{2}|{)'
          match: after
        processors:
          - add_fields:
              fields:
                ecs.version: 1.10.0
              target: ''
        exclude_lines:
          - '^(OpenJDK|Java HotSpot).* Server VM '
          - '^CommandLine flags: '
          - '^Memory: '
          - '^{'
      - id: logfile-elasticsearch.server-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.server
          type: logs
        exclude_files:
          - .gz$
          - _slowlog.log$
          - _access.log$
          - _deprecation.log$
        paths:
          - /var/log/elasticsearch/*_server.json
      - id: logfile-elasticsearch.slowlog-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.slowlog
          type: logs
        exclude_files:
          - .gz$
        paths:
          - /var/log/elasticsearch/*_index_search_slowlog.json
          - /var/log/elasticsearch/*_index_indexing_slowlog.json
  - id: elasticsearch/metrics-elasticsearch-3bed89b0-3bff-40e6-95c2-43f6d203d7db
    name: elasticsearch-1
    revision: 2
    type: elasticsearch/metrics
    use_output: default
    meta:
      package:
        name: elasticsearch
        version: 1.3.0
    data_stream:
      namespace: default
    package_policy_id: 3bed89b0-3bff-40e6-95c2-43f6d203d7db
    streams:
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.ccr-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.ccr
          type: metrics
        period: 10s
        hosts:
          - 'http://localhost:9200'
        scope: node
        metricsets:
          - ccr
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.cluster_stats-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.cluster_stats
          type: metrics
        period: 10s
        hosts:
          - 'http://localhost:9200'
        scope: node
        metricsets:
          - cluster_stats
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.enrich-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.enrich
          type: metrics
        period: 10s
        hosts:
          - 'http://localhost:9200'
        scope: node
        metricsets:
          - enrich
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.index-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.index
          type: metrics
        period: 10s
        hosts:
          - 'http://localhost:9200'
        scope: node
        metricsets:
          - index
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.index_recovery-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.index_recovery
          type: metrics
        period: 10s
        hosts:
          - 'http://localhost:9200'
        scope: node
        metricsets:
          - index_recovery
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.index_summary-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.index_summary
          type: metrics
        period: 10s
        hosts:
          - 'http://localhost:9200'
        scope: node
        metricsets:
          - index_summary
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.ml_job-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.ml_job
          type: metrics
        period: 10s
        hosts:
          - 'http://localhost:9200'
        scope: node
        metricsets:
          - ml_job
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.node-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.node
          type: metrics
        period: 10s
        hosts:
          - 'http://localhost:9200'
        scope: node
        metricsets:
          - node
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.node_stats-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.node_stats
          type: metrics
        period: 10s
        hosts:
          - 'http://localhost:9200'
        scope: node
        metricsets:
          - node_stats
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.pending_tasks-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.pending_tasks
          type: metrics
        period: 10s
        hosts:
          - 'http://localhost:9200'
        scope: node
        metricsets:
          - pending_tasks
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.shard-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.shard
          type: metrics
        period: 10s
        hosts:
          - 'http://localhost:9200'
        scope: node
        metricsets:
          - shard
        ssl: null

Integration Policy After Upgrade & Adding Conditions/Leader Election

inputs:
  - id: logfile-elasticsearch-3bed89b0-3bff-40e6-95c2-43f6d203d7db
    name: elasticsearch-1
    revision: 3
    type: logfile
    use_output: default
    meta:
      package:
        name: elasticsearch
        version: 1.3.0
    data_stream:
      namespace: default
    package_policy_id: 3bed89b0-3bff-40e6-95c2-43f6d203d7db
    streams:
      - id: logfile-elasticsearch.audit-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.audit
          type: logs
        condition: k8s.labels == abc
        exclude_files:
          - .gz$
        paths:
          - /var/log/elasticsearch/*_audit.json
        processors:
          - add_locale: null
          - add_fields:
              fields:
                ecs.version: 1.10.0
              target: ''
          - decode_json_fields:
              fields:
                - message
              target: _json
          - rename:
              ignore_missing: true
              fields:
                - from: _json.request.body
                  to: _request
          - drop_fields:
              fields:
                - _json
          - detect_mime_type:
              field: _request
              target: http.request.mime_type
          - drop_fields:
              ignore_missing: true
              fields:
                - _request
      - id: logfile-elasticsearch.deprecation-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.deprecation
          type: logs
        condition: k8s.labels == abc
        exclude_files:
          - .gz$
          - _slowlog.log$
          - _access.log$
        paths:
          - /var/log/elasticsearch/*_deprecation.json
      - id: logfile-elasticsearch.gc-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.gc
          type: logs
        condition: k8s.labels == abc
        exclude_files:
          - .gz$
        paths:
          - '/var/log/elasticsearch/gc.log.[0-9]*'
          - /var/log/elasticsearch/gc.log
        multiline:
          negate: true
          pattern: '^(\[?[0-9]{4}-[0-9]{2}-[0-9]{2}|{)'
          match: after
        processors:
          - add_fields:
              fields:
                ecs.version: 1.10.0
              target: ''
        exclude_lines:
          - '^(OpenJDK|Java HotSpot).* Server VM '
          - '^CommandLine flags: '
          - '^Memory: '
          - '^{'
      - id: logfile-elasticsearch.server-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.server
          type: logs
        condition: k8s.labels == abc
        exclude_files:
          - .gz$
          - _slowlog.log$
          - _access.log$
          - _deprecation.log$
        paths:
          - /var/log/elasticsearch/*_server.json
      - id: logfile-elasticsearch.slowlog-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.slowlog
          type: logs
        condition: k8s.labels == abc
        exclude_files:
          - .gz$
        paths:
          - /var/log/elasticsearch/*_index_search_slowlog.json
          - /var/log/elasticsearch/*_index_indexing_slowlog.json
  - id: elasticsearch/metrics-elasticsearch-3bed89b0-3bff-40e6-95c2-43f6d203d7db
    name: elasticsearch-1
    revision: 3
    type: elasticsearch/metrics
    use_output: default
    meta:
      package:
        name: elasticsearch
        version: 1.3.0
    data_stream:
      namespace: default
    package_policy_id: 3bed89b0-3bff-40e6-95c2-43f6d203d7db
    streams:
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.ccr-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.ccr
          type: metrics
        period: 10s
        condition: '${kubernetes_leaderelection.leader} == true and k8s.label == abc'
        hosts:
          - 'http://localhost:9200'
        scope: node
        metricsets:
          - ccr
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.cluster_stats-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.cluster_stats
          type: metrics
        period: 10s
        condition: '${kubernetes_leaderelection.leader} == true and k8s.label == abc'
        hosts:
          - 'http://localhost:9200'
        scope: node
        metricsets:
          - cluster_stats
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.enrich-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.enrich
          type: metrics
        period: 10s
        condition: '${kubernetes_leaderelection.leader} == true and k8s.label == abc'
        hosts:
          - 'http://localhost:9200'
        scope: node
        metricsets:
          - enrich
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.index-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.index
          type: metrics
        period: 10s
        condition: '${kubernetes_leaderelection.leader} == true and k8s.label == abc'
        hosts:
          - 'http://localhost:9200'
        scope: node
        metricsets:
          - index
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.index_recovery-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.index_recovery
          type: metrics
        period: 10s
        condition: '${kubernetes_leaderelection.leader} == true and k8s.label == abc'
        hosts:
          - 'http://localhost:9200'
        scope: node
        metricsets:
          - index_recovery
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.index_summary-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.index_summary
          type: metrics
        period: 10s
        condition: '${kubernetes_leaderelection.leader} == true and k8s.label == abc'
        hosts:
          - 'http://localhost:9200'
        scope: node
        metricsets:
          - index_summary
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.ml_job-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.ml_job
          type: metrics
        period: 10s
        condition: '${kubernetes_leaderelection.leader} == true and k8s.label == abc'
        hosts:
          - 'http://localhost:9200'
        scope: node
        metricsets:
          - ml_job
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.node-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.node
          type: metrics
        period: 10s
        condition: '${kubernetes_leaderelection.leader} == true and k8s.label == abc'
        hosts:
          - 'http://localhost:9200'
        scope: node
        metricsets:
          - node
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.node_stats-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.node_stats
          type: metrics
        period: 10s
        condition: '${kubernetes_leaderelection.leader} == true and k8s.label == abc'
        hosts:
          - 'http://localhost:9200'
        scope: node
        metricsets:
          - node_stats
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.pending_tasks-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.pending_tasks
          type: metrics
        period: 10s
        condition: '${kubernetes_leaderelection.leader} == true and k8s.label == abc'
        hosts:
          - 'http://localhost:9200'
        scope: node
        metricsets:
          - pending_tasks
        ssl: null
      - id: >-
          elasticsearch/metrics-elasticsearch.stack_monitoring.shard-3bed89b0-3bff-40e6-95c2-43f6d203d7db
        data_stream:
          dataset: elasticsearch.stack_monitoring.shard
          type: metrics
        period: 10s
        condition: '${kubernetes_leaderelection.leader} == true and k8s.label == abc'
        hosts:
          - 'http://localhost:9200'
        scope: node
        metricsets:
          - shard
        ssl: null

How to test this PR locally

Install currently released Elasticsearch integration (1.2.0 as of writing this PR)
Upgrade to 1.3.0
Observe new fields
Test adding & removing contents from the fields

Related issues

Closes Elasticsearch Integration - Support setting condition #5319

Screenshots

elasticmachine · 2023-02-22T01:45:57Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-02-23T16:02:18.725+0000
Duration: 30 min 24 sec

Test stats 🧪

Test	Results
Failed	0
Passed	55
Skipped	0
Total	55

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

joshdover · 2023-02-23T09:31:03Z

If I merge #4597 before this enhancement, this will only be available for 8.7+ since my PR requires newer versions of Agent. I lean towards merging mine first since it's been open for a while but I can give this one up to the EOW to be get reviewed and merged.

I will trigger CI on this to get things moving.

@miltonhultgren @klacabane it would be good to get your feedback on whether or not we want this enhancement. It seems reasonable to me but you all are the experts.

joshdover · 2023-02-23T13:26:06Z

/test

elasticmachine · 2023-02-23T13:59:03Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`5/5`)	💚
Files	100.0% (`9/9`)	💚
Classes	100.0% (`9/9`)	💚
Methods	88.073% (`96/109`)	👎 -4.351
Lines	91.98% (`562/611`)	👎 -3.771
Conditionals	100.0% (`0/0`)	💚

klacabane · 2023-02-23T15:10:15Z

packages/elasticsearch/manifest.yml

+          - name: leaderelection
+            type: bool
+            title: Leader Election
+            description: Enable leaderelection between a set of Elastic Agents running on Kubernetes. Useful for when scope is `cluster`.


Would it make sense to link to this documentation ?

Think that doc only applies to Standalone deployments, I believe for fleet managed deployments, this automatically happens. (For the most part the wording was copy/pasted from other integrations that implement Leader Election, can be added though if desired)

packages/elasticsearch/manifest.yml

klacabane

LGTM

klacabane · 2023-02-23T16:02:04Z

/test

elasticmachine · 2023-02-23T17:47:56Z

Package elasticsearch - 1.3.0 containing this change is available at https://epr.elastic.co/search?package=elasticsearch

elasticmachine · 2023-02-23T17:47:58Z

Package elasticsearch - 1.3.0 containing this change is available at https://epr.elastic.co/search?package=elasticsearch

…tegration (elastic#5353) * Add condition and leader election to elasticsearch integration * Update changelog link * Update condition description wording

Add condition and leader election to elasticsearch integration

eaf6d87

BenB196 requested a review from a team as a code owner February 22, 2023 01:42

Update changelog link

66903ac

BenB196 mentioned this pull request Feb 22, 2023

[Kibana] Add condition configuration for Kibana logs and metrics #5359

Merged

4 tasks

BenB196 changed the title ~~Add condition and leader election to Elasticsearch integration~~ [Elasticsearch] Add condition and leader election to Elasticsearch integration Feb 22, 2023

klacabane added enhancement New feature or request Integration:elasticsearch Elasticsearch Team:Infra Monitoring UI - DEPRECATED Label for the Infrastructure Monitoring UI team. - DEPRECATED - Use Team:obs-ux-infra_services labels Feb 23, 2023

klacabane assigned BenB196 Feb 23, 2023

klacabane mentioned this pull request Feb 23, 2023

Add ingest_pipeline data streams and dashboards to Elasticsearch package #4597

Merged

20 tasks

klacabane self-requested a review February 23, 2023 14:16

klacabane reviewed Feb 23, 2023

View reviewed changes

Update condition description wording

4c30612

klacabane approved these changes Feb 23, 2023

View reviewed changes

klacabane merged commit 795e97f into elastic:main Feb 23, 2023

BenB196 deleted the add-conditions-elasticsearch branch February 23, 2023 17:37

BenB196 mentioned this pull request Mar 16, 2023

[Logstash] Add ability to set condition for logs and metrics. #5557

Merged

4 tasks

klacabane mentioned this pull request Apr 24, 2023

Add ssl support to elasticsearch ingest pipeline datastream #5939

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Elasticsearch] Add condition and leader election to Elasticsearch integration #5353

[Elasticsearch] Add condition and leader election to Elasticsearch integration #5353

BenB196 commented Feb 22, 2023 •

edited

Loading

elasticmachine commented Feb 22, 2023 •

edited

Loading

Build stats

Test stats 🧪

joshdover commented Feb 23, 2023

joshdover commented Feb 23, 2023

elasticmachine commented Feb 23, 2023 •

edited

Loading

klacabane Feb 23, 2023

BenB196 Feb 23, 2023

klacabane left a comment

klacabane commented Feb 23, 2023

elasticmachine commented Feb 23, 2023

elasticmachine commented Feb 23, 2023

[Elasticsearch] Add condition and leader election to Elasticsearch integration #5353

[Elasticsearch] Add condition and leader election to Elasticsearch integration #5353

Conversation

BenB196 commented Feb 22, 2023 • edited Loading

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Feb 22, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

joshdover commented Feb 23, 2023

joshdover commented Feb 23, 2023

elasticmachine commented Feb 23, 2023 • edited Loading

🌐 Coverage report

klacabane Feb 23, 2023

Choose a reason for hiding this comment

BenB196 Feb 23, 2023

Choose a reason for hiding this comment

klacabane left a comment

Choose a reason for hiding this comment

klacabane commented Feb 23, 2023

elasticmachine commented Feb 23, 2023

elasticmachine commented Feb 23, 2023

BenB196 commented Feb 22, 2023 •

edited

Loading

elasticmachine commented Feb 22, 2023 •

edited

Loading

elasticmachine commented Feb 23, 2023 •

edited

Loading