Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Elasticsearch] Add condition and leader election to Elasticsearch integration #5353

Merged
merged 3 commits into from
Feb 23, 2023
Merged

[Elasticsearch] Add condition and leader election to Elasticsearch integration #5353

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
eaf6d87
Add condition and leader election to elasticsearch integration
BenB196 Feb 22, 2023
66903ac
Update changelog link
BenB196 Feb 22, 2023
4c30612
Update condition description wording
BenB196 Feb 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions packages/elasticsearch/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,12 @@
# newer versions go on top
- version: "1.3.0"
changes:
- description: Add conditional support to Elasticsearch log and metrics inputs
type: enhancement
link: https://github.com/elastic/integrations/issues/5353
- description: Add leader election support to Elasticsearch inputs metrics inputs
type: enhancement
link: https://github.com/elastic/integrations/issues/5353
- version: "1.2.0"
changes:
- description: Add period variable to define polling frequency
Expand Down
3 changes: 3 additions & 0 deletions packages/elasticsearch/data_stream/audit/agent/stream/log.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@ paths:
{{#each paths as |path i|}}
- {{path}}
{{/each}}
{{#if condition}}
condition: {{ condition }}
{{/if}}
exclude_files: [".gz$"]
processors:
- add_locale: ~
Expand Down
11 changes: 11 additions & 0 deletions packages/elasticsearch/data_stream/ccr/agent/stream/stream.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,15 @@ password: {{password}}
period: {{period}}
{{#if ssl}}
ssl: {{ssl}}
{{/if}}
{{#if leaderelection }}
{{#if condition }}
condition: ${kubernetes_leaderelection.leader} == true and {{ condition }}
{{ else }}
condition: ${kubernetes_leaderelection.leader} == true
{{/if}}
{{ else }}
{{#if condition }}
condition: {{ condition }}
{{/if}}
{{/if}}
11 changes: 11 additions & 0 deletions packages/elasticsearch/data_stream/cluster_stats/agent/stream/stream.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,15 @@ password: {{password}}
period: {{period}}
{{#if ssl}}
ssl: {{ssl}}
{{/if}}
{{#if leaderelection }}
{{#if condition }}
condition: ${kubernetes_leaderelection.leader} == true and {{ condition }}
{{ else }}
condition: ${kubernetes_leaderelection.leader} == true
{{/if}}
{{ else }}
{{#if condition }}
condition: {{ condition }}
{{/if}}
{{/if}}
3 changes: 3 additions & 0 deletions packages/elasticsearch/data_stream/deprecation/agent/stream/log.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,7 @@ paths:
{{#each paths as |path i|}}
- {{path}}
{{/each}}
{{#if condition}}
condition: {{ condition }}
{{/if}}
exclude_files: [".gz$","_slowlog.log$","_access.log$"]
11 changes: 11 additions & 0 deletions packages/elasticsearch/data_stream/enrich/agent/stream/stream.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,15 @@ password: {{password}}
period: {{period}}
{{#if ssl}}
ssl: {{ssl}}
{{/if}}
{{#if leaderelection }}
{{#if condition }}
condition: ${kubernetes_leaderelection.leader} == true and {{ condition }}
{{ else }}
condition: ${kubernetes_leaderelection.leader} == true
{{/if}}
{{ else }}
{{#if condition }}
condition: {{ condition }}
{{/if}}
{{/if}}
3 changes: 3 additions & 0 deletions packages/elasticsearch/data_stream/gc/agent/stream/log.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@ paths:
{{#each paths as |path i|}}
- {{path}}
{{/each}}
{{#if condition}}
condition: {{ condition }}
{{/if}}
exclude_files: [".gz$"]
exclude_lines: ["^(OpenJDK|Java HotSpot).* Server VM ", "^CommandLine flags: ", "^Memory: ", "^{"] # exclude JVM8 banner and JSON
multiline:
Expand Down
11 changes: 11 additions & 0 deletions packages/elasticsearch/data_stream/index/agent/stream/stream.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,15 @@ password: {{password}}
period: {{period}}
{{#if ssl}}
ssl: {{ssl}}
{{/if}}
{{#if leaderelection }}
{{#if condition }}
condition: ${kubernetes_leaderelection.leader} == true and {{ condition }}
{{ else }}
condition: ${kubernetes_leaderelection.leader} == true
{{/if}}
{{ else }}
{{#if condition }}
condition: {{ condition }}
{{/if}}
{{/if}}
11 changes: 11 additions & 0 deletions packages/elasticsearch/data_stream/index_recovery/agent/stream/stream.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,15 @@ password: {{password}}
period: {{period}}
{{#if ssl}}
ssl: {{ssl}}
{{/if}}
{{#if leaderelection }}
{{#if condition }}
condition: ${kubernetes_leaderelection.leader} == true and {{ condition }}
{{ else }}
condition: ${kubernetes_leaderelection.leader} == true
{{/if}}
{{ else }}
{{#if condition }}
condition: {{ condition }}
{{/if}}
{{/if}}
11 changes: 11 additions & 0 deletions packages/elasticsearch/data_stream/index_summary/agent/stream/stream.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,15 @@ password: {{password}}
period: {{period}}
{{#if ssl}}
ssl: {{ssl}}
{{/if}}
{{#if leaderelection }}
{{#if condition }}
condition: ${kubernetes_leaderelection.leader} == true and {{ condition }}
{{ else }}
condition: ${kubernetes_leaderelection.leader} == true
{{/if}}
{{ else }}
{{#if condition }}
condition: {{ condition }}
{{/if}}
{{/if}}
11 changes: 11 additions & 0 deletions packages/elasticsearch/data_stream/ml_job/agent/stream/stream.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,15 @@ password: {{password}}
period: {{period}}
{{#if ssl}}
ssl: {{ssl}}
{{/if}}
{{#if leaderelection }}
{{#if condition }}
condition: ${kubernetes_leaderelection.leader} == true and {{ condition }}
{{ else }}
condition: ${kubernetes_leaderelection.leader} == true
{{/if}}
{{ else }}
{{#if condition }}
condition: {{ condition }}
{{/if}}
{{/if}}
11 changes: 11 additions & 0 deletions packages/elasticsearch/data_stream/node/agent/stream/stream.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,15 @@ password: {{password}}
period: {{period}}
{{#if ssl}}
ssl: {{ssl}}
{{/if}}
{{#if leaderelection }}
{{#if condition }}
condition: ${kubernetes_leaderelection.leader} == true and {{ condition }}
{{ else }}
condition: ${kubernetes_leaderelection.leader} == true
{{/if}}
{{ else }}
{{#if condition }}
condition: {{ condition }}
{{/if}}
{{/if}}
11 changes: 11 additions & 0 deletions packages/elasticsearch/data_stream/node_stats/agent/stream/stream.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,15 @@ password: {{password}}
period: {{period}}
{{#if ssl}}
ssl: {{ssl}}
{{/if}}
{{#if leaderelection }}
{{#if condition }}
condition: ${kubernetes_leaderelection.leader} == true and {{ condition }}
{{ else }}
condition: ${kubernetes_leaderelection.leader} == true
{{/if}}
{{ else }}
{{#if condition }}
condition: {{ condition }}
{{/if}}
{{/if}}
11 changes: 11 additions & 0 deletions packages/elasticsearch/data_stream/pending_tasks/agent/stream/stream.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,15 @@ password: {{password}}
period: {{period}}
{{#if ssl}}
ssl: {{ssl}}
{{/if}}
{{#if leaderelection }}
{{#if condition }}
condition: ${kubernetes_leaderelection.leader} == true and {{ condition }}
{{ else }}
condition: ${kubernetes_leaderelection.leader} == true
{{/if}}
{{ else }}
{{#if condition }}
condition: {{ condition }}
{{/if}}
{{/if}}
3 changes: 3 additions & 0 deletions packages/elasticsearch/data_stream/server/agent/stream/log.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,7 @@ paths:
{{#each paths as |path i|}}
- {{path}}
{{/each}}
{{#if condition}}
condition: {{ condition }}
{{/if}}
exclude_files: [".gz$","_slowlog.log$","_access.log$","_deprecation.log$"]
11 changes: 11 additions & 0 deletions packages/elasticsearch/data_stream/shard/agent/stream/stream.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,15 @@ password: {{password}}
period: {{period}}
{{#if ssl}}
ssl: {{ssl}}
{{/if}}
{{#if leaderelection }}
{{#if condition }}
condition: ${kubernetes_leaderelection.leader} == true and {{ condition }}
{{ else }}
condition: ${kubernetes_leaderelection.leader} == true
{{/if}}
{{ else }}
{{#if condition }}
condition: {{ condition }}
{{/if}}
{{/if}}
3 changes: 3 additions & 0 deletions packages/elasticsearch/data_stream/slowlog/agent/stream/log.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,7 @@ paths:
{{#each paths as |path i|}}
- {{path}}
{{/each}}
{{#if condition}}
condition: {{ condition }}
{{/if}}
exclude_files: [".gz$"]
25 changes: 24 additions & 1 deletion packages/elasticsearch/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name: elasticsearch
title: Elasticsearch
version: 1.2.0
version: 1.3.0
description: Elasticsearch Integration
type: integration
icons:
Expand All @@ -21,6 +21,14 @@ policy_templates:
- type: logfile
title: Collect Elasticsearch logs
description: "Collecting audit, deprecation, gc, server and slowlog logs from Elasticsearch instances (input: logfile)"
vars:
- name: condition
title: Condition
description: Condition to filter when to collect this input
type: text
multi: false
required: false
show_user: false
- type: elasticsearch/metrics
title: Collect Elasticsearch metrics
description: Collect Elasticsearch metrics about indices, CCR, cluster stats, machine learning or node statistics
Expand Down Expand Up @@ -67,5 +75,20 @@ policy_templates:
#certificate_authorities: ["/etc/ca.crt"]
#certificate: "/etc/client.crt"
#key: "/etc/client.key"
- name: leaderelection
type: bool
title: Leader Election
description: Enable leaderelection between a set of Elastic Agents running on Kubernetes. Useful for when scope is `cluster`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to link to this documentation ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think that doc only applies to Standalone deployments, I believe for fleet managed deployments, this automatically happens. (For the most part the wording was copy/pasted from other integrations that implement Leader Election, can be added though if desired)

multi: false
required: true
show_user: false
default: false
- name: condition
title: Condition
description: Condition to filter when to collect this input
type: text
multi: false
required: false
show_user: false
owner:
github: elastic/infra-monitoring-ui