[Metricbeat][Kubernetes] Extend state_node with more conditions #23905

ioandr · 2021-02-08T11:18:12Z

What does this PR do?

This PR extends the state_node module of the Metricbeat's Kubernetes module with more status conditions, such as MemoryPressure, DiskPressure, OutOfDisk and PIDPressure.

Refactor the opFilter Prometheus metric helper to allow the declaration of multiple Metricbeat fields from a single Prometheus metric, based on allowed label values.

Why is it important?

Adding more status conditions of Kubernetes nodes is useful for monitoring and alerting purposes as it allows one to use conditions like:

Is any node running out of memory?
Is any node running out of disk?
Is any node running out of PIDs?

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
~~I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Closes [Metricbeat] Add support for node conditions to state_node metricset #18049

Use cases

Screenshots

Logs

2021-02-08T12:34:38.470+0200    DEBUG   [processors]    processing/processors.go:203    Publish event: {                                                                                                                                      
  "@timestamp": "2021-02-08T10:34:38.307Z",                                                                                                                                                                                                   
  "@metadata": {                                                                                                                                                                                                                              
    "beat": "metricbeat",                                                                                                                                                                                                                     
    "type": "_doc",                                                                                                                                                                                                                           
    "version": "8.0.0"                                                                                                                                                                                                                        
  },                                                                                                                                                                                                                                          
  "event": {                                                                                                                                                                                                                                  
    "module": "kubernetes",                                                                                                                                                                                                                   
    "duration": 162891130,                                                                                                                                                                                                                    
    "dataset": "kubernetes.node"                                                                                                                                                                                                              
  },                                                                                                                                                                                                                                          
  "kubernetes": {                                                                                                                                                                                                                             
    "node": {                                                                                                                                                                                                                                 
      "name": "node1",                                                                                                                                                                                                                      
      "status": {                                                                                                                                                                                                                             
        "unschedulable": false,                                                                                                                                                                                                               
        "memorypressure": "false",                                                                                                                                                                                                            
        "diskpressure": "false",                                                                                                                                                                                                              
        "pidpressure": "false",                                                                                                                                                                                                               
        "ready": "true"                                                                                                                                                                                                                       
      }                                                                                                                                                                                                                                       
    }                                                                                                                                                                                                                                         
  },                                                                                                                                                                                                                                          
  "ecs": {                                                                                                                                                                                                                                    
    "version": "1.7.0"                                                                                                                                                                                                                        
  },  
...

elasticmachine · 2021-02-08T11:25:10Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #23905 updated
Start Time: 2021-02-16T11:32:32.416+0000
Duration: 53 min 32 sec
Commit: e5019eb

Test stats 🧪

Test	Results
Failed	0
Passed	8658
Skipped	2122
Total	10780

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	8658
Skipped	2122
Total	10780

botelastic · 2021-02-08T11:29:40Z

This pull request doesn't have a Team:<team> label.

ChrsMark · 2021-02-08T11:33:24Z

/test

elasticmachine · 2021-02-08T21:34:15Z

Pinging @elastic/integrations (Team:Integrations)

ChrsMark · 2021-02-09T08:37:22Z

/test

ChrsMark

Thank you for working on this @ioandr! LGTM

ChrsMark · 2021-02-09T08:40:08Z

@jsoriano do you want to have a look too?

jsoriano

Nice 👌

metricbeat/module/kubernetes/state_node/_meta/fields.yml

metricbeat/module/kubernetes/state_node/_meta/test/ksm.v1.8.0.expected

…3905) Signed-off-by: Ioannis Androulidakis <[email protected]>

…astic#23905) Signed-off-by: Ioannis Androulidakis <[email protected]>

jsoriano · 2021-02-16T11:26:15Z

There are some linting failures, @ioandr try to run mage update inside the metricbeat and x-pack/metricbeat directories, and commit the changes.

Signed-off-by: chrismark <[email protected]>

ChrsMark · 2021-02-16T11:40:08Z

Did the update so as to unblock it, @ioandr no need to run it again. Let's see if CI is happy and we can merge this one.

…tic#23905) (cherry picked from commit 125c29d)

…-arm * upstream/master: [Metricbeat][Kubernetes] Extend state_node with more conditions (elastic#23905) [CI] googleStorageUploadExt step (elastic#24048) Check fields are documented for aws metricsets (elastic#23887) Update go-concert to 0.1.0 (elastic#23770) [Libbeat][New Processor] XML Decode (elastic#23678) Fix: bad substitution of API key (elastic#24036) [Filebeat] Add Pensando DFW Module (elastic#21063) [Filebeat] Check if processor is supported by ES version (elastic#23763) Syslog system tests: be more forgiving (elastic#24021)

…) (#24057) (cherry picked from commit 125c29d) Co-authored-by: Ioannis Androulidakis <[email protected]>

…dows-7 * upstream/master: (332 commits) Use ECS v1.8.0 (elastic#24086) Add support for postgresql csv logs (elastic#23334) [Heartbeat] Refactor config system (elastic#23467) [CI] install docker-compose with retry (elastic#24069) Add nodes to filebeat-kubernetes.yaml ClusterRole - fixes elastic#24051 (elastic#24052) updating manifest files for filebeat threatintel module (elastic#24074) Add Zeek Signatures (elastic#23772) Update Beats to ECS 1.8.0 (elastic#23465) Support running Docker logging plugin on ARM64 (elastic#24034) Fix ec2 metricset fields.yml and add integration test (elastic#23726) Only build targz and zip versions of Beats if PACKAGES is set in agent (elastic#24060) [Filebeat] Add field definitions for known Netflow/IPFIX vendor fields (elastic#23773) [Elastic Agent] Enroll with Fleet Server (elastic#23865) [Filebeat] Convert logstash logEvent.action objects to strings (elastic#23944) [Ingest Management] Fix reloading of log level for services (elastic#24055) Add Agent standalone k8s manifest (elastic#23679) [Metricbeat][Kubernetes] Extend state_node with more conditions (elastic#23905) [CI] googleStorageUploadExt step (elastic#24048) Check fields are documented for aws metricsets (elastic#23887) Update go-concert to 0.1.0 (elastic#23770) ...

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Feb 8, 2021

ChrsMark assigned ChrsMark and ioandr Feb 8, 2021

ChrsMark self-requested a review February 8, 2021 11:29

ChrsMark removed their assignment Feb 8, 2021

ChrsMark added the Team:Integrations Label for the Integrations team label Feb 8, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Feb 8, 2021

ChrsMark added needs_team Indicates that the issue/PR needs a Team:* label review labels Feb 8, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Feb 8, 2021

ChrsMark added kubernetes Enable builds in the CI for kubernetes v7.12.0 labels Feb 8, 2021

ioandr force-pushed the feature-state-node-conditions branch from 71482ce to 21f4b0e Compare February 8, 2021 19:44

ioandr force-pushed the feature-state-node-conditions branch from 21f4b0e to 848efcc Compare February 8, 2021 21:08

ioandr force-pushed the feature-state-node-conditions branch from 7cf36a7 to 3695892 Compare February 8, 2021 21:14

ioandr force-pushed the feature-state-node-conditions branch from 3695892 to cf8a815 Compare February 8, 2021 21:17

ioandr force-pushed the feature-state-node-conditions branch from 4017267 to 72ca845 Compare February 8, 2021 21:22

ioandr force-pushed the feature-state-node-conditions branch from 72ca845 to f648d86 Compare February 8, 2021 21:33

ioandr marked this pull request as ready for review February 8, 2021 21:34

ChrsMark approved these changes Feb 9, 2021

View reviewed changes

ChrsMark added the test-plan Add this PR to be manual test plan label Feb 9, 2021

jsoriano added the needs_backport PR is waiting to be backported to other branches. label Feb 15, 2021

jsoriano reviewed Feb 15, 2021

View reviewed changes

metricbeat/module/kubernetes/state_node/_meta/fields.yml Outdated Show resolved Hide resolved

metricbeat/module/kubernetes/state_node/_meta/test/ksm.v1.8.0.expected Show resolved Hide resolved

ioandr added 2 commits February 16, 2021 12:25

[Metricbeat][Kubernetes] Extend state_node with conditions (elastic#2…

33d7406

…3905) Signed-off-by: Ioannis Androulidakis <[email protected]>

fixup! [Metricbeat][Kubernetes] Extend state_node with conditions (el…

98cd81b

…astic#23905) Signed-off-by: Ioannis Androulidakis <[email protected]>

ioandr force-pushed the feature-state-node-conditions branch from f648d86 to 98cd81b Compare February 16, 2021 10:44

jsoriano assigned ChrsMark Feb 16, 2021

mage update

e5019eb

Signed-off-by: chrismark <[email protected]>

ChrsMark merged commit 125c29d into elastic:master Feb 16, 2021

ChrsMark mentioned this pull request Feb 16, 2021

Cherry-pick #23905 to 7.x: [Metricbeat][Kubernetes] Extend state_node with more conditions #24057

Merged

12 tasks

ChrsMark pushed a commit to ChrsMark/beats that referenced this pull request Feb 16, 2021

[Metricbeat][Kubernetes] Extend state_node with more conditions (elas…

85359a3

…tic#23905) (cherry picked from commit 125c29d)

ChrsMark removed the needs_backport PR is waiting to be backported to other branches. label Feb 16, 2021

ChrsMark added a commit that referenced this pull request Feb 16, 2021

[Metricbeat][Kubernetes] Extend state_node with more conditions (#23905…

7085358

…) (#24057) (cherry picked from commit 125c29d) Co-authored-by: Ioannis Androulidakis <[email protected]>

This was referenced Feb 25, 2021

Fix state_node conditions ' types #24241

Merged

Cherry-pick #24241 to 7.12: Fix state_node conditions ' types #24244

Merged

Cherry-pick #24241 to 7.x: Fix state_node conditions ' types #24245

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Metricbeat][Kubernetes] Extend state_node with more conditions #23905

[Metricbeat][Kubernetes] Extend state_node with more conditions #23905

ioandr commented Feb 8, 2021 •

edited

Loading

elasticmachine commented Feb 8, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

botelastic bot commented Feb 8, 2021

ChrsMark commented Feb 8, 2021

elasticmachine commented Feb 8, 2021

ChrsMark commented Feb 9, 2021

ChrsMark left a comment

ChrsMark commented Feb 9, 2021

jsoriano left a comment

jsoriano commented Feb 16, 2021

ChrsMark commented Feb 16, 2021

[Metricbeat][Kubernetes] Extend state_node with more conditions #23905

[Metricbeat][Kubernetes] Extend state_node with more conditions #23905

Conversation

ioandr commented Feb 8, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

Related issues

Use cases

Screenshots

Logs

elasticmachine commented Feb 8, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

botelastic bot commented Feb 8, 2021

ChrsMark commented Feb 8, 2021

elasticmachine commented Feb 8, 2021

ChrsMark commented Feb 9, 2021

ChrsMark left a comment

Choose a reason for hiding this comment

ChrsMark commented Feb 9, 2021

jsoriano left a comment

Choose a reason for hiding this comment

jsoriano commented Feb 16, 2021

ChrsMark commented Feb 16, 2021

ioandr commented Feb 8, 2021 •

edited

Loading

elasticmachine commented Feb 8, 2021 •

edited by jenkins-beats-ci bot

Loading