Add containerd package and cpu,memory,blkio data streams #2522

MichaelKatsoulis · 2022-01-12T13:53:17Z

What does this PR do?

After merging elastic/beats#29247 which introduced containerd module this PR
creates containerd package with cpu, memory and blkio data streams.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Bring up the elastic-stack version 8.1.0-SNAPSHOT
elastic-package up --version=8.1.0-SNAPSHOT -v -d
Bring up a Kubernetes cluster with kind and containerd runtime kind create cluster --image 'kindest/node:v1.21.1
Docker exec in the kind kubernetes node and configure containerd to server metrics. Follow instructions here.
Deploy elastic agent with the same version and enrol it to fleet.
Add containerd integration in a policy and assign it to the agent.
Watch containerd cpu, memory and blkio metrics being populated.

Related issues

Closes Create integration for containerd module #2373

Screenshots

elasticmachine · 2022-01-12T14:08:19Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-01-24T16:13:25.174+0000
Duration: 17 min 44 sec
Commit: 3418b4b

Test stats 🧪

Test	Results
Failed	0
Passed	10
Skipped	0
Total	10

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

…ation

tetianakravchenko · 2022-01-18T08:14:54Z

packages/containerd/data_stream/blkio/fields/fields.yml

+
+    - name: write.bytes
+      type: long
+      format: bytes


should units and metric_type be added as well?

for other fields as well - https://github.com/elastic/observability-dev/blob/fef856060729b1ff233774941711303a59a77b30/docs/dc/integrations/development/generic_guidelines.md#document-all-fields

tetianakravchenko · 2022-01-18T08:25:00Z

packages/containerd/data_stream/blkio/fields/ecs.yml

@@ -0,0 +1,2 @@
+- external: ecs
+  name: container.id


should dimension: true be set for this fields? as I see usually it is set on *.name field - https://github.com/elastic/integrations/blob/master/packages/kubernetes/data_stream/container/fields/base-fields.yml#L127-L128, but it is not possible to expose name, right?

the same for memory and cpu datastreams

Yes right it is not possible. But I think that container.id is the best place.

tetianakravchenko · 2022-01-18T08:27:31Z

packages/containerd/data_stream/blkio/fields/ecs.yml

@@ -0,0 +1,2 @@
+- external: ecs


in sample event there is an ecs.version, should it be added here?

actually I thought that those fields are more or less a default for each integration - https://github.com/elastic/integrations/blob/master/packages/kubernetes/data_stream/container/fields/ecs.yml, or?

It seems that some are added by default like the ecs.version, service.address and service.type. The rest are not default. For example containerd adds container.id while kubernetes adds orchestrator.cluster.name

tetianakravchenko · 2022-01-18T08:33:14Z

packages/containerd/data_stream/cpu/fields/fields.yml

+
+    - name: usage.user.ns
+      type: double
+      metric_type: gauge


unit: nanos ?
https://github.com/elastic/package-spec/blob/master/versions/1/data_stream/fields/fields.spec.yml#L78

elasticmachine · 2022-01-19T10:07:21Z

Pinging @elastic/integrations (Team:Integrations)

MichaelKatsoulis · 2022-01-19T10:26:19Z

/test

MichaelKatsoulis · 2022-01-19T12:17:55Z

@akshay-saraswat could you take a look at the screenshot of the dashboard I have created for containerd? Unfortunately we cannot extract informations from the metrics like containers' statuses which would be something nice to see in a dashboard.

ChrsMark

Overall looks good. Just 2 comments:

In the visualisations Read/Write Bytes Per Container the Y axis has too big numbers, wondering if we could somehow curate it and make it be shown like 2M instead of 2.000.000 etc. Not sure if possible through Kibana though, maybe there is a setting in the Visualisation options. Also I see that you are using Lens Line charts. I remember in the past we prefered using TSVB charts though for many reasons. @kaiyan-sheng does it happen to have more context here?
I see the package has not tests, but you can add them with a mock server already and ensure that package is tested. See https://github.com/elastic/integrations/tree/master/packages/docker/_dev/deploy/docker as an example and a past k8s version https://github.com/elastic/integrations/tree/b189a92444c4dfc33a3462fdad21b40a5bf8ae4e/packages/kubernetes/_dev/deploy/docker

MichaelKatsoulis · 2022-01-20T10:59:06Z

Overall looks good. Just 2 comments:

In the visualisations Read/Write Bytes Per Container the Y axis has too big numbers, wondering if we could somehow curate it and make it be shown like 2M instead of 2.000.000 etc. Not sure if possible through Kibana though, maybe there is a setting in the Visualisation options. Also I see that you are using Lens Line charts. I remember in the past we prefered using TSVB charts though for many reasons. @kaiyan-sheng does it happen to have more context here?

Yes we can. I had set the value type to default instead of bytes. Also I have created the same dashboard with TSVB as well. So I am good either way.

kaiyan-sheng · 2022-01-20T20:51:01Z

@ChrsMark Yep @MichaelKatsoulis got it, setting the value type to bytes will give 2M instead of 2,000,000 🙂

MichaelKatsoulis · 2022-01-24T11:47:54Z

@ChrsMark , @tetianakravchenko could do a final review? I decided to go with the TSVB dashboard. I also added mock tests.

tetianakravchenko · 2022-01-24T13:35:21Z

@MichaelKatsoulis did you update the screenshot after the latest dashboard changes?

MichaelKatsoulis · 2022-01-24T13:40:56Z

@MichaelKatsoulis did you update the screenshot after the latest dashboard changes?

Yes the screenshot is correct , just without the TSVB word in the dashboard name

tetianakravchenko

LGTM!

ChrsMark

* Add containerd package and cpu,memory,blkio data streams

Add containerd package and cpu,memory,blkio data streams

513722c

MichaelKatsoulis marked this pull request as draft January 12, 2022 13:53

MichaelKatsoulis self-assigned this Jan 12, 2022

MichaelKatsoulis added the Team:Integrations Label for the Integrations team label Jan 12, 2022

MichaelKatsoulis added 3 commits January 14, 2022 16:20

Update package

776b728

Add docs

21833ba

Merge remote-tracking branch 'upstream/master' into containerd_integr…

a7c82a3

…ation

MichaelKatsoulis requested review from ChrsMark and tetianakravchenko January 17, 2022 12:42

tetianakravchenko reviewed Jan 18, 2022

View reviewed changes

MichaelKatsoulis added 2 commits January 18, 2022 14:16

Add containerd dashboard

7645d9e

Update fields

205e659

MichaelKatsoulis marked this pull request as ready for review January 19, 2022 10:07

MichaelKatsoulis added 2 commits January 19, 2022 12:32

Update the screenshot and logo

4603696

Add blkio device field

ce1c893

ChrsMark reviewed Jan 20, 2022

View reviewed changes

MichaelKatsoulis added 3 commits January 20, 2022 13:29

Add tsvb dashboard

cf32edf

Add mock tests

1aade35

Fix endpoints

e0a9ae7

MichaelKatsoulis added 2 commits January 24, 2022 11:09

Add tests to all data streams

71d00cf

Add new line in sample file

3418b4b

tetianakravchenko approved these changes Jan 24, 2022

View reviewed changes

ChrsMark approved these changes Jan 25, 2022

View reviewed changes

MichaelKatsoulis merged commit b11ebb1 into elastic:main Jan 25, 2022

MichaelKatsoulis mentioned this pull request Jan 28, 2022

Dashboard for containerd module elastic/beats#29592

Closed

eyalkraft pushed a commit to build-security/integrations that referenced this pull request Mar 30, 2022

Add containerd package and cpu,memory,blkio data streams (elastic#2522)

1f74b0e

* Add containerd package and cpu,memory,blkio data streams

ChrsMark mentioned this pull request Sep 26, 2022

Feature/istio metrics #4253

Merged

5 tasks

andrewkroh added Integration:containerd Containerd New Integration Issue or pull request for creating a new integration package. labels Aug 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add containerd package and cpu,memory,blkio data streams #2522

Add containerd package and cpu,memory,blkio data streams #2522

MichaelKatsoulis commented Jan 12, 2022 •

edited

Loading

elasticmachine commented Jan 12, 2022 •

edited

Loading

Build stats

Test stats 🧪

tetianakravchenko Jan 18, 2022

tetianakravchenko Jan 18, 2022 •

edited

Loading

MichaelKatsoulis Jan 18, 2022

tetianakravchenko Jan 18, 2022 •

edited

Loading

MichaelKatsoulis Jan 18, 2022

tetianakravchenko Jan 18, 2022 •

edited

Loading

elasticmachine commented Jan 19, 2022

MichaelKatsoulis commented Jan 19, 2022

MichaelKatsoulis commented Jan 19, 2022

ChrsMark left a comment

MichaelKatsoulis commented Jan 20, 2022

kaiyan-sheng commented Jan 20, 2022

MichaelKatsoulis commented Jan 24, 2022

tetianakravchenko commented Jan 24, 2022

MichaelKatsoulis commented Jan 24, 2022

tetianakravchenko left a comment

ChrsMark left a comment

Add containerd package and cpu,memory,blkio data streams #2522

Add containerd package and cpu,memory,blkio data streams #2522

Conversation

MichaelKatsoulis commented Jan 12, 2022 • edited Loading

What does this PR do?

Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Jan 12, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

tetianakravchenko Jan 18, 2022

Choose a reason for hiding this comment

tetianakravchenko Jan 18, 2022 • edited Loading

Choose a reason for hiding this comment

MichaelKatsoulis Jan 18, 2022

Choose a reason for hiding this comment

tetianakravchenko Jan 18, 2022 • edited Loading

Choose a reason for hiding this comment

MichaelKatsoulis Jan 18, 2022

Choose a reason for hiding this comment

tetianakravchenko Jan 18, 2022 • edited Loading

Choose a reason for hiding this comment

elasticmachine commented Jan 19, 2022

MichaelKatsoulis commented Jan 19, 2022

MichaelKatsoulis commented Jan 19, 2022

ChrsMark left a comment

Choose a reason for hiding this comment

MichaelKatsoulis commented Jan 20, 2022

kaiyan-sheng commented Jan 20, 2022

MichaelKatsoulis commented Jan 24, 2022

tetianakravchenko commented Jan 24, 2022

MichaelKatsoulis commented Jan 24, 2022

tetianakravchenko left a comment

Choose a reason for hiding this comment

ChrsMark left a comment

Choose a reason for hiding this comment

MichaelKatsoulis commented Jan 12, 2022 •

edited

Loading

elasticmachine commented Jan 12, 2022 •

edited

Loading

tetianakravchenko Jan 18, 2022 •

edited

Loading

tetianakravchenko Jan 18, 2022 •

edited

Loading

tetianakravchenko Jan 18, 2022 •

edited

Loading