Coredns metricbeat module #10585

ChrsMark · 2019-02-05T19:11:03Z

This PR implements CoreDNS module described on #10432.

Co-Authored-By: Ioannis Androulidakis [email protected]

elasticmachine · 2019-02-05T19:11:05Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

metricbeat/module/coredns/_meta/fields.yml

metricbeat/module/coredns/stats/stats.go

elasticcla · 2019-02-10T15:05:08Z

Hi @ChrsMark, we have found your signature in our records, but it seems like you have signed with a different e-mail than the one used in yout Git commit. Can you please add both of these e-mails into your Github profile (they can be hidden), so we can match your e-mails to your Github profile?

ChrsMark · 2019-02-11T07:22:39Z

@ruflin , could you have a look on this, especially on fields declaration? We follow the apiserver's metricset approach (from Kuberentes module), however it was a little bit confusing the fact that every "fetch" is split into multiple events, according to the label matching. This leads into not being able to provide a sample data.json though.
Any thoughts on this?

ruflin · 2019-02-12T12:43:51Z

For the data.json there is also a generator method that you can pass a condition, see here: https://github.com/elastic/beats/blob/master/metricbeat/mb/testing/data_generator.go#L66 This should help to generate a "nicer" data.json.

I think the label matching is expected but @jsoriano might be a better person to comment on this.

ruflin

Looks already pretty good:

Changelog?
Add a system check for field documentation (I think it's off at the moment)
How is the data.json generated at the moment

Glad to see you could use the prometheus metricset wrapper.

metricbeat/module/coredns/stats/_meta/fields.yml

jsoriano · 2019-02-13T16:00:08Z

You can find examples of conditional events writting in rabbitmq and system socket metricsets.

ioandr · 2019-02-19T12:42:29Z

Hey @ruflin, @jsoriano! We have added almost everything except for the unit-test. We will also need to curate the metric fields a bit in order to reach the desired results.

As far as the unit-test is concerned, is there any automated way to produce the metrics.expected used in the assertion?

@exekias might know more on this since he had done the same for apiserver in the kubernetes metricbeat module

ioandr · 2019-02-22T09:13:12Z

Hey @ruflin, @jsoriano! We have added almost everything except for the unit-test. We will also need to curate the metric fields a bit in order to reach the desired results.

As far as the unit-test is concerned, is there any automated way to produce the metrics.expected used in the assertion?

@exekias might know more on this since he had done the same for apiserver in the kubernetes metricbeat module

We got around the issue of auto-generating the expected metrics by using the -update_expected flag in the go test command, as @ChrsMark indicated.

ioandr · 2019-02-22T09:36:39Z

Hey, just a heads-up on the current state:

we have done most of the work described in #10432 and removed WIP from the PR title, so you might want to have a second review of this when you have time.

Let us know for any further changes, as we consider some minor commits to finalize the format of the metrics and the CoreDNS module.

jsoriano

Thanks! this is looking quite good, I have added some comments, mainly about the fields mappings. Let me know what you think.

Also, do you happen to have a dashboard for this? 😇

CHANGELOG.asciidoc

CHANGELOG.next.asciidoc

metricbeat/module/coredns/_meta/Dockerfile

metricbeat/module/coredns/stats/_meta/data_request_duration_sec_event.json

metricbeat/module/coredns/stats/_meta/data_request_size_bytes_event.json

metricbeat/module/coredns/stats/_meta/test/metrics

metricbeat/module/coredns/stats/_meta/data_request_type_event.json

metricbeat/module/coredns/stats/_meta/fields.yml

ChrsMark · 2019-02-22T12:39:47Z

Thanks for reviewing @jsoriano . We will iterate over the comments soon!

As far as the dashboard is concerned, sure thing in a follow up PR!☺️

Signed-off-by: Chris Mark <[email protected]>

Signed-off-by: Chris <[email protected]> Co-Authored-By: Ioannis Androulidakis <[email protected]>

Signed-off-by: Chris Mark <[email protected]>

Signed-off-by: Ioannis Androulidakis <[email protected]>

* Enable DO bit metrics with dnssec in DNS request * Merge request and response size bytes metric * Generate updated JSON files and fix integration test * Generate updated metrics and metrics.expected files Signed-off-by: Ioannis Androulidakis <[email protected]>

Signed-off-by: Chris <[email protected]>

Signed-off-by: Chris Mark <[email protected]>

Signed-off-by: Ioannis Androulidakis <[email protected]>

Signed-off-by: Chris Mark <[email protected]>

Signed-off-by: Chris <[email protected]>

Signed-off-by: Chris Mark <[email protected]>

jsoriano · 2019-03-19T09:36:09Z

jenkins, test this

ioandr · 2019-03-19T09:41:57Z

@jsoriano thanks for triggering Jenkins again, we forced-pushed to rebase on the latest master

ChrsMark · 2019-03-19T14:50:44Z

Failing test seems to be irrelevant. (wished for a greeny Jenkins 😞 , but not this time)

jsoriano · 2019-03-19T17:37:13Z

Well, failing test looks irrelevant to this, yes, shipping it. Thanks for your work here!

ChrsMark · 2019-03-20T06:26:26Z

@jsoriano 👍 .
Would you like me to open an issue about testdata leftover? I could work on it and I have something in mind already but I would like to crosscheck the approach with one of the project's maintainers to ensure its validity. So, let me know.

ruflin · 2019-03-20T09:17:45Z

@ChrsMark Would be great. You can also directly open a PR if you already have something in mind, no need for an issue.

ChrsMark requested review from a team as code owners February 5, 2019 19:11

ChrsMark force-pushed the coredns-metricbeat-module branch from 54a01b5 to 4617c08 Compare February 5, 2019 19:27

ruflin added module review Metricbeat Metricbeat Team:Integrations Label for the Integrations team labels Feb 6, 2019

ruflin mentioned this pull request Feb 6, 2019

Add support to monitor CoreDNS #10432

Closed

9 tasks

ruflin reviewed Feb 6, 2019

View reviewed changes

metricbeat/module/coredns/_meta/fields.yml Outdated Show resolved Hide resolved

metricbeat/module/coredns/stats/stats.go Outdated Show resolved Hide resolved

ruflin reviewed Feb 12, 2019

View reviewed changes

metricbeat/module/coredns/stats/_meta/fields.yml Outdated Show resolved Hide resolved

metricbeat/module/coredns/stats/_meta/fields.yml Outdated Show resolved Hide resolved

ChrsMark force-pushed the coredns-metricbeat-module branch 4 times, most recently from 2a9534a to 8dee1bf Compare February 19, 2019 12:37

ioandr force-pushed the coredns-metricbeat-module branch from 8dee1bf to 679810a Compare February 22, 2019 08:58

ChrsMark changed the title ~~[WIP] Coredns metricbeat module~~ Coredns metricbeat module Feb 22, 2019

jsoriano reviewed Feb 22, 2019

View reviewed changes

ChrsMark force-pushed the coredns-metricbeat-module branch 4 times, most recently from 894ae81 to 46af5c0 Compare February 26, 2019 09:41

ChrsMark and others added 14 commits March 19, 2019 10:46

Add draft for datageneration

62a6e98

Signed-off-by: Chris Mark <[email protected]>

Add coredns servcie in docker-compose of metricbeat

6ff4d12

Signed-off-by: Chris Mark <[email protected]>

Fix Dockerfile & add data generation

7b970f1

Signed-off-by: Chris <[email protected]> Co-Authored-By: Ioannis Androulidakis <[email protected]>

Add system test

07f2feb

Signed-off-by: Chris Mark <[email protected]>

Add metrics.expected and update unit test

7fba267

Signed-off-by: Ioannis Androulidakis <[email protected]>

Review changes

3985f7b

Signed-off-by: Ioannis Androulidakis <[email protected]>

Update docs

63ada52

Signed-off-by: Chris <[email protected]>

Change Prometheus helper to scale double precision histogram keys

c58060f

Signed-off-by: Chris Mark <[email protected]>

Update generated JSON and metrics.expected

b1bd96c

Signed-off-by: Ioannis Androulidakis <[email protected]>

Fix scaling approach

d3e45ed

Signed-off-by: Chris Mark <[email protected]>

Fix docs and CHANGELOG

ebb8066

Signed-off-by: Chris Mark <[email protected]>

Use multiplier option of Prometheus Fields

c3c9025

Signed-off-by: Chris Mark <[email protected]>

Use the fixed Hist multiplier

2b5d62d

Signed-off-by: Chris <[email protected]>

ioandr force-pushed the coredns-metricbeat-module branch from e639a70 to 1c81377 Compare March 19, 2019 08:48

ChrsMark force-pushed the coredns-metricbeat-module branch from 1c81377 to cc34863 Compare March 19, 2019 09:16

Fix data structures

cc34863

Signed-off-by: Chris Mark <[email protected]>

jsoriano approved these changes Mar 19, 2019

View reviewed changes

jsoriano added the v7.2.0 label Mar 19, 2019

jsoriano mentioned this pull request Mar 19, 2019

Task/use etcd metrics endpoint #11280

Merged

jsoriano merged commit cb23a8e into elastic:master Mar 19, 2019

ChrsMark mentioned this pull request Mar 20, 2019

[Metricbeat] Add testdata for coredns module #11332

Merged

ChrsMark mentioned this pull request Mar 27, 2019

Add logo for CoreDNS elastic/eui#1771

Closed

ChrsMark deleted the coredns-metricbeat-module branch August 28, 2019 12:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Coredns metricbeat module #10585

Coredns metricbeat module #10585

ChrsMark commented Feb 5, 2019 •

edited

Loading

elasticmachine commented Feb 5, 2019

elasticcla commented Feb 10, 2019

ChrsMark commented Feb 11, 2019

ruflin commented Feb 12, 2019

ruflin left a comment

jsoriano commented Feb 13, 2019

ioandr commented Feb 19, 2019 •

edited

Loading

ioandr commented Feb 22, 2019

ioandr commented Feb 22, 2019 •

edited

Loading

jsoriano left a comment

ChrsMark commented Feb 22, 2019

jsoriano commented Mar 19, 2019

ioandr commented Mar 19, 2019

ChrsMark commented Mar 19, 2019 •

edited

Loading

jsoriano commented Mar 19, 2019

ChrsMark commented Mar 20, 2019

ruflin commented Mar 20, 2019

Coredns metricbeat module #10585

Coredns metricbeat module #10585

Conversation

ChrsMark commented Feb 5, 2019 • edited Loading

elasticmachine commented Feb 5, 2019

elasticcla commented Feb 10, 2019

ChrsMark commented Feb 11, 2019

ruflin commented Feb 12, 2019

ruflin left a comment

Choose a reason for hiding this comment

jsoriano commented Feb 13, 2019

ioandr commented Feb 19, 2019 • edited Loading

ioandr commented Feb 22, 2019

ioandr commented Feb 22, 2019 • edited Loading

jsoriano left a comment

Choose a reason for hiding this comment

ChrsMark commented Feb 22, 2019

jsoriano commented Mar 19, 2019

ioandr commented Mar 19, 2019

ChrsMark commented Mar 19, 2019 • edited Loading

jsoriano commented Mar 19, 2019

ChrsMark commented Mar 20, 2019

ruflin commented Mar 20, 2019

ChrsMark commented Feb 5, 2019 •

edited

Loading

ioandr commented Feb 19, 2019 •

edited

Loading

ioandr commented Feb 22, 2019 •

edited

Loading

ChrsMark commented Mar 19, 2019 •

edited

Loading