Metricbeat: Add mapping for docker metrics per cpu #6843

jsoriano · 2018-04-12T15:21:13Z

ruflin · 2018-04-13T05:46:38Z

This will probably not work. You have to use dynamic templates for this: https://www.elastic.co/guide/en/elasticsearch/reference/6.2/dynamic-templates.html

So far the way we use dynamic templates is that we specify it as following:

    - name: labels
      type: object
      object_type: keyword
      description: >
        Image labels.

In the template this outputs:

        {
          "docker.image.labels": {
            "mapping": {
              "type": "keyword"
            },
            "match_mapping_type": "string",
            "path_match": "docker.image.labels.*"
          }
        },

I could be that for this case an extension to our template generation is needed. Best have a look at libbeat/template/processor.go and create a test in processor_test.go to see if you get the expected output.

houndci-bot · 2018-04-13T17:14:31Z

libbeat/template/processor.go

@@ -242,12 +258,16 @@ func addDynamicTemplate(f *common.Field, properties common.MapStr, matchType str
 	if len(f.Path) > 0 {
 		path = f.Path + "."
 	}
+	path_match := path + f.Name
+	if !strings.ContainsRune(path_match, '*') {
+		path_match += ".*"


don't use underscores in Go names; var path_match should be pathMatch

houndci-bot · 2018-04-13T17:14:31Z

libbeat/template/processor.go

@@ -242,12 +258,16 @@ func addDynamicTemplate(f *common.Field, properties common.MapStr, matchType str
 	if len(f.Path) > 0 {
 		path = f.Path + "."
 	}
+	path_match := path + f.Name


don't use underscores in Go names; var path_match should be pathMatch

jsoriano · 2018-04-13T17:20:32Z

Trying to generate the dynamic mapping, still have to test it for real.

ruflin · 2018-04-16T06:59:45Z

libbeat/template/processor.go

@@ -208,6 +209,21 @@ func (p *Processor) object(f *common.Field) common.MapStr {
 	}

 	switch f.ObjectType {
+	case "scaled_float":


Probably we will have a request for float and integer here pretty soon. As soon this pops up we should refactor the code to remove the duplicated parts, if possible

Just saw further below that long we already have. float should be rather simple to add in the same manner.

For scaled float I could reuse the existing scaledFloat method.

We can still clean up the code in a follow up PR.

ruflin · 2018-04-16T07:02:30Z

Change looks good to, looking forward to your manual testing. The template can be exported with metricbeat export template.

jsoriano · 2018-04-16T15:18:05Z

libbeat/template/processor.go

@@ -208,6 +209,9 @@ func (p *Processor) object(f *common.Field) common.MapStr {
 	}

 	switch f.ObjectType {
+	case "scaled_float":
+		dynProperties = p.scaledFloat(f)
+		addDynamicTemplate(f, dynProperties, matchType("*"))


I use * by default because we want it to match at least all numeric types.

jsoriano · 2018-04-16T16:30:43Z

jenkins, test it again please, it seems that there has been an issue with pypi

jsoriano · 2018-04-16T16:36:35Z

jenkins, test it

jsoriano · 2018-04-16T16:39:47Z

jenkins, test this again

ruflin

Did you manage to test this locally? Code LGTM. PR will need need a rebase because of the CHANGELOG.

ruflin · 2018-04-17T07:19:18Z

libbeat/template/processor.go

@@ -208,6 +209,21 @@ func (p *Processor) object(f *common.Field) common.MapStr {
 	}

 	switch f.ObjectType {
+	case "scaled_float":


We can still clean up the code in a follow up PR.

jsoriano · 2018-04-18T08:45:28Z

Yes, I could test it locally, it works fine :)

Branch rebased.

ruflin · 2018-04-18T13:46:00Z

CHANGELOG.asciidoc

@@ -79,6 +79,8 @@ https://github.com/elastic/beats/compare/v6.0.0-beta2...master[Check the HEAD di
 - Fix Kubernetes calculated fields store. {pull}6564{6564}
 - Exclude bind mounts in fsstat and filesystem metricsets. {pull}6819[6819]
 - Don't stop Metricbeat if aerospike server is down. {pull}6874[6874]
+- Fix Kubernetes calculated fields store. {pull}6564[6564]


Seems like that line of changelog sneaked in?

Umm, I don't know why this happens :? removing it

rebase with 3 way merges. Unfortunately a common issue.

ctindel · 2018-04-19T12:16:18Z

@ruflin Will this go into any of the 6.x releases as well?

ruflin · 2018-04-19T13:23:47Z

@ctindel yes, 6.3

jsoriano added module review Metricbeat Metricbeat labels Apr 12, 2018

jsoriano requested a review from ruflin April 12, 2018 15:21

jsoriano force-pushed the docker-per-cpu-fields-mapping branch from 2e3ea14 to 2025f9f Compare April 12, 2018 15:23

jsoriano added the in progress Pull request is currently in progress. label Apr 13, 2018

jsoriano force-pushed the docker-per-cpu-fields-mapping branch from 2025f9f to b0fcb07 Compare April 13, 2018 17:14

houndci-bot reviewed Apr 13, 2018

View reviewed changes

jsoriano force-pushed the docker-per-cpu-fields-mapping branch 3 times, most recently from 11b7a99 to e144525 Compare April 13, 2018 17:19

jsoriano force-pushed the docker-per-cpu-fields-mapping branch from e144525 to 80a4e3c Compare April 13, 2018 17:28

ruflin reviewed Apr 16, 2018

View reviewed changes

jsoriano force-pushed the docker-per-cpu-fields-mapping branch 3 times, most recently from 74fe717 to 333aa1c Compare April 16, 2018 15:09

jsoriano commented Apr 16, 2018

View reviewed changes

ruflin reviewed Apr 17, 2018

View reviewed changes

jsoriano force-pushed the docker-per-cpu-fields-mapping branch from 333aa1c to 9553a5e Compare April 18, 2018 08:42

jsoriano removed the in progress Pull request is currently in progress. label Apr 18, 2018

ruflin reviewed Apr 18, 2018

View reviewed changes

jsoriano added 2 commits April 19, 2018 00:59

Metricbeat: Add mapping for docker metrics per cpu

ddca573

Add dynamic mapping for scaled_float type

b4ee1da

jsoriano force-pushed the docker-per-cpu-fields-mapping branch from 9553a5e to b4ee1da Compare April 18, 2018 22:59

ruflin merged commit 8e29a8b into elastic:master Apr 19, 2018

graphaelli mentioned this pull request May 7, 2018

Marks: First measurement decides if field is mapped to long or float elastic/apm-server#708

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Metricbeat: Add mapping for docker metrics per cpu #6843

Metricbeat: Add mapping for docker metrics per cpu #6843

jsoriano commented Apr 12, 2018

ruflin commented Apr 13, 2018

houndci-bot Apr 13, 2018

houndci-bot Apr 13, 2018

jsoriano commented Apr 13, 2018

ruflin Apr 16, 2018

ruflin Apr 16, 2018

jsoriano Apr 16, 2018

ruflin Apr 17, 2018

ruflin commented Apr 16, 2018

jsoriano Apr 16, 2018 •

edited

Loading

jsoriano commented Apr 16, 2018

jsoriano commented Apr 16, 2018

jsoriano commented Apr 16, 2018

ruflin left a comment

ruflin Apr 17, 2018

jsoriano commented Apr 18, 2018

ruflin Apr 18, 2018

jsoriano Apr 18, 2018

ruflin Apr 18, 2018

ctindel commented Apr 19, 2018

ruflin commented Apr 19, 2018

Metricbeat: Add mapping for docker metrics per cpu #6843

Metricbeat: Add mapping for docker metrics per cpu #6843

Conversation

jsoriano commented Apr 12, 2018

ruflin commented Apr 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsoriano commented Apr 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented Apr 16, 2018

jsoriano Apr 16, 2018 • edited Loading

Choose a reason for hiding this comment

jsoriano commented Apr 16, 2018

jsoriano commented Apr 16, 2018

jsoriano commented Apr 16, 2018

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsoriano commented Apr 18, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ctindel commented Apr 19, 2018

ruflin commented Apr 19, 2018

jsoriano Apr 16, 2018 •

edited

Loading