Perfmon wildcard queries #4502

martinscholz83 · 2017-06-14T09:29:35Z

Add the possibility to perform wildcard queries for instances. See #3828

karmi · 2017-06-14T09:29:36Z

Hi @maddin2016, we have found your signature in our records, but it seems like you have signed with a different e-mail than the one used in yout Git commit. Can you please add both of these e-mails into your Github profile (they can be hidden), so we can match your e-mails to your Github profile?

elasticmachine · 2017-06-14T09:29:37Z

Can one of the admins verify this patch?

elasticmachine · 2017-06-14T09:32:03Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically on build-eu-00. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

martinscholz83 · 2017-06-14T09:44:55Z

Hey @andrewkroh, @ruflin. Sorry for the delay. I added the function to perform wildcard queries. I just check if the path contains (*)\. Because a wildcard query return all instances, i decided to put them together each as an object. For example

diskio: {
    write: {
        bytes: {
            "C:": 2500.0000,
            "D": 1200.0000
            }
        }
    }

Maybe you have any other suggestions.

andrewkroh

Thanks for continuing to improve the perfmon metricset!

There is a binary file (metricbeat/debug) in the PR that I think was added accidentally.

andrewkroh · 2017-06-14T19:32:04Z

metricbeat/module/windows/perfmon/pdh_integration_windows_test.go

+	}
+	defer handle.query.Close()
+
+	values, err := handle.Read()


Why is the error not checked? If you intentially are going to ignore an error then I recommend writing the assignment as values, _ := handle.Read() to be clear that it was intentional.

andrewkroh · 2017-06-14T19:39:07Z

metricbeat/module/windows/perfmon/pdh_integration_windows_test.go

+		println(k)
+	}
+
+	if m, n := values["processor"].(common.MapStr); n {


If processor or processor.time or processors.time.pct are not found, should this be a test failure? I'm pretty sure this block could be simplified to:

pcts, err := values.GetValue("processors.time.pct`) if err != nil { t.Fatal(err) } assert.NotEmpty(t, pcts)

VS Code Intellisense wasn't working 😄 That's what i've searched for.

andrewkroh · 2017-06-14T19:41:02Z

metricbeat/module/windows/perfmon/pdh_integration_windows_test.go

+	}
+
+	for k, _ := range values {
+		println(k)


Use t.Log() so that the -v flag is honored for tests.

andrewkroh · 2017-06-14T19:46:26Z

metricbeat/module/windows/perfmon/pdh_windows.go

@@ -77,6 +80,26 @@ func PdhGetFormattedCounterValue(counter PdhCounterHandle, format PdhCounterForm
 	return counterType, &value, nil
 }

+func PdhGetFormattedCounterArray(counter PdhCounterHandle, format PdhCounterFormat) (int, *[]PdhCounterValueItem, error) {


Why not return []PdhCounterValueItem instead of a *[]PdhCounterValueItem?

Also I don't think it should return the int when that can be obtained using len() on the returned slice.

After looking at the PdhCounterValueItem and how it's used, I think it would be best to return a different struct that has the a Name field that is a string. This way the users of the method don't need to follow the pointer and convert the UTF16 bytes to UTF8.

The returned slice is larger. So we need the bufferCount value to iterate over the slice to get the items which has a value.

andrewkroh · 2017-06-14T21:08:40Z

metricbeat/module/windows/perfmon/zpdh_windows.go

+)
+
+var (
+	errERROR_IO_PENDING error = syscall.Errno(errnoERROR_IO_PENDING)


I don't see where this is used?

This is coming from go generate. I think its since go 1.8.x.

andrewkroh · 2017-06-14T21:37:18Z

Because a wildcard query return all instances, i decided to put them together each as an object.

Because the are separate instances of the same counter it seems like each instance should be its own event so that they can easily be compared to each other like apples to apples. For example you might want to create a line graph showing diskio for C, D, and E all at once. Having a data structure similar to what we have now with the non-wildcard would make this easier. But a field would need to be added specifying the instance name (like C, D, or E).

{
  "windows":{
    "perfmon":{
      "disk":{
        "name":"A",
        "bytes":{
          "read":{
            "total":0
          }
        }
      }
    }
  }
}

So maybe in the config you have something like:

instance_label: disk.name
measurement_label: disk.bytes.read.total

martinscholz83 · 2017-06-21T08:03:40Z

@andrewkroh, thanks for your comments. Based on this i have changed the logic and fixed the test. To do is to separate the instances.

martinscholz83 · 2017-06-23T07:33:50Z

Maybe we should make this config standard. Lets take the following config

perfmon.counters:
    - instance_label: processor.name
      measurement_label: processor.time.total
      query: '\Processor Information(_Total)\% Processor Time'

This would produce the following output

{
  "windows":{
    "perfmon":{
      "processor":{
        "name":"_Total",
        "time":{
            "total":0          
        }
      }
    }
  }
}

So here perfmon produce only one value _Total. Unfortunately the api doesn't return the instance name for a single instance. So we have to extract it from the query string. For wildcard queries we can get the instance name from the returned array.

What do you think about?

andrewkroh

Yes, I think having a single way to configure all the counters is more clear to the user.

andrewkroh · 2017-06-23T22:04:56Z

metricbeat/module/windows/perfmon/pdh_windows.go

+						break
+					}
+				}
+				runes := utf16.Decode(a[:size:size])


That second size should be unnecessary as that is the capacity, right?

andrewkroh · 2017-06-23T22:08:18Z

metricbeat/module/windows/perfmon/pdh_windows.go

+
+			for i = 0; i < bufferCount; i++ {
+				value := CounterValueItem{}
+				a := (*[1<<30 - 1]uint16)(unsafe.Pointer(pdhValues[i].SzName))


We should try to find a more "safe" way to do this in the future. Using the unsafe pointer to cast an arbitrary size array worries me. But that's for another PR another time. :)

andrewkroh · 2017-06-23T22:08:41Z

metricbeat/module/windows/perfmon/pdh_windows.go

@@ -80,24 +85,43 @@ func PdhGetFormattedCounterValue(counter PdhCounterHandle, format PdhCounterForm
 	return counterType, &value, nil
 }

-func PdhGetFormattedCounterArray(counter PdhCounterHandle, format PdhCounterFormat) (int, *[]PdhCounterValueItem, error) {
+func PdhGetFormattedCounterArray(counter PdhCounterHandle, format PdhCounterFormat) ([]CounterValueItem, error) {


That looks better.

martinscholz83 · 2017-07-05T07:44:05Z

@andrewkroh, tests are passing on my local machine. Can you please run the tests again. Thanks!

andrewkroh · 2017-07-05T17:40:44Z

jenkins, test it

andrewkroh · 2017-07-05T17:42:48Z

Ok, I triggered a rebuild of AppVeyor and Jenkins.

martinscholz83 · 2017-07-06T06:29:35Z

What is this TestData test which is failing??

andrewkroh · 2017-07-06T15:14:48Z

I'm seeing it fail before getting to TestData.

Building metricbeat
# github.com/elastic/beats/metricbeat/module/system/process
module\system\process\helper.go:398: undefined: sort.Slice
module\system\process\helper.go:405: undefined: sort.Slice

But TestData is used to generate the _meta/data.json file that contains a sample event that will show up in the docs.

martinscholz83 · 2017-07-07T09:12:06Z

🙈 This happens if you work on 3 different machines 😑

martinscholz83 · 2017-07-07T10:21:56Z

~~Why it doesn't implement EventFetcher interface?~~
My mistake. Called wrong function f := mbtest.NewEventsFetcher(t, config). Missed the s.

andrewkroh · 2017-07-11T02:58:42Z

🎉 Thanks!

`alias` was removed in elastic#4502.

`alias` was removed in #4502.

martinscholz83 changed the title ~~Perfmon wildcards~~ Perfmon wildcard queries Jun 14, 2017

andrewkroh reviewed Jun 14, 2017

View reviewed changes

andrewkroh approved these changes Jun 23, 2017

View reviewed changes

martinscholz83 and others added 11 commits July 7, 2017 10:19

Add PDH wildcard queries

c8f809f

Add example

aaf9bac

Add integration test

fcc008d

Fix test

66dcd77

Remove debug file

cda983b

Change return value

206f966

Change return value for method PdhGetFormattedCounterArray

70e2428

Fix slice

a7e4d87

Separate instances

ae416e7

FIx tests

5f6c28f

Fix TestWildCardQuery

60dabc5

martinscholz83 force-pushed the perfmon-wildcards branch from 22d7493 to 60dabc5 Compare July 7, 2017 08:20

Fix TestData

abefd90

Fixed wrong function call

53e5c19

andrewkroh merged commit 5f959cd into elastic:master Jul 11, 2017

andrewkroh added :Windows Metricbeat Metricbeat v6.0.0-beta1 labels Jul 11, 2017

martinscholz83 mentioned this pull request Jul 12, 2017

Add instance_name config option for perfmon metricset #4655

Merged

andrewkroh mentioned this pull request Sep 7, 2017

Added required alias property to documentation #4800

Merged

andrewkroh added a commit to andrewkroh/beats that referenced this pull request Sep 7, 2017

Remove alias from perfmon docs

47da645

`alias` was removed in elastic#4502.

andrewkroh mentioned this pull request Sep 7, 2017

Remove alias from perfmon docs #5130

Merged

tsg pushed a commit that referenced this pull request Sep 8, 2017

Remove alias from perfmon docs (#5130)

ac9561d

`alias` was removed in #4502.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Perfmon wildcard queries #4502

Perfmon wildcard queries #4502

martinscholz83 commented Jun 14, 2017

karmi commented Jun 14, 2017

elasticmachine commented Jun 14, 2017

elasticmachine commented Jun 14, 2017

martinscholz83 commented Jun 14, 2017 •

edited

Loading

andrewkroh left a comment

andrewkroh Jun 14, 2017

andrewkroh Jun 14, 2017

martinscholz83 Jun 21, 2017

andrewkroh Jun 14, 2017

andrewkroh Jun 14, 2017

andrewkroh Jun 14, 2017

andrewkroh Jun 14, 2017

martinscholz83 Jun 21, 2017

andrewkroh Jun 14, 2017

martinscholz83 Jun 21, 2017

andrewkroh commented Jun 14, 2017 •

edited

Loading

martinscholz83 commented Jun 21, 2017 •

edited

Loading

martinscholz83 commented Jun 23, 2017 •

edited

Loading

andrewkroh left a comment

andrewkroh Jun 23, 2017

andrewkroh Jun 23, 2017

andrewkroh Jun 23, 2017

martinscholz83 commented Jul 5, 2017

andrewkroh commented Jul 5, 2017

andrewkroh commented Jul 5, 2017

martinscholz83 commented Jul 6, 2017

andrewkroh commented Jul 6, 2017 •

edited

Loading

martinscholz83 commented Jul 7, 2017

martinscholz83 commented Jul 7, 2017 •

edited

Loading

andrewkroh commented Jul 11, 2017

Perfmon wildcard queries #4502

Perfmon wildcard queries #4502

Conversation

martinscholz83 commented Jun 14, 2017

karmi commented Jun 14, 2017

elasticmachine commented Jun 14, 2017

elasticmachine commented Jun 14, 2017

martinscholz83 commented Jun 14, 2017 • edited Loading

andrewkroh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh commented Jun 14, 2017 • edited Loading

martinscholz83 commented Jun 21, 2017 • edited Loading

martinscholz83 commented Jun 23, 2017 • edited Loading

andrewkroh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martinscholz83 commented Jul 5, 2017

andrewkroh commented Jul 5, 2017

andrewkroh commented Jul 5, 2017

martinscholz83 commented Jul 6, 2017

andrewkroh commented Jul 6, 2017 • edited Loading

martinscholz83 commented Jul 7, 2017

martinscholz83 commented Jul 7, 2017 • edited Loading

andrewkroh commented Jul 11, 2017

martinscholz83 commented Jun 14, 2017 •

edited

Loading

andrewkroh commented Jun 14, 2017 •

edited

Loading

martinscholz83 commented Jun 21, 2017 •

edited

Loading

martinscholz83 commented Jun 23, 2017 •

edited

Loading

andrewkroh commented Jul 6, 2017 •

edited

Loading

martinscholz83 commented Jul 7, 2017 •

edited

Loading