[wmi] Significant performance improvements on default Windows checks

[TheCloudlessSky]

We're just starting out with Datadog and it's great. After a week of testing, I deployed the Datadog Agent to a couple of test Windows hosts (m3.medium EC2 instances). I noticed the CPU of the hosts jumping to 20-30% for 1-2 seconds every ~15 seconds (only the default checks were enabled). I checked and found that WmiPrvSE.exe and ddagent.exe were the culprits. This was reported to the support team. In the mean time, I wanted to investigate on my own.

Because the shell.exe is shipped, it allowed me to easily debug the problems. I narrowed it down to the default Windows checks and the WMI calls.

Problems

1. There is a WMI query in win32.Processes that does not use the result.
2. The wmi Python package has lots of overhead to the queries.
3. The queries didn't explicitly specify the properties (too much data being returned).
4. Most of the queries use Win32_PerfFormattedData_* (which are really slow).

Solutions

1. Remove this empty check.
2. Similar to [wmi] new AgentCheck for WMI metrics #1917 (which I found and also support pointed me to), stop using the wmi package because of the large overhead.
3. Always specify the properties in the queries.
4. Use the Win32_PerfRawData_* classes and compute the values our self (this is the main performance gain). The MSDN documentation is a pain to work with, but once you understand how to calculate the values, it's easy and has huge performance improvements.

DISCLAIMER: I'm not primarily a Python programmer, so the code may not be idiomatic Python (sorry).

I originally implemented the changes before I learned about doing COM directly (from #1917). I found the wmi package still had a large amount of overhead. I re-implemented with direct COM and that's what I'm presenting to you today.

I created this sample gist. It shows the use of COM, explicit properties, and Win32_PerfRawData_* classes to improve performance. If this looks good, I'll create a PR integrating this with the real Windows checks. I created a _WMISampler class that holds the previous/current sample taken from the raw performance counter and does the calculation of the value.

Results

On my dev laptop (Intel i7-3520M), the default Windows checks take:

• Originally: ~2200ms
• Now: ~150ms

Here's a graph of 10 samples for each version (original and my fixed wmi version):

This leads to an almost nil performance impact to run the default Windows checks.

Questions

1. Are there concerns with using calculated Win32_PerfRawData_* classes over Win32_PerfFormattedData_* classes? The formatted classes are computed from multiple samples. So re-measuring might return an inaccurate value since the default Datadog Agent check time is 15 seconds. Using the raw values should give a much more accurate measurement.
2. Can someone look over my calculations for the counter types? The little bit of math needs to be checked with the algorithms on MSDN.
3. Do the COM objects need to be released that could cause memory leaks? [wmi] new AgentCheck for WMI metrics #1917 did not release any COM objects (so I followed suit).

Happy Friday 😄

[alq666]

@TheCloudlessSky very nice work 🙇

[alq666]

@remh @yannmh kindly take a look

[remh]

Oh wow. Awesome work @TheCloudlessSky . We were actually talking about porting the new WMI check to the system check as well. Thanks a lot for this contribution and feedback, we'll review this very shortly.

[yannmh]

Thanks a lot for the nice work and the detailed explanations @TheCloudlessSky !

As you mentionned, we recently worked on a new WMI AgentCheck after stating the same slowness issues with the wmi Python package. Contrary to the current one, it is using COM directly, and yet, performance improvements were significant: it roughly divided by 10 the CPU time consumed.

Looking at your summary, the results you are getting are even more impressive 🚀. I am looking forward to spend some time reading and diving into your work. In particular I am curious to compare/benchmark the results we are getting by querying and computing from PerfRawData_* or directly from PerfFormattedData_* classes.

Best,

Yann

[TheCloudlessSky]

@yannmh Awesome. It's very exciting seeing the huge performance differences between PerfRawData_* and PerfFormattedData_*. Please let me know if I can help in any way!

[TheCloudlessSky]

I've also found that other Windows checks (e.g. IIS) use PerfFormattedData_* classes that cause ~400ms to execute on m3.medium instances.

I think what would be best is to use the _WMISampler class when any WMI sample is needed. Then, rather than hard-coding the property calculators, we use the wbemFlagUseAmendedQualifiers flag with ExecQuery() and use the Qualifiers_ property for the first time any query executes (so that a performance hit isn't taken on each query). Then, the only thing left to do is map the CounterType qualifier value to a function that does the calculation.

[TheCloudlessSky]

I've updated the Gist:

• Split WMISampler into its own file.
• Load/cache the Qualifiers_ on first query to get the CounterType.
• Completed calculators for all classes required in win32.py (see counter_type_calculators). Other counter types will need to be implemented if this is used elsewhere.
• Allow the connection info (host, namespace, username, password) to be specified for the WMISampler.
• Use the wbemFlagForwardOnly flag to improve enumeration/memory performance (though not by much).

[yannmh]

@TheCloudlessSky,

I put everything together and made a proper PR #2011 against Datadog Agent. I'd appreciate having your feedback on it.

Again thank you so much for the hard work ! I had a really great time reviewing your work. The performances gain are very impressive (similar to the one you described) and I can't wait to see the changes deployed with the new Datadog Agent release.

[TheCloudlessSky]

Great! I'm closing this in favor of your PR.