PowerDNS recursor check

[AntoCard]

Check for PowerDNS recursor metrics. Needs pdns-recursor experimental web server active.

[remh]

You should import simplejson instead, it's more efficient

[remh]

Thanks a lot @AntoCard for this!
According to this page https://doc.powerdns.com/md/recursor/stats/ there are a lot of metrics that can be collected.

I think we should explicitly list the metrics that we want to collect (it will be also easier to send them with the more appropriate type).

Thanks again!

[irabinovitch]

@AntoCard I wanted to echo Remi's thanks here too. This looks great. Lets sync up on what metrics make sense to pull in by default, and maybe collaborate on a readme for setup so that folks know what the requirements/dependencies are.

[AntoCard]

Hi guys, thanks for the feedback. It's really useful. I'll work in improving this and update my PR. Cheers.

[irabinovitch]

@AntoCard Have you had a chance to review the feedback? We'd love to include this in an upcoming release.

[AntoCard]

Hi @irabinovitch, I'm going to collect a list of the metrics we find useful here in Notonthehighstreet, maybe we can use some of them as default metrics.

[irabinovitch]

@AntoCard Sounds good. Looking forward to it!

[irabinovitch]

@AntoCard Were you able to look at collecting a list of metrics? We're hoping for a short window between 5.7 and 5.8, so would be great to get this addressed and included early.

[AntoCard]

@irabinovitch this are the queries we use more often:

pdns.answers0_1 Queries answered within 1 millisecond
pdns.answers1_10 Queries answered within 10 milliseconds
pdns.answers10_100 Queries answered within 100 milliseconds
pdns.answers100_1000 Queries answered within 1 second
pdns.answers_slow Queries answered after 1 second
pdns.outgoing_timeouts Number of timeouts on outgoing UDP queries
pdns.noerror_answers Number of times it answered NOERROR
pdns.servfail_answers Number of times it answered SERVFAIL

[janeczku]

This PR is great!
Here is a basic set of the most useful metrics reported by PowerDNS recursor:

Gauges

concurrent-queries
cache-entries

Counters

questions
tcp-questions
all-outqueries
tcp-outqueries
servfail-answers
noerror-answers
cache-hits
cache-misses

Histograms

Would be nice if the PR could be extended to derive histogram metrics from these cumulative histogram counters. Otherwise we can of course just submit them as counters.

answers0-1 (0-1ms latency)
answers1-10 (1-10ms latency)
answers10-100 (10-100ms latency)
answers100-1000 (100-1000ms latency)
answers-slow (>1000ms latency)

[janeczku]

@AntoCard 😆 you beat me to it by a minute!

[AntoCard]

@janeczku your list is great! Are you using it in a Docker/Mesos environment?

[janeczku]

Almost. I am running multiple instances of dnsdist and PowerDNS in a Docker/Rancher environment. I am currently tracking PowerDNS metrics with New Relic but, well, i really would like to switch over to DataDog... 😆

[remh]

Nitpick but you should remove the .j2 file extension.

[remh]

Thanks a lot guys! Do you think you could add a test similar to our other integrations tests ? It can be mocked if it's not easy to set up powerdns

[remh]

Also, we should likely only include the metrics that are relevant. It's better than collecting metrics that are not useful

[degemer]

Closing in favor of #2490