Add minimum cache TTL for successful DNS responses #18986

ansell · 2020-06-04T23:22:03Z

What does this PR do?

An enhancement to add a minimum alternative cache TTL to the libbeat dns processor for successful DNS responses. This ensures that TTL=0 successful reverse DNS responses can be cached to avoid sending the same reverse DNS request again within a short period of time.

Why is it important?

The libbeat dns processor is used as a reverse DNS annotator for auditbeat events. Some of these IP addresses respond to reverse DNS requests with TTL=0 in the responses. These were causing load issues for my systems when I had the reverse DNS processor enabled for auditbeat.

Checklist

My code follows the style guidelines of this project
~~- [ ] I have commented my code, particularly in hard-to-understand areas~~
I have made corresponding changes to the documentation
~~- [ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

Sign Elastic CLA

How to test this PR locally

The auditbeat system/socket module that I had the issue with is not open-sourced, so I have not been able to test its completeness myself. I am submitting this in good faith that it can be completed by someone with access to that module for testing purposes, or who is experienced enough to add the necessary tests here.

Related issues

Closes Add a minimum TTL to auditbeat reverse DNS resolution #18709

Use cases

This will add a cache for successful DNS responses that will not always correspond to the upstream DNS TTL value. This is acceptable as it avoids a possible denial-of-service for auditbeat enabled systems, and it is configurable by users.

elasticmachine · 2020-06-04T23:22:05Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

elasticmachine · 2020-06-04T23:22:05Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

elasticmachine · 2020-06-04T23:26:05Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [andrewkroh commented: run tests]
Start Time: 2020-08-02T17:21:46.199+0000
Duration: 87 min 49 sec

Test stats 🧪

Test	Results
Failed	0
Passed	14465
Skipped	1373
Total	15838

elasticmachine · 2020-06-05T09:02:12Z

Pinging @elastic/siem (Team:SIEM)

marc-gr · 2020-07-15T10:55:15Z

Hello @ansell and thanks for opening this PR!

Could you rebase this from master and add an entry in the CHANGELOG.next.asciidoc file? Or if you prefer we take over from this point please say so! Whatever works best for you

marc-gr · 2020-07-15T10:55:23Z

jenkins run tests

Signed-off-by: Peter Ansell <[email protected]>

ansell · 2020-07-16T08:40:13Z

@marc-gr I followed the basic instructions that @andrewkroh mentioned on the linked issue but I wasn't able to get the tests to work locally to verify my approach was appropriate.

I would prefer if you took over at this point. I have rebased onto the current master and pushed again.

marc-gr · 2020-07-16T08:47:16Z

@marc-gr I followed the basic instructions that @andrewkroh mentioned on the linked issue but I wasn't able to get the tests to work locally to verify my approach was appropriate.

I would prefer if you took over at this point. I have rebased onto the current master and pushed again.

No problem!

marc-gr · 2020-07-16T09:31:39Z

jenkins run tests

marc-gr · 2020-07-16T09:32:20Z

I fixed the set method and added a unit test. Please @andrewkroh take a look in case I missed something when you have a minute.

andrewkroh

I have requested a change to ensure the TTL returned by the PTRLookupCache consistently reflects the cache TTL rather than the DNS response TTL.

Additionally this needs

Changelog entry
Doc update to include the new parameter

libbeat/processors/dns/config.go

libbeat/processors/dns/cache_test.go

libbeat/processors/dns/cache.go

Co-authored-by: Andrew Kroh <[email protected]>

…beat-dns-cache

marc-gr · 2020-07-17T08:04:37Z

jenkins run tests

andrewkroh

LGTM, thanks @ansell and @marc-gr .

Signed-off-by: Peter Ansell <[email protected]>

ansell · 2020-07-18T03:47:27Z

I made one small change to make the error message match the configuration item it references.

marc-gr · 2020-07-20T13:24:45Z

jenkins run tests

…beat-dns-cache

marc-gr · 2020-07-21T11:54:21Z

jenkins run tests

…beat-dns-cache

andrewkroh · 2020-08-02T17:21:21Z

run tests

An enhancement to add a minimum alternative cache TTL to the libbeat dns processor for successful DNS responses. This ensures that TTL=0 successful reverse DNS responses can be cached to avoid sending the same reverse DNS request again within a short period of time. The libbeat dns processor is used as a reverse DNS annotator for auditbeat events. Some of these IP addresses respond to reverse DNS requests with TTL=0 in the responses. These were causing load issues for my systems when I had the reverse DNS processor enabled for auditbeat. The new settings is `success_cache.min_ttl`. Closes elastic#18709 Signed-off-by: Peter Ansell <[email protected]> Co-authored-by: Marc Guasch <[email protected]> (cherry picked from commit 72da5a6)

…esponses (#20525) * Add success cache minimum TTL for DNS responses (#18986) An enhancement to add a minimum alternative cache TTL to the libbeat dns processor for successful DNS responses. This ensures that TTL=0 successful reverse DNS responses can be cached to avoid sending the same reverse DNS request again within a short period of time. The libbeat dns processor is used as a reverse DNS annotator for auditbeat events. Some of these IP addresses respond to reverse DNS requests with TTL=0 in the responses. These were causing load issues for my systems when I had the reverse DNS processor enabled for auditbeat. The new settings is `success_cache.min_ttl`. Closes #18709 Signed-off-by: Peter Ansell <[email protected]> Co-authored-by: Marc Guasch <[email protected]> (cherry picked from commit 72da5a6) * Update CHANGELOG.next.asciidoc Co-authored-by: Peter Ansell <[email protected]>

An enhancement to add a minimum alternative cache TTL to the libbeat dns processor for successful DNS responses. This ensures that TTL=0 successful reverse DNS responses can be cached to avoid sending the same reverse DNS request again within a short period of time. The libbeat dns processor is used as a reverse DNS annotator for auditbeat events. Some of these IP addresses respond to reverse DNS requests with TTL=0 in the responses. These were causing load issues for my systems when I had the reverse DNS processor enabled for auditbeat. The new settings is `success_cache.min_ttl`. Closes elastic#18709 Signed-off-by: Peter Ansell <[email protected]> Co-authored-by: Marc Guasch <[email protected]>

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 4, 2020

ansell mentioned this pull request Jun 4, 2020

Add a minimum TTL to auditbeat reverse DNS resolution #18709

Closed

andresrc added the Team:SIEM label Jun 5, 2020

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 5, 2020

issue elastic#18709 : Add minimum cache TTL for successful DNS responses

6044af7

Signed-off-by: Peter Ansell <[email protected]>

ansell force-pushed the issue-18709-auditbeat-dns-cache branch from b736d97 to 6044af7 Compare July 16, 2020 08:36

Fix cache set and add unit test

f560dc4

marc-gr requested a review from andrewkroh July 16, 2020 09:32

andrewkroh requested changes Jul 16, 2020

View reviewed changes

marc-gr and others added 4 commits July 17, 2020 09:38

Apply suggestions from code review

7b14f82

Co-authored-by: Andrew Kroh <[email protected]>

Merge remote-tracking branch 'upstream/master' into issue-18709-audit…

c3ed27b

…beat-dns-cache

Add CHANGELOG and doc entries

e199f01

Override ptr.TTL with minTTL and cleanup unused code

61a2a69

marc-gr requested a review from andrewkroh July 17, 2020 08:24

andrewkroh approved these changes Jul 17, 2020

View reviewed changes

issue elastic#18709 : Change error message to match config item change

4fc6e1f

Signed-off-by: Peter Ansell <[email protected]>

Merge remote-tracking branch 'upstream/master' into issue-18709-audit…

0a52c8f

…beat-dns-cache

marc-gr added 2 commits July 21, 2020 13:52

Merge remote-tracking branch 'upstream/master' into issue-18709-audit…

810ea67

…beat-dns-cache

Reorder CHANGELOG.next.asciidoc

09d1464

marc-gr and others added 2 commits July 22, 2020 16:07

Merge remote-tracking branch 'upstream/master' into issue-18709-audit…

636c1f5

…beat-dns-cache

Merge branch 'master' into issue-18709-auditbeat-dns-cache

505dd91

andrewkroh added :Processors enhancement libbeat labels Aug 2, 2020

andrewkroh added the needs_backport PR is waiting to be backported to other branches. label Aug 10, 2020

andrewkroh merged commit 72da5a6 into elastic:master Aug 10, 2020

andrewkroh changed the title ~~issue #18709 : Add minimum cache TTL for successful DNS responses~~ Add minimum cache TTL for successful DNS responses Aug 10, 2020

andrewkroh mentioned this pull request Aug 10, 2020

Cherry-pick #18986 to 7.x: Add minimum cache TTL for successful DNS responses #20525

Merged

5 tasks

andrewkroh added v7.10.0 and removed needs_backport PR is waiting to be backported to other branches. labels Aug 10, 2020

ycombinator mentioned this pull request Sep 14, 2020

Adds a default for failure_cache.min_ttl #21085

Merged

andrewkroh mentioned this pull request Sep 28, 2020

Cherry-pick #21085 to 7.x: Adds a default for failure_cache.min_ttl #21353

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add minimum cache TTL for successful DNS responses #18986

Add minimum cache TTL for successful DNS responses #18986

ansell commented Jun 4, 2020 •

edited by andrewkroh

Loading

elasticmachine commented Jun 4, 2020

elasticmachine commented Jun 4, 2020

elasticmachine commented Jun 4, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

elasticmachine commented Jun 5, 2020

marc-gr commented Jul 15, 2020 •

edited

Loading

marc-gr commented Jul 15, 2020

ansell commented Jul 16, 2020

marc-gr commented Jul 16, 2020

marc-gr commented Jul 16, 2020

marc-gr commented Jul 16, 2020

andrewkroh left a comment •

edited by marc-gr

Loading

marc-gr commented Jul 17, 2020

andrewkroh left a comment

ansell commented Jul 18, 2020

marc-gr commented Jul 20, 2020

marc-gr commented Jul 21, 2020

andrewkroh commented Aug 2, 2020

Add minimum cache TTL for successful DNS responses #18986

Add minimum cache TTL for successful DNS responses #18986

Conversation

ansell commented Jun 4, 2020 • edited by andrewkroh Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

elasticmachine commented Jun 4, 2020

elasticmachine commented Jun 4, 2020

elasticmachine commented Jun 4, 2020 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

elasticmachine commented Jun 5, 2020

marc-gr commented Jul 15, 2020 • edited Loading

marc-gr commented Jul 15, 2020

ansell commented Jul 16, 2020

marc-gr commented Jul 16, 2020

marc-gr commented Jul 16, 2020

marc-gr commented Jul 16, 2020

andrewkroh left a comment • edited by marc-gr Loading

Choose a reason for hiding this comment

marc-gr commented Jul 17, 2020

andrewkroh left a comment

Choose a reason for hiding this comment

ansell commented Jul 18, 2020

marc-gr commented Jul 20, 2020

marc-gr commented Jul 21, 2020

andrewkroh commented Aug 2, 2020

ansell commented Jun 4, 2020 •

edited by andrewkroh

Loading

elasticmachine commented Jun 4, 2020 •

edited by jenkins-beats-ci bot

Loading

marc-gr commented Jul 15, 2020 •

edited

Loading

andrewkroh left a comment •

edited by marc-gr

Loading