Call host parser only once when building light metricsets #20149
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jsoriano
added
refactoring
Metricbeat
botelastic
bot
added
needs_team
💔 Tests Failed
Expand to view the summary
Build stats
Test stats 🧪
Test errorsExpand to view the tests failures
Steps errorsExpand to view the steps failures
Log outputExpand to view the last 100 lines of log output
|
|
It LGTM. I know this is a draft but I just wanted to understand how this small piece works: The really confusing part for me is the "overriden defaults". That's why I didn't touch it myself to change it. Can you elaborate briefly what does it mean? Because it seems obvious that, in the past, we were running host parser twice on purpose for some reason. My question is why Beats did it a bit more complicated in the past (running hostparser twice) when we could do it in a straightforward way (this PR). I mean, maybe we are missing something in this PR that may break if we merge it into Beats |
|
I'll also summon @mtojek in case he can spend few minutes peeking around this, as it was related with something he needed and he may realize of something |
|
@sayden let me try to explain all this 🙂 Metricbeat module builder calls the host parser if the registration has anyone defined, here: beats/metricbeat/mb/builders.go Lines 121 to 137 in 10bfeb4 Lightweight metricsets are not registered. When needed, they create a registration based on the registration of the metricset they are based on, here: beats/metricbeat/mb/lightmetricset.go Line 44 in 10bfeb4 Then on this registration we replace some things:
("Overriding the defaults" on this context is to use the defaults included in the On the initial implementation we found that host parsers can use other settings apart of This seemed to work, till @mtojek found that the scheme got lost, the problem is that information in beats/metricbeat/mb/builders.go Line 136 in 10bfeb4 So the second call to the host parser could be missing some of the original configuration. Then he added #16205 to workaround this specific issue with the scheme, and it worked well for all our current cases because they use in general normal URLs. Now you have found another case that doesn't work. On this PR I am removing Marcin's change, but I am not touching his tests, so in principle this should work as he expected. I will add him to the review in any case. |
jsoriano
added
the
needs_backport
|
Pinging @elastic/integrations-services (Team:Services) |
jsoriano
changed the title
sayden
approved these changes
Jul 23, 2020
|
Failures in auditbeat are also happening in master. |
mtojek
approved these changes
Jul 23, 2020
6 tasks
jsoriano
added
v7.10.0
and removed
needs_backport
jsoriano
added a commit
to jsoriano/beats
that referenced
this pull request
Jul 23, 2020
) When using light modules, host parser is called twice. First by the actual implementation of the metricset, and second after adding the configuration defined in the light module manifest. Second call might be missing data as the original host is modified after the first call, causing problems. This change disables host parser in the registration created for light modules, and makes only the "second call" inside the factory. It also removes previous fix for URLs as it shouldn't be needed anymore. (cherry picked from commit 6c82292)
jsoriano
added a commit
to jsoriano/beats
that referenced
this pull request
Jul 23, 2020
) When using light modules, host parser is called twice. First by the actual implementation of the metricset, and second after adding the configuration defined in the light module manifest. Second call might be missing data as the original host is modified after the first call, causing problems. This change disables host parser in the registration created for light modules, and makes only the "second call" inside the factory. It also removes previous fix for URLs as it shouldn't be needed anymore. (cherry picked from commit 6c82292)
6 tasks
This was referenced Jul 23, 2020
jsoriano
added a commit
that referenced
this pull request
Jul 24, 2020
…20189) When using light modules, host parser is called twice. First by the actual implementation of the metricset, and second after adding the configuration defined in the light module manifest. Second call might be missing data as the original host is modified after the first call, causing problems. This change disables host parser in the registration created for light modules, and makes only the "second call" inside the factory. It also removes previous fix for URLs as it shouldn't be needed anymore. (cherry picked from commit 6c82292)
jsoriano
added a commit
that referenced
this pull request
Jul 24, 2020
…20190) When using light modules, host parser is called twice. First by the actual implementation of the metricset, and second after adding the configuration defined in the light module manifest. Second call might be missing data as the original host is modified after the first call, causing problems. This change disables host parser in the registration created for light modules, and makes only the "second call" inside the factory. It also removes previous fix for URLs as it shouldn't be needed anymore. (cherry picked from commit 6c82292)
v1v
added a commit
to v1v/beats
that referenced
this pull request
Jul 27, 2020
…ne-2.0 * upstream/master: (41 commits) adding possibility to override content-type checks, it was breaking certain webhooks that is not able to set content-headers at all. Still defaults to application/json (elastic#20232) fix: use a fixed worker type for tests (elastic#20130) [Ingest Manager] Prepare packaging for endpoint and asc files (elastic#20186) [Packetbeat] HTTP: Improve support for 100-continue elastic#15830 (elastic#19349) Increase index.max_docvalue_fields_search to 200 (elastic#20218) [Ingest Manager] Prevent closing closed reader (elastic#20214) [Metricbeat] Use MySQL Host Parser in Query metricset (elastic#20191) Testing: Ignore timestamp from cylance/protect dataset (elastic#20211) [Filebeat] Ignore cylance.protect timestamps while testing (elastic#20207) [CI] remove codecov step (elastic#20102) [docs] Indicate that SYSTEM user is required on Windows to use Endpoint (elastic#20172) Remove f5/firepass rsa2elk fileset (elastic#20160) [Elastic Agent] Improve GRPC stop to be more relaxed. (elastic#20118) Fix fileset field prefixing (elastic#20170) Fix terminating pod autodiscover issue (elastic#20084) Call host parser only once when building light metricsets (elastic#20149) [CI] fix null string with contains (elastic#20182) [Ingest Manager] Fix failing unit tests on windows (elastic#20127) [Filebeat] Update crowdstrike module (elastic#20138) [docs] Add x-pack role to relevant metricsets (elastic#20167) ...
v1v
added a commit
to v1v/beats
that referenced
this pull request
Jul 29, 2020
* upstream/7.9: (32 commits) feat(ci): support storing artifacts for PRs in separate dirs (elastic#20282) (elastic#20301) Cisco ASA: Fix message 106100 (elastic#20245) (elastic#20277) [CI] Change upstream reference (elastic#20296) (elastic#20297) [docs] Fix Windows download link for agent (elastic#20258) (elastic#20290) Cherry-pick to 7.9: [docs] Rename release highlights to what's new (elastic#20255) (elastic#20285) Elastic agent on k8s (elastic#19727) (elastic#20262) [Filebeat Module] Defender ATP - Adding dashboard (elastic#20058) (elastic#20093) fix: use a fixed worker type for tests (elastic#20130) (elastic#20247) [Elastic Agent] Fix Windows powershell install service script (elastic#20203) (elastic#20252) [Ingest Manager] Fixed unzip on older windows (elastic#20088) (elastic#20109) adding possibility to override content-type checks, it was breaking certain webhooks that is not able to set content-headers at all. Still defaults to application/json (elastic#20232) (elastic#20237) [Filebeat][Gsuite] Make GSuite docs more clear (elastic#19981) (elastic#20067) Increase index.max_docvalue_fields_search to 200 (elastic#20218) (elastic#20221) Call host parser only once when building light metricsets (elastic#20149) (elastic#20190) [Metricbeat] Use MySQL Host Parser in Query metricset (elastic#20191) (elastic#20212) [Filebeat] Ignore cylance.protect timestamps while testing (elastic#20207) (elastic#20217) [libbeat] Fix write error in ensureWriter.Write (elastic#20112) (elastic#20145) Cherry-pick elastic#20127 to 7.9: Fix failing unit tests on windows (elastic#20180) Remove f5/firepass rsa2elk fileset (elastic#20160) (elastic#20206) Cherry-pick elastic#20138 to 7.9: [Filebeat] Update crowdstrike module (elastic#20177) ...
melchiormoulin
pushed a commit
to melchiormoulin/beats
that referenced
this pull request
Oct 14, 2020
) When using light modules, host parser is called twice. First by the actual implementation of the metricset, and second after adding the configuration defined in the light module manifest. Second call might be missing data as the original host is modified after the first call, causing problems. This change disables host parser in the registration created for light modules, and makes only the "second call" inside the factory. It also removes previous fix for URLs as it shouldn't be needed anymore.
leweafan
pushed a commit
to leweafan/beats
that referenced
this pull request
Apr 28, 2023
) (elastic#20190) When using light modules, host parser is called twice. First by the actual implementation of the metricset, and second after adding the configuration defined in the light module manifest. Second call might be missing data as the original host is modified after the first call, causing problems. This change disables host parser in the registration created for light modules, and makes only the "second call" inside the factory. It also removes previous fix for URLs as it shouldn't be needed anymore. (cherry picked from commit 57d1265)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Call host parser only once on Metricbeat light modules.
Why is it important?
When using light modules, host parser is called twice. First by the actual implementation
of the metricset, and second after adding the configuration defined in the light module
manifest. Second call might be missing data as the original host is modified after the first
call, causing problems as the ones described in #16205 and #19849.
Checklist
I have added tests that prove my fix is effective or that my feature works(Covered by tests from Fix: don't miss address scheme #16205).CHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.How to test this PR locally
Related issues
Use cases