-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Report FQDN on host.name enabled by a feature flag #34456
Conversation
This pull request does not have a backport label.
To fixup this pull request, you need to add the backport labels for the needed
|
There are a few other references to the
|
It appears that the Windows event logs might be the FQDN by default. We should explicitly try to test and inspect the |
This pull request is now in conflicts. Could you fix it? 🙏
|
luckily the huge majority of these files are the same, There is indeed a few places that indeed use a different approach to populate |
f22ab3e
to
7338984
Compare
79506a8
to
8ce8fb0
Compare
8ce8fb0
to
d36e29d
Compare
@@ -222,6 +222,7 @@ func (r *Renderer) renderSystem(handle EvtHandle, event *winevent.Event) error { | |||
event.Channel = data.(string) | |||
case EvtSystemComputer: | |||
event.Computer = data.(string) | |||
// TODO: set event.Computer to FQDN if features.FQDN() is true? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@efd6 @andrewkroh In this PR we're trying to set host.name
in events generated by Beats to the FQDN if the Beat config contains features.fqdn.enabled: true
. What's the best way to retrieve the FQDN on a Windows host? I assume this is the line in code where we'd want to set event.Computer
to the FQDN if the feature is enabled?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
First, a little background on where this event.Computer
value comes from... I think the way Windows event log records work is that each event log record contains a ComputerName
value that is determined at the time the event is generated. If the computer is not a member of a Active Directory domain when the log is written then it gets recorded with a short hostname (e.g. mymachine). If the computer is a member of an AD domain then it gets recorded as <hostname>.<ad-domain-name>
(e.g. mymachine.company.com).
And be aware that the event log reader can read archived .evtx files that may have originated on a different machine (with a different hostname that the reader). Similarly event log records can be forwarded from one host to another and in this case they will have the source machine's hostname (or <hostname>.<domain>
if it was a domain member).
For data integrity purposes I think we will want to keep the winlog.computer_name
field exactly as it was in the source event log record. That value is copied directly into host.name
so that is probably the location where we need to make some changes/decisions.
beats/winlogbeat/eventlog/eventlog.go
Line 101 in 34a87e5
winevent.AddOptional(m, "host.name", e.Computer) |
-
Perhaps we let
add_host_metadata
overwrite thehost.name
value with the appropriate host name value based on the features. Could make an exception for when the data is forwarded from another host or the data is read from an archived log? This is similar to what we tried to achieve in the defaultwinlogbeat.yml
config except I don't think this config overwrites thehost.name
value that was set from the ComputerName.beats/winlogbeat/winlogbeat.yml
Lines 139 to 140 in cbd1a38
- add_host_metadata: when.not.contains.tags: forwarded -
In the case of forwarded logs or logs read from an .evtx file, should we try and normalize the value? For example if features.fqdn.enabled=false then do we try to trim the domain suffix if it is present in the event log's
ComputerName
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the background and suggestions, @andrewkroh. I've created a separate issue to handle FQDN reporting for Windows hosts, as the use case urgently driving the FQDN feature implementation is for Beats running in containerized environments, which tend to be *nix most of the times. I've created #34782 to follow up on implementing FQDN reporting for Windows hosts.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to keep this TODO in the code? From our previous discussion I believe we concluded we likely don't need to touch this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch, we don't need this TODO any more. Removing...
1fa2caa
to
3548c07
Compare
This pull request is now in conflicts. Could you fix it? 🙏
|
@cmacknz Thanks for posting the
For the first two results, related to hostnames on Windows, I'll follow up in #34456 (comment). That leaves the last result, related to the Additionally, not seen in the |
26c38ad
to
053ed7a
Compare
454e4b0
to
b0e457f
Compare
} | ||
if state == client.UnitStateStopped { | ||
if expected.Features != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we disable the feature if expected.Features == nil
and the current Features configuration is not nil? Is this a case you need to handle, or does the agent always provided a non-nil Features in expected?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the beginning, before any feature flags are set, Agent will send a nil
value for expected.Features
. So this check here guards against that condition, preventing any unnecessary processing. Thereafter, whenever a feature flag gets set, Agent will only send non-nil values for expected.Features
; see https://github.com/elastic/elastic-agent/pull/2218/files#diff-5570d8d3d94d053efb7e59bcad53e531bcf3a0ff5b8d3da9cc3334d859f095c9R176-R181
Since expected.Features
is built to contain multiple features' configuration in it, we would only disable a feature if that particular feature's configuration within expected.Features
indicated as such. For example, if expected.Features.FQDN.Enabled
was set to false
. AFAIU we have not designed for a use case of mass-disabling all features by setting expecting.Features
to nil
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
AFAIU we have not designed for a use case of mass-disabling all features by setting expecting.Features to nil.
That doesn't mean someone won't make that edit in the policy. IMO the most obvious thing to do is to translate that to disabling all features, that maps to what happens when you remove an input from the policy. We can file a follow up issue to deal with this, it is definitely an edge case.
* wip * wip * wip * something is broken * debug logs * avoid panics * wip broken * wip - perhaps fixed double configuring features * using triggers to know what changed * it works, but the agent-client is sending feature changes every time * debug logs * more debug logs * wip * it works * remove debug logs * fix after rebase * update elastic-agent-client and make notice * license headres * Add TODO for Winlogbeat FQDN implementation * Removing unnecesary code * Use RWMutex for greater read efficiency * Move mutex inside struct * Better error handling * Temporarily updating dependency * Always store FQDN in Beat info * Set hostname in monitoring registries after FQDN config has been evaluated * Pass down FQDN flag from add_host_metadata processor * Better error handling * Add temporary notice override * Changing log level * Update dependency on elastic-agent-system-metrics * Running go mod tidy * Update elastic-agent-system-metrics dependency version * Add feature flags configuration section to reference yml * Bump up version of elastic-agent-client depdendency * Updating reference file * Updating reference files * Move feature flags setting into same method where log level setting is handled * Updating docs * Adding CHANGELOG entry * Remove replace directive for go-sysinfo * Running go mod tidy * Remove replace directive for elastic-agent-client dependency * Remove personal repos from overrides.json * Updating NOTICE * Bump up version for elastic-agent-client dependency * Update NOTICE * Removing noisy debug logging * Removing commented out code that's not needed * Fixing error message * Make default flag setting more explicit * Adding config with no features to unit test * Add helper method to beat.Info for returning FQDN-aware hostname * Adding license header to test file * Remove use of errors package * Check error from data.Put * Expire add_host_metadata cache immediately on FQDN feature flag value change * Fix casing in error message * Have callback registration function accept ID * Fixing go formatting * Account for feature flags in TestManagerV2 test * Update logging for unit changes * Remove TODO from wineventlog code * Updating dependency * Include shared file in docs * Bumping dependency on elastic-agent-client * Include features config in diagnostics output * Update comment * Updating NOTICE * Re-formatting * Running make update --------- Co-authored-by: Shaunak Kashyap <[email protected]> (cherry picked from commit faf4c11) # Conflicts: # go.mod # go.sum
* wip * wip * wip * something is broken * debug logs * avoid panics * wip broken * wip - perhaps fixed double configuring features * using triggers to know what changed * it works, but the agent-client is sending feature changes every time * debug logs * more debug logs * wip * it works * remove debug logs * fix after rebase * update elastic-agent-client and make notice * license headres * Add TODO for Winlogbeat FQDN implementation * Removing unnecesary code * Use RWMutex for greater read efficiency * Move mutex inside struct * Better error handling * Temporarily updating dependency * Always store FQDN in Beat info * Set hostname in monitoring registries after FQDN config has been evaluated * Pass down FQDN flag from add_host_metadata processor * Better error handling * Add temporary notice override * Changing log level * Update dependency on elastic-agent-system-metrics * Running go mod tidy * Update elastic-agent-system-metrics dependency version * Add feature flags configuration section to reference yml * Bump up version of elastic-agent-client depdendency * Updating reference file * Updating reference files * Move feature flags setting into same method where log level setting is handled * Updating docs * Adding CHANGELOG entry * Remove replace directive for go-sysinfo * Running go mod tidy * Remove replace directive for elastic-agent-client dependency * Remove personal repos from overrides.json * Updating NOTICE * Bump up version for elastic-agent-client dependency * Update NOTICE * Removing noisy debug logging * Removing commented out code that's not needed * Fixing error message * Make default flag setting more explicit * Adding config with no features to unit test * Add helper method to beat.Info for returning FQDN-aware hostname * Adding license header to test file * Remove use of errors package * Check error from data.Put * Expire add_host_metadata cache immediately on FQDN feature flag value change * Fix casing in error message * Have callback registration function accept ID * Fixing go formatting * Account for feature flags in TestManagerV2 test * Update logging for unit changes * Remove TODO from wineventlog code * Updating dependency * Include shared file in docs * Bumping dependency on elastic-agent-client * Include features config in diagnostics output * Update comment * Updating NOTICE * Re-formatting * Running make update --------- Co-authored-by: Shaunak Kashyap <[email protected]> (cherry picked from commit faf4c11) # Conflicts: # go.mod # go.sum
* wip * wip * wip * something is broken * debug logs * avoid panics * wip broken * wip - perhaps fixed double configuring features * using triggers to know what changed * it works, but the agent-client is sending feature changes every time * debug logs * more debug logs * wip * it works * remove debug logs * fix after rebase * update elastic-agent-client and make notice * license headres * Add TODO for Winlogbeat FQDN implementation * Removing unnecesary code * Use RWMutex for greater read efficiency * Move mutex inside struct * Better error handling * Temporarily updating dependency * Always store FQDN in Beat info * Set hostname in monitoring registries after FQDN config has been evaluated * Pass down FQDN flag from add_host_metadata processor * Better error handling * Add temporary notice override * Changing log level * Update dependency on elastic-agent-system-metrics * Running go mod tidy * Update elastic-agent-system-metrics dependency version * Add feature flags configuration section to reference yml * Bump up version of elastic-agent-client depdendency * Updating reference file * Updating reference files * Move feature flags setting into same method where log level setting is handled * Updating docs * Adding CHANGELOG entry * Remove replace directive for go-sysinfo * Running go mod tidy * Remove replace directive for elastic-agent-client dependency * Remove personal repos from overrides.json * Updating NOTICE * Bump up version for elastic-agent-client dependency * Update NOTICE * Removing noisy debug logging * Removing commented out code that's not needed * Fixing error message * Make default flag setting more explicit * Adding config with no features to unit test * Add helper method to beat.Info for returning FQDN-aware hostname * Adding license header to test file * Remove use of errors package * Check error from data.Put * Expire add_host_metadata cache immediately on FQDN feature flag value change * Fix casing in error message * Have callback registration function accept ID * Fixing go formatting * Account for feature flags in TestManagerV2 test * Update logging for unit changes * Remove TODO from wineventlog code * Updating dependency * Include shared file in docs * Bumping dependency on elastic-agent-client * Include features config in diagnostics output * Update comment * Updating NOTICE * Re-formatting * Running make update --------- Co-authored-by: Shaunak Kashyap <[email protected]> (cherry picked from commit faf4c11) # Conflicts: # go.mod # go.sum
…flag (#34860) * Report FQDN on host.name enabled by a feature flag (#34456) * wip * wip * wip * something is broken * debug logs * avoid panics * wip broken * wip - perhaps fixed double configuring features * using triggers to know what changed * it works, but the agent-client is sending feature changes every time * debug logs * more debug logs * wip * it works * remove debug logs * fix after rebase * update elastic-agent-client and make notice * license headres * Add TODO for Winlogbeat FQDN implementation * Removing unnecesary code * Use RWMutex for greater read efficiency * Move mutex inside struct * Better error handling * Temporarily updating dependency * Always store FQDN in Beat info * Set hostname in monitoring registries after FQDN config has been evaluated * Pass down FQDN flag from add_host_metadata processor * Better error handling * Add temporary notice override * Changing log level * Update dependency on elastic-agent-system-metrics * Running go mod tidy * Update elastic-agent-system-metrics dependency version * Add feature flags configuration section to reference yml * Bump up version of elastic-agent-client depdendency * Updating reference file * Updating reference files * Move feature flags setting into same method where log level setting is handled * Updating docs * Adding CHANGELOG entry * Remove replace directive for go-sysinfo * Running go mod tidy * Remove replace directive for elastic-agent-client dependency * Remove personal repos from overrides.json * Updating NOTICE * Bump up version for elastic-agent-client dependency * Update NOTICE * Removing noisy debug logging * Removing commented out code that's not needed * Fixing error message * Make default flag setting more explicit * Adding config with no features to unit test * Add helper method to beat.Info for returning FQDN-aware hostname * Adding license header to test file * Remove use of errors package * Check error from data.Put * Expire add_host_metadata cache immediately on FQDN feature flag value change * Fix casing in error message * Have callback registration function accept ID * Fixing go formatting * Account for feature flags in TestManagerV2 test * Update logging for unit changes * Remove TODO from wineventlog code * Updating dependency * Include shared file in docs * Bumping dependency on elastic-agent-client * Include features config in diagnostics output * Update comment * Updating NOTICE * Re-formatting * Running make update --------- Co-authored-by: Shaunak Kashyap <[email protected]> (cherry picked from commit faf4c11) # Conflicts: # go.mod # go.sum * Remove conflict markers * Updating NOTICE.txt * Fixing dependency versions * Fix CHANGELOG --------- Co-authored-by: Anderson Queiroz <[email protected]> Co-authored-by: Shaunak Kashyap <[email protected]>
* wip * wip * wip * something is broken * debug logs * avoid panics * wip broken * wip - perhaps fixed double configuring features * using triggers to know what changed * it works, but the agent-client is sending feature changes every time * debug logs * more debug logs * wip * it works * remove debug logs * fix after rebase * update elastic-agent-client and make notice * license headres * Add TODO for Winlogbeat FQDN implementation * Removing unnecesary code * Use RWMutex for greater read efficiency * Move mutex inside struct * Better error handling * Temporarily updating dependency * Always store FQDN in Beat info * Set hostname in monitoring registries after FQDN config has been evaluated * Pass down FQDN flag from add_host_metadata processor * Better error handling * Add temporary notice override * Changing log level * Update dependency on elastic-agent-system-metrics * Running go mod tidy * Update elastic-agent-system-metrics dependency version * Add feature flags configuration section to reference yml * Bump up version of elastic-agent-client depdendency * Updating reference file * Updating reference files * Move feature flags setting into same method where log level setting is handled * Updating docs * Adding CHANGELOG entry * Remove replace directive for go-sysinfo * Running go mod tidy * Remove replace directive for elastic-agent-client dependency * Remove personal repos from overrides.json * Updating NOTICE * Bump up version for elastic-agent-client dependency * Update NOTICE * Removing noisy debug logging * Removing commented out code that's not needed * Fixing error message * Make default flag setting more explicit * Adding config with no features to unit test * Add helper method to beat.Info for returning FQDN-aware hostname * Adding license header to test file * Remove use of errors package * Check error from data.Put * Expire add_host_metadata cache immediately on FQDN feature flag value change * Fix casing in error message * Have callback registration function accept ID * Fixing go formatting * Account for feature flags in TestManagerV2 test * Update logging for unit changes * Remove TODO from wineventlog code * Updating dependency * Include shared file in docs * Bumping dependency on elastic-agent-client * Include features config in diagnostics output * Update comment * Updating NOTICE * Re-formatting * Running make update --------- Co-authored-by: Shaunak Kashyap <[email protected]>
What does this PR do?
This PR enhances Beats to accept feature flag configuration either in standalone or managed mode. It also enhances Beats to report their host's FQDN when the FQDN feature flag is set.
For Beats running in standalone mode, feature flag configuration is defined in the
<beat>.yml
configuration file as follows:For example, to run Metricbeat standalone with the FQDN feature flag enabled, configure the following section in
metricbeat.yml
:Why is it important?
To enable use cases, particularly involving running multiple Beats in a containerized environment, where it's useful for every Beat to report it's host's FQDN.
Checklist
CHANGELOG.next.asciidoc
orCHANGELOG-developer.next.asciidoc
.Author's Checklist
Effect on
add_host_metadata
processorWith FQDN feature flag disabled
beats_standalone_add_host_metadata_fqdn_disabled.mov
With FQDN feature flag enabled
beats_standalone_add_host_metadata_fqdn_enabled.mov
Effect on
http.enabled: true
settingWith FQDN feature flag disabled
beats_standalone_http_enabled_fqdn_disabled.mov
With FQDN feature flag enabled
beats_standalone_http_enabled_fqdn_enabled.mov
How to test this PR locally
See screen recordings in previous section.
Related issues
Use cases
Screenshots
Logs