Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Beats use of non-ECS field agent.hostname causing user confusion #16377

Closed
4 tasks
MikePaquette opened this issue Feb 17, 2020 · 6 comments · Fixed by #18328
Closed
4 tasks

Beats use of non-ECS field agent.hostname causing user confusion #16377

MikePaquette opened this issue Feb 17, 2020 · 6 comments · Fixed by #18328
Assignees
@andrewkroh
Labels
ecs Filebeat Filebeat Team:Integrations Label for the Integrations team

Comments

@MikePaquette
Copy link

MikePaquette commented Feb 17, 2020 •
edited
Loading

During the introduction of ECS-support with the 7.0 release, some breaking changes were made in certain fields populated by beats.

Specifically, a new field agent.hostname was introduced - however this field is not an ECS field, and it's continued use and documentation are creating user confusion.

A recent SIEM discuss topic highlights the issue: 219088

Assumptions:

  • All module dashboards, apps, and other ECS data consumers should use the ECS field host.name to display general information about the host object.
  • In special cases, such as DNS, where the visualization is specifically looking for the hostname field, host.hostname may be used.
  • All beats that operate locally on a host should be populating host.name and host.hostname in their default configurations, either by enabling add_host_metadata, or some other means.
  • Beats that operate remotely, such as heartbeat, should be populating observer.hostname

Recommended changes:

  • Remove mention of agent.hostname from the beats docs and replace it with host.name and optionally host.hostname.
  • Change target of alias beat.hostname from agent.hostname to host.hostname in https://github.com/elastic/beats/blob/master/libbeat/_meta/fields.common.yml
  • Consider marking agent.hostname as deprecated and stop populating it in 8.0
  • Consider adding a new alias from agent.hostname to host.hostname once it is no longer populated, to enable any analysis content that used the agent.hostname field to continue to operate
@jsoriano
Copy link
Member

Should observer.hostname be used too for the hostname of beats that forward data from other hosts? (#13920)

@MikePaquette
Copy link
Author

@jsoriano Yes. (I tried to capture that in the 4th assumption bullet above).

@MikePaquette
Copy link
Author

There is a related behavior that is also causing problems for the user who posted in discuss 219088.

It appears that when the user sets the name: in the beats .yml file, that value not only gets populated to agent.name (expected), but also gets populated in host.name (not expected).

For example, this setting (happens to be in auditbeat.yml):
image

Results in this document being indexed in auditbeat-7.6.0-*:
image

In addition, there is an existing alias in https://github.com/elastic/beats/blob/master/libbeat/_meta/fields.common.yml that likewise seems to map the legacy beat.name to host.name. I think this should be mapped to agent.name.

image

@andresrc andresrc added the Team:Integrations Label for the Integrations team label Mar 12, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations (Team:Integrations)

@andrewkroh andrewkroh mentioned this issue Apr 26, 2020
Merged
5 tasks
@andrewkroh
Copy link
Member

We want to ensure that we continue to have an identifying name for the agent in events when we remove agent.hostname. So in #18000 I have modified the code to set agent.name to the hostname by default. It can be overwritten with a custom identifer by setting name in the config. This change will be merged in to 7.x too.

Then for master (8.0) I will remove the agent.hostname.

@andrewkroh andrewkroh assigned andrewkroh and unassigned exekias Apr 30, 2020
andrewkroh added a commit to andrewkroh/beats that referenced this issue May 2, 2020
@andrewkroh
Set agent.name to hostname by default
22e18a2 
Since ECS does not define agent.hostname it will be removed in a future release.
In order to always have a field available to identify the agent by name we will
set the agent.name to hostname unless the user has provided a custom name.

Relates elastic#16377
andrewkroh added a commit that referenced this issue May 2, 2020
@andrewkroh
Set agent.name to hostname by default (#18000)
3711ee6 
Since ECS does not define agent.hostname it will be removed in a future release.
In order to always have a field available to identify the agent by name we will
set the agent.name to hostname unless the user has provided a custom name.

Relates #16377
@andrewkroh andrewkroh mentioned this issue May 2, 2020
Merged
5 tasks
andrewkroh added a commit to andrewkroh/beats that referenced this issue May 2, 2020
@andrewkroh
Set agent.name to hostname by default (elastic#18000)
d341018 
Since ECS does not define agent.hostname it will be removed in a future release.
In order to always have a field available to identify the agent by name we will
set the agent.name to hostname unless the user has provided a custom name.

Relates elastic#16377

(cherry picked from commit 3711ee6)
andrewkroh added a commit that referenced this issue May 4, 2020
@andrewkroh
Set agent.name to hostname by default (#18000) (#18155)
e1ce07e 
Since ECS does not define agent.hostname it will be removed in a future release.
In order to always have a field available to identify the agent by name we will
set the agent.name to hostname unless the user has provided a custom name.

Relates #16377

(cherry picked from commit 3711ee6)
@sayden sayden mentioned this issue May 4, 2020
Closed
@andrewkroh andrewkroh mentioned this issue May 6, 2020
Merged
5 tasks
andrewkroh added a commit to andrewkroh/beats that referenced this issue May 6, 2020
@andrewkroh
Remove agent.hostname field from libbeat
4af663b 
agent.hostname is not part of ECS so this removes the field. Instead the
agent.name field can be used for this purpose and it it part of ECS.
This add a migration alias pointing from agent.hostname to agent.name.

Closes elastic#16377
@andrewkroh
Copy link
Member

I have open the PR to remove agent.hostname for 8.0. See #18328.

andrewkroh added a commit that referenced this issue May 12, 2020
@andrewkroh
Remove agent.hostname field from libbeat (#18328)
78b957a 
agent.hostname is not part of ECS so this removes the field. Instead the
agent.name field can be used for this purpose and it it part of ECS.
This adds an alias pointing from agent.hostname to agent.name.

Closes #16377
@zube zube bot added [zube]: Done and removed [zube]: Meta labels May 12, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@andrewkroh andrewkroh

Labels
ecs Filebeat Filebeat Team:Integrations Label for the Integrations team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Remove agent.hostname field from libbeat andrewkroh/beats
6 participants
@jsoriano @exekias @andresrc @andrewkroh @elasticmachine @MikePaquette