Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Filebeat] Allow modules to set the ecs.version field value #17688

Closed
andrewkroh opened this issue Apr 13, 2020 · 7 comments · Fixed by #19159
Closed

[Filebeat] Allow modules to set the ecs.version field value #17688

andrewkroh opened this issue Apr 13, 2020 · 7 comments · Fixed by #19159
Assignees
@leehinman
Labels
ecs Filebeat Filebeat needs_team Indicates that the issue/PR needs a Team:* label

Comments

@andrewkroh
Copy link
Member

Describe the enhancement:

As a user of Filebeat modules I would like the ECS version number (ecs.version) to reflect what version the data from the module comforms to rather than what version of the schema has been imported by libbeat.

Describe a specific use case for the enhancement or feature:

It's helpful to have an accurate ecs.version field in the data when you are trying to build queries for the data. You might need one query for earlier versions and then can use a generic query for the later versions that have categories and types.

It will also serve as documentation of when the module was last updated for ECS.

Relates: #16089
@andrewkroh
Copy link
Member Author

I verified that modules can set ecs.version without having libbeat overwrite the value. It could be as simple as putting an add_fields processor into the module's config with the ECS version. This works:

diff --git a/x-pack/filebeat/module/panw/panos/config/input.yml b/x-pack/filebeat/module/panw/panos/config/input.yml
index 929237b99..a1aeda8f5 100644
--- a/x-pack/filebeat/module/panw/panos/config/input.yml
+++ b/x-pack/filebeat/module/panw/panos/config/input.yml
@@ -18,6 +18,11 @@ exclude_files: [".gz$"]
 tags: {{.tags}}
 
 processors:
+  - add_fields:
+      target: ecs
+      fields:
+        version: 1.6.0-beta
+
   - add_locale: ~
 
   - decode_csv_fields:

@leehinman What do you think of that approach?

@leehinman
Copy link
Contributor

I like it I'll add to all the filesets I've updated.

@andrewkroh
Copy link
Member Author

I want to expand this issue to include explicitly setting the ecs.version for Packetbeat, Auditbeat, and Winlogbeat.

  • Winlobeat Security Modules
  • Winlogbeat Sysmon Module
  • Packetbeat - We could do this for Packetbeat as whole by using custom modifier rather than WithECS. See example.
  • Auditbeat - Do the same thing as Packetbeat.

@andrewkroh andrewkroh mentioned this issue Jun 12, 2020
Merged
@andrewkroh
Copy link
Member Author

I have opened #19159 to cover Auditbeat, Winlogbeat, Packetbeat.

@andrewkroh
Copy link
Member Author

@leehinman The Filebeat part of this issue is complete, right?

@leehinman leehinman mentioned this issue Jun 15, 2020
Merged
5 tasks
leehinman added a commit to leehinman/beats that referenced this issue Jun 15, 2020
@leehinman
Explicitly set ECS version in Filebeat modules.
7fb13c5 
- Add test to check if ecs.version is set
- add_fields to azure/activitylogs
- add_fields to azure/auditlogs
- add_fields to azure/signinlogs
- add_fields to checkpoint/firewall
- add_fields to crowdstrike/falcon
- add_fields to fortinet/firewall
- add_fields to traefik/access

Relates elastic#17688
@leehinman
Copy link
Contributor

@leehinman The Filebeat part of this issue is complete, right?

Once #19198 is merged.

leehinman added a commit that referenced this issue Jun 16, 2020
@leehinman
[Filebeat] Explicitly set ECS version in Filebeat modules. (#19198)
81b0c3a 
* Explicitly set ECS version in Filebeat modules.

- Add test to check if ecs.version is set
- add_fields to azure/activitylogs
- add_fields to azure/auditlogs
- add_fields to azure/signinlogs
- add_fields to checkpoint/firewall
- add_fields to crowdstrike/falcon
- add_fields to fortinet/firewall
- add_fields to traefik/access

Relates #17688
leehinman added a commit to leehinman/beats that referenced this issue Jun 16, 2020
@leehinman
[Filebeat] Explicitly set ECS version in Filebeat modules. (elastic#1…
77515f7 
…9198)

* Explicitly set ECS version in Filebeat modules.

- Add test to check if ecs.version is set
- add_fields to azure/activitylogs
- add_fields to azure/auditlogs
- add_fields to azure/signinlogs
- add_fields to checkpoint/firewall
- add_fields to crowdstrike/falcon
- add_fields to fortinet/firewall
- add_fields to traefik/access

Relates elastic#17688

(cherry picked from commit 81b0c3a)
@leehinman leehinman mentioned this issue Jun 16, 2020
Merged
5 tasks
leehinman added a commit that referenced this issue Jun 25, 2020
@leehinman
[Filebeat] Explicitly set ECS version in Filebeat modules. (#19198) (#…
b85a764 
…19223)

* Explicitly set ECS version in Filebeat modules.

- Add test to check if ecs.version is set
- add_fields to azure/activitylogs
- add_fields to azure/auditlogs
- add_fields to azure/signinlogs
- add_fields to checkpoint/firewall
- add_fields to crowdstrike/falcon
- add_fields to fortinet/firewall
- add_fields to traefik/access

Relates #17688

(cherry picked from commit 81b0c3a)
andrewkroh added a commit to andrewkroh/beats that referenced this issue Jul 13, 2020
@andrewkroh
Specify an ECS version in Auditbeat/Packetbeat/Winlogbeat
ae6beb1 
When we update the Beat (include all of its modules) we will then bump the ECS
version that it includes in events.

I went for a less granular approach than what is being used in Filebeat because
I think it's desirable to move a whole beat to a new ECS version "at once" and
more realistic to do so with these Beats that have fewer updates. By "at once" I
mean we won't release a version that is partially updated. This implies that if
we will be making multiple commits that we should use a feature branch to
ensure the update is atomic.

Closes elastic#17688
andrewkroh added a commit that referenced this issue Jul 13, 2020
@andrewkroh
Specify an ECS version in Auditbeat/Packetbeat/Winlogbeat (#19159)
256b50d 
When we update the Beat (include all of its modules) we will then bump the ECS
version that it includes in events.

I went for a less granular approach than what is being used in Filebeat because
I think it's desirable to move a whole beat to a new ECS version "at once" and
more realistic to do so with these Beats that have fewer updates. By "at once" I
mean we won't release a version that is partially updated. This implies that if
we will be making multiple commits that we should use a feature branch to
ensure the update is atomic.

Closes #17688
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 13, 2020
@botelastic
Copy link

botelastic bot commented Jul 13, 2020

This issue doesn't have a Team:<team> label.

andrewkroh added a commit to andrewkroh/beats that referenced this issue Jul 13, 2020
@andrewkroh
Specify an ECS version in Auditbeat/Packetbeat/Winlogbeat (elastic#19159
f171021 
)

When we update the Beat (include all of its modules) we will then bump the ECS
version that it includes in events.

I went for a less granular approach than what is being used in Filebeat because
I think it's desirable to move a whole beat to a new ECS version "at once" and
more realistic to do so with these Beats that have fewer updates. By "at once" I
mean we won't release a version that is partially updated. This implies that if
we will be making multiple commits that we should use a feature branch to
ensure the update is atomic.

Closes elastic#17688

(cherry picked from commit 256b50d)
@andrewkroh andrewkroh mentioned this issue Jul 13, 2020
Merged
andrewkroh added a commit that referenced this issue Jul 14, 2020
@andrewkroh
Specify an ECS version in Auditbeat/Packetbeat/Winlogbeat (#19159) (#…
09daf92 
…19862)

When we update the Beat (include all of its modules) we will then bump the ECS
version that it includes in events.

I went for a less granular approach than what is being used in Filebeat because
I think it's desirable to move a whole beat to a new ECS version "at once" and
more realistic to do so with these Beats that have fewer updates. By "at once" I
mean we won't release a version that is partially updated. This implies that if
we will be making multiple commits that we should use a feature branch to
ensure the update is atomic.

Closes #17688

(cherry picked from commit 256b50d)
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this issue Oct 14, 2020
@leehinman @melchiormoulin
[Filebeat] Explicitly set ECS version in Filebeat modules. (elastic#1…
7712029 
…9198)

* Explicitly set ECS version in Filebeat modules.

- Add test to check if ecs.version is set
- add_fields to azure/activitylogs
- add_fields to azure/auditlogs
- add_fields to azure/signinlogs
- add_fields to checkpoint/firewall
- add_fields to crowdstrike/falcon
- add_fields to fortinet/firewall
- add_fields to traefik/access

Relates elastic#17688
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this issue Oct 14, 2020
@andrewkroh @melchiormoulin
Specify an ECS version in Auditbeat/Packetbeat/Winlogbeat (elastic#19159
adbd6c7 
)

When we update the Beat (include all of its modules) we will then bump the ECS
version that it includes in events.

I went for a less granular approach than what is being used in Filebeat because
I think it's desirable to move a whole beat to a new ECS version "at once" and
more realistic to do so with these Beats that have fewer updates. By "at once" I
mean we won't release a version that is partially updated. This implies that if
we will be making multiple commits that we should use a feature branch to
ensure the update is atomic.

Closes elastic#17688
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@leehinman leehinman

Labels
ecs Filebeat Filebeat needs_team Indicates that the issue/PR needs a Team:* label
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Specify an ECS version in Auditbeat/Packetbeat/Winlogbeat andrewkroh/beats
3 participants
@andresrc @andrewkroh @leehinman