[Packetbeat] - Change field names to follow ECS #7968

andrewkroh · 2018-08-15T01:02:28Z

For the next major release we want to change some of the common field names in Packetbeat to follow the Elastic Common Schema (ECS). This will make it much easier to query data related data from different sources.

We should break this task down into smaller pieces that are easily reviewable. Perhaps something like this.

Map the fields to ECS in a spreadsheet and review.
- Map the common fields that shared by all protocols (e.g. ip, client_ip, etc.)
- Map Packetbeat flow data
- Map the protocols that are commonly used in security analytics (e.g. dns, tls, http, icmp).
- Check the application protocols to see if there are any fields that can be mapped to ECS.
Convert one protocol at a time including updating dashboards.

Flows Convert Packetbeat Flows to ECS #9121
Process Info [Packetbeat] Restructure client/server and process fields #9303
Client / Server [Packetbeat] Restructure client/server and process fields #9303
Populated source and destination when client/server are used.
Network metrics
Direction (in/out -> incoming/outgoing)
Dashboards
Documentation examples
Evaluate what fields for 6.6 should receive forward looking aliases (like ip -> server.ip)

Protocol Change Pull Requests

AMQP Update AMQP protocol to use ECS fields #10090
Cassandra Update Cassandra protocol to use ECS fields #10093
DHCPv4 Update DHCPv4 protocol to use ECS fields #10089
DNS [Packetbeat] Update DNS protocol to use ECS fields #9941
- Removed trailing dot from domain names reported by the DNS protocol.
HTTP Update HTTP protocol to use ECS fields #9976
ICMP Update ICMP protocol to use ECS fields #10062
Memcache Update Memcache protocol to use ECS fields #10189
MongoDB Update MongoDB protocol to use ECS fields #10158
MySQL Update MySQL protocol to use ECS fields #10155
NFS Update NFS protocol to use ECS fields #10153
Redis Update Redis protocol to use ECS fields #10126
TLS Update TLS protocol to use ECS fields #9980
Thrift Update Thrift protocol to use ECS fields #10125
pgSQL Update pgSQL protocol to use ECS fields #10147

The text was updated successfully, but these errors were encountered:

The makes changes to the event format generated by Packetbeat's flow feature. Field Changes - type -> event.type - transport -> network.transport - flow_id -> flow.id - final -> flow.final - vlan -> flow.vlan - start_time -> event.start - last_time -> event.end - source.stats.net_bytes_total -> source.bytes - source.stats.net_packets_total -> source.packets - dest.stats.net_bytes_total -> destination.bytes - dest.stats.net_packets_total -> destination.packets Added - network.bytes - network.packets - event.duration Frames with multiple levels of encapsulation like 802.1q with "Q-in-Q" will result in certain fields becoming an array with the outer most metadata being listed first (e.g. source.ip, destination.ip, flow.vlan). Any dashboards associated with flows are not updated in this change. Part of elastic#7968.

* Update ECS fields Updating schema to commit 349406f59e5c7c80a20c9f213370d2601b73f040. Some fields were removed so I added placeholder fields with TODO statements. Those fields can be removed after modules using the fields are updated accordingly. * Convert Packetbeat Flows to ECS The makes changes to the event format generated by Packetbeat's flow feature. Field Changes - type -> event.type - transport -> network.transport - flow_id -> flow.id - final -> flow.final - vlan -> flow.vlan - start_time -> event.start - last_time -> event.end - source.stats.net_bytes_total -> source.bytes - source.stats.net_packets_total -> source.packets - dest.stats.net_bytes_total -> destination.bytes - dest.stats.net_packets_total -> destination.packets Added - network.bytes - network.packets - event.duration Frames with multiple levels of encapsulation like 802.1q with "Q-in-Q" will result in certain fields becoming an array with the outer most metadata being listed first (e.g. source.ip, destination.ip, flow.vlan). Any dashboards associated with flows are not updated in this change. Part of #7968.

This updates the DNS protocol to have more closely follow ECS. The DNS tunneling dashboard has been updated to work with the new field names. In order to better interoperate with other data sources the trailing dot has been removed from domain names. For example, previously Packetbeat would produce `dns.question.name:elastic.co.` and now it will simply produce `dns.question.name:elastic.co`. It's a breaking change but it will be make it easier to pivot with other data sources. Part of elastic#7968 Here's a summary of what fields changed. Changed - bytes_in -> source.bytes - bytes_out -> destination.bytes - notes -> error.message - responsetime -> event.duration (unit are now nanoseconds) - transport -> network.transport Added - event.end - event.start - network.bytes - network.community_id - network.protocol = dns - network.transport = udp/tcp - network.type Unchanged Packetbeat Fields - method - dns opcode - query = class {{ dns.question.class }}, type {{ dns.question.type }}, {{ dns.question.name }} - request - text representation of the entire request - response - text representation of the entire response - resource - dns.question.name - status - type = dns (we might remove this since we have event.dataset)

* Vendor github.com/elastic/ecs/code/go/ecs Update NOTICE * Update DNS protocol to use ECS fields This updates the DNS protocol to have more closely follow ECS. The DNS tunneling dashboard has been updated to work with the new field names. In order to better interoperate with other data sources the trailing dot has been removed from domain names. For example, previously Packetbeat would produce `dns.question.name:elastic.co.` and now it will simply produce `dns.question.name:elastic.co`. It's a breaking change but it will be make it easier to pivot with other data sources. Part of #7968 Here's a summary of what fields changed. Changed - bytes_in -> source.bytes - bytes_out -> destination.bytes - notes -> error.message - responsetime -> event.duration (unit are now nanoseconds) - transport -> network.transport Added - event.end - event.start - network.bytes - network.community_id - network.protocol = dns - network.transport = udp/tcp - network.type Unchanged Packetbeat Fields - method - dns opcode - query = class {{ dns.question.class }}, type {{ dns.question.type }}, {{ dns.question.name }} - request - text representation of the entire request - response - text representation of the entire response - resource - dns.question.name - status - type = dns (we might remove this since we have event.dataset) * Add packetbeat fields to ecs-migration.yml No aliases yet because these fields are still used. * Fix DNS overview dashboard There was an ID collision over the `DNS` ID. * Update dashboards.yml * Send zero value time.Duration With low resolution timers on Windows it's very possible that an event can have an event.duration=0 and we want to send that field when this occurs.

That dashboards were updated too. There weren't many changes needed w.r.t. fields but I did update the visualizations and saved searches to include `[Packetbeat]` in their names. I added a python test case for TLS and discovered a few fields that were not documented so I updated the fields docs accordingly. Here's a summary of what fields changed. Part of elastic#7968 Changed - responsetime -> event.duration (unit are now nanoseconds) Added - event.dataset = tls - event.end - event.start - network.community_id - network.protocol = tls - network.type - source.domain (added if there's a SNI value) Unchanged Packetbeat Fields - status - type = http (we might remove this since we have event.dataset)

* Update TLS protocol to use ECS fields That dashboards were updated too. There weren't many changes needed w.r.t. fields but I did update the visualizations and saved searches to include `[Packetbeat]` in their names. I added a python test case for TLS and discovered a few fields that were not documented so I updated the fields docs accordingly. Here's a summary of what fields changed. Part of #7968 Changed - responsetime -> event.duration (unit are now nanoseconds) Added - event.dataset = tls - event.end - event.start - network.community_id - network.protocol = tls - network.type - source.domain (added if there's a SNI value) Unchanged Packetbeat Fields - status - type = http (we might remove this since we have event.dataset) * Update TLS Session Resume to remove xxx-nope

Only a few fields were changed. No dashboards were changed because there are no ICMP specific dashboards. Here's a summary of what fields changed. Part of elastic#7968 Changed - responsetime -> event.duration (unit are now nanoseconds) - bytes_in -> source.bytes - bytes_out -> destination.bytes Added - event.dataset = icmp - event.end - event.start - network.community_id - network.transport = icmp or ipv6-icmp - network.type = ipv4/ipv6 Unchanged Packetbeat Fields - status - type = icmp (we might remove this since we have event.dataset) - path = destination.ip (what is requested, not sure if this still makes sense)

Here's a summary of what fields changed. Part of #7968 Changed - bytes_in -> source.bytes - bytes_out -> destination.bytes - responsetime -> event.duration (unit are now nanoseconds) Added - source - destination - event.dataset = redis - event.end - event.start - network.community_id - network.transport = tcp - network.protocol = redis - network.bytes - network.type Unchanged Packetbeat Fields - method - resource - path - query - status - type = redis (we might remove this since we have event.dataset)

That dashboard was updated too. Here's a summary of what fields changed. Part of #7968 Changed - rpc.call_size -> source.bytes - rpc.reply_size -> destination.bytes - rpc.time -> event.duration (unit are now nanoseconds) Added - destination - event.dataset = nfs - event.end - event.start - network.bytes - network.community_id - network.protocol = nfsv4 or nfsv3 - network.transport = tcp - network.type - source - source.domain - set to the caller's RPC machine name Removed - rpc.time_str - Kibana should provide a human-readable form for event.duration rather than each event supplying a string. Unchanged Packetbeat Fields - status - type = nfs (we might remove this since we have event.dataset)

That dashboard was updated too. Here's a summary of what fields changed. Part of #7968 Changed - bytes_in -> source.bytes - bytes_out -> destination.bytes - notes -> error.message - responsetime -> event.duration (unit are now nanoseconds) Added - destination - event.dataset = mysql - event.end - event.start - network.bytes - network.community_id - network.protocol = mysql - network.transport = tcp - network.type - source Removed - mysql.iserror - It was a boolean that reflected the same information as `status` which uses OK or Error. Unchanged Packetbeat Fields - method - query - params - path - status - type = mysql (we might remove this since we have event.dataset)

That dashboard was updated too. Here's a summary of what fields changed. Part of #7968 Changed - bytes_in -> source.bytes - bytes_out -> destination.bytes - responsetime -> event.duration (unit are now nanoseconds) Added - destination - event.dataset = mongodb - event.end - event.start - network.bytes - network.community_id - network.protocol = mongodb - network.transport = tcp - network.type - source Removed - mysql.iserror - It was a boolean that reflected the same information as `status` which uses OK or Error. Unchanged Packetbeat Fields - method - resource - query - status - type = mongodb (we might remove this since we have event.dataset)

Here's a summary of what fields changed. Part of elastic#7968 Changed - bytes_in -> source.bytes - bytes_out -> destination.bytes - responsetime -> event.duration (unit are now nanoseconds) - transport -> network.transport = udp or tcp Added - destination - event.dataset = memcache - event.end - event.start - network.bytes - network.community_id - network.protocol = memcache - network.type - source Unchanged Packetbeat Fields - status - type = memcache (we might remove this since we have event.dataset)

Here's a summary of what fields changed. Part of #7968 Changed - bytes_in -> source.bytes - bytes_out -> destination.bytes - responsetime -> event.duration (unit are now nanoseconds) - transport -> network.transport = udp or tcp Added - destination - event.dataset = memcache - event.end - event.start - network.bytes - network.community_id - network.protocol = memcache - network.type - source Unchanged Packetbeat Fields - status - type = memcache (we might remove this since we have event.dataset)

That dashboard was updated too. Here's a summary of what fields changed. Part of elastic#7968 Changed - bytes_in -> source.bytes - bytes_out -> destination.bytes - notes -> error.message - responsetime -> event.duration (unit are now nanoseconds) Added - destination - event.dataset = pgsql - event.end - event.start - network.bytes - network.community_id - network.protocol = pgsql - network.transport = tcp - network.type - source Removed - pgsql.iserror - It was a boolean that reflected the same information as `status` which uses OK or Error. - pgsql.query - It was in fields.yml but not in the code. Unchanged Packetbeat Fields - method - query - status - type = pgsql (we might remove this since we have event.dataset)

* Update pgSQL protocol to use ECS fields That dashboard was updated too. Here's a summary of what fields changed. Part of #7968 Changed - bytes_in -> source.bytes - bytes_out -> destination.bytes - notes -> error.message - responsetime -> event.duration (unit are now nanoseconds) Added - destination - event.dataset = pgsql - event.end - event.start - network.bytes - network.community_id - network.protocol = pgsql - network.transport = tcp - network.type - source Removed - pgsql.iserror - It was a boolean that reflected the same information as `status` which uses OK or Error. - pgsql.query - It was in fields.yml but not in the code. Unchanged Packetbeat Fields - method - query - status - type = pgsql (we might remove this since we have event.dataset) * add event.end and event.duration * Add PR number to changelog * Remove time_zone from Kibana viz

andrewkroh · 2019-01-24T00:48:41Z

@MikePaquette @webmat I have not added event.kind, event.category, or event.action fields to anything in Packetbeat.

I'm assuming we will want:

event.kind = event
event.category = network_traffic

For event.action I'm not sure what to use. Do we want something that's protocol specific like "dns-query", "http-post", "myqsl-select"?

webmat · 2019-01-24T03:16:28Z

I haven't had time to look into event categorization much, yet. What's important for GA is to get all the breaking changes squared away. By this I mean the field name changes and type changes.

Event categorization can be considered gradual additions, during the 7.x line. Conversely, rushing to get an answer for these fields may be annoying later. Not a huge breaking change if an event used to be event.category:network_traffic and becomes event.category:flow, but a breaking change nonetheless.

The exception may be event.kind, where Packetbeat should have event across the board. But the other two, my recommendation is to wait.

MikePaquette · 2019-01-24T13:14:54Z

@andrewkroh @webmat

I am 👍 for event.kind = event
I am also in favor of going ahead and populating event.category = network_traffic. Even though we warn users that we may define values for this field, I know I'l be pushing for "network_traffic" to be one of those values.
Also in favor of populating event.action. I was thinking something more generic, but I like your idea of the protocol-specific values. (although the field is named event.action, we are typically using it more like as if it were event.activity) and those protocol specific values would fit. Did @adriansr populate these fields in the filebeat netflow input? We should be consistent.

webmat · 2019-01-24T14:55:41Z

Yeah the consistency is specifically why I say we should wait after 7.0 FF, when we have the time to have a look around, consider all of our existing modules / sources, as well as others we expect to support shortly.

Adding those after 7.0 is perfectly fine. Changing the value after the fact (if we get it wrong in a rush) is a breaking change (although a small one).

- Add aliases - Clean up config files - Remove unused fields from fields.common.yml - Remove time_zone from dashboards - Update navigation links on dashboards to include all dashboards - Add DNS / TLS to overview dashboard to highlight the capabilities - Update fields used in the documentation - Move RPC fields to NFS from Mongo (they were in the wrong package) Part of elastic#7968

- Add aliases - Clean up config files - Remove unused fields from fields.common.yml - Remove time_zone from dashboards - Update navigation links on dashboards to include all dashboards - Add DNS / TLS to overview dashboard to highlight the capabilities - Update fields used in the documentation - Move RPC fields to NFS from Mongo (they were in the wrong package) - Add scale to responsetime - Consolidate changelog entries to point to issue Part of #7968

Part of elastic#7968 Adds event.kind = event and event.category = network_traffic to all Packetbeat events. Packetbeat flow events will additional have event.action = network_flow (same as Filebeat netflow). This also does some cleanup of redundant and unused code that resulted from the ECS migration.

andrewkroh · 2019-01-28T03:03:30Z

I opened #10357 to add:

event.kind = event
event.category = network_traffic (already used by Filebeat netflow)
event.action = network_flow - for Packetbeat flow events only (already used by Filebeat netflow)

We can add event.action to other protocols later.

MikePaquette · 2019-01-28T11:35:18Z

👍

* Add event.kind and event.category Part of #7968 Adds event.kind = event and event.category = network_traffic to all Packetbeat events. Packetbeat flow events will additional have event.action = network_flow (same as Filebeat netflow). This also does some cleanup of redundant and unused code that resulted from the ECS migration. * Set method for http

* Update ECS fields Updating schema to commit 349406f59e5c7c80a20c9f213370d2601b73f040. Some fields were removed so I added placeholder fields with TODO statements. Those fields can be removed after modules using the fields are updated accordingly. * Convert Packetbeat Flows to ECS The makes changes to the event format generated by Packetbeat's flow feature. Field Changes - type -> event.type - transport -> network.transport - flow_id -> flow.id - final -> flow.final - vlan -> flow.vlan - start_time -> event.start - last_time -> event.end - source.stats.net_bytes_total -> source.bytes - source.stats.net_packets_total -> source.packets - dest.stats.net_bytes_total -> destination.bytes - dest.stats.net_packets_total -> destination.packets Added - network.bytes - network.packets - event.duration Frames with multiple levels of encapsulation like 802.1q with "Q-in-Q" will result in certain fields becoming an array with the outer most metadata being listed first (e.g. source.ip, destination.ip, flow.vlan). Any dashboards associated with flows are not updated in this change. Part of elastic#7968.

andrewkroh added enhancement Packetbeat labels Aug 15, 2018

webmat mentioned this issue Sep 18, 2018

Getting ECS to 1.0 elastic/ecs#115

Closed

26 tasks

andrewkroh added the SecOps label Oct 29, 2018

ruflin mentioned this issue Oct 31, 2018

Beta1 related but not blocker tasks elastic/ecs#161

Closed

39 tasks

webmat added the ecs label Nov 13, 2018

This was referenced Nov 30, 2018

[Packetbeat] Restructure client/server and process fields #9303

Merged

Convert Packetbeat Flows to ECS #9121

Merged

ruflin mentioned this issue Dec 4, 2018

Transition Beats to ECS #8655

Closed

andrewkroh mentioned this issue Jan 8, 2019

[Packetbeat] Update DNS protocol to use ECS fields #9941

Merged

andrewkroh mentioned this issue Jan 10, 2019

Update TLS protocol to use ECS fields #9980

Merged

andrewkroh mentioned this issue Jan 14, 2019

Update ICMP protocol to use ECS fields #10062

Merged

andrewkroh mentioned this issue Jan 18, 2019

Update Memcache protocol to use ECS fields #10189

Merged

andrewkroh mentioned this issue Jan 19, 2019

Finish ECS migration for Packetbeat #10193

Merged

andrewkroh mentioned this issue Jan 28, 2019

Add event.kind and event.category #10357

Merged

andrewkroh closed this as completed Jan 30, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Packetbeat] - Change field names to follow ECS #7968

[Packetbeat] - Change field names to follow ECS #7968

andrewkroh commented Aug 15, 2018 •

edited

Loading

andrewkroh commented Jan 24, 2019

webmat commented Jan 24, 2019 •

edited

Loading

MikePaquette commented Jan 24, 2019

webmat commented Jan 24, 2019

andrewkroh commented Jan 28, 2019

MikePaquette commented Jan 28, 2019

[Packetbeat] - Change field names to follow ECS #7968

[Packetbeat] - Change field names to follow ECS #7968

Comments

andrewkroh commented Aug 15, 2018 • edited Loading

andrewkroh commented Jan 24, 2019

webmat commented Jan 24, 2019 • edited Loading

MikePaquette commented Jan 24, 2019

webmat commented Jan 24, 2019

andrewkroh commented Jan 28, 2019

MikePaquette commented Jan 28, 2019

andrewkroh commented Aug 15, 2018 •

edited

Loading

webmat commented Jan 24, 2019 •

edited

Loading