Fix PANW bad mapping of client/source and server/dest packets and bytes

[adriansr]

PANW pipeline was mapping:

Bytes sent to client/destination bytes.
Bytes received to server/source bytes.
Packets sent to server/destination packets.
Packets received to client/source packets.

All of these mappings are wrong.

The correct is:
Bytes sent to client/source bytes.
Bytes received to server/destination bytes.
Packets sent to client/source packets.
Packets received to server/destination packets.

Also adding some missing ECS mappings in the process.

Fixes #18522

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[adriansr]

Reviewed all the mappings just in case something else was off. Didn't find any other mapping error, but a few fields that can now be mapped to ECS. Can move that to a separate PR if we're not comfortable labeling those as a bug and backporting to a bugfix.

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request Fix PANW bad mapping of client/source and server/dest packets and bytes #18525 opened]

• Start Time: 2020-05-14T15:07:16.142+0000

• Duration: 69 min 51 sec (4130813)

Test stats 🧪

Test	Results
Failed	0
Passed	1184
Skipped	128
Total	1312

[andrewkroh]

FYI @willemdh. Please let us know if you see any issues in this change.

[andrewkroh]

LGTM. Thanks!

[willemdh]

@adriansr @andrewkroh Thanks for fixing this finally. One more issue in the panw module ingest pipeline:

event.end seems to be calculated in the ingest pipeline:

      {
        "script" : {
          "if" : "ctx?.event?.duration != null",
          "params" : {
            "NANOS_IN_A_SECOND" : 1000000000
          },
          "source" : """long nanos = ctx['event']['duration'] * params.NANOS_IN_A_SECOND; ctx['event']['duration'] = nanos; def start = ctx.event?.start; if (start != null) {
   ctx.event['end'] = ZonedDateTime.parse(start).plusNanos(nanos);
}
""",
          "lang" : "painless"
        }
      },

/usr/share/filebeat/module/panw/panos/config/input.yml

        event.start: 35
        event.duration: 36

Can't post live events here, but these are the raw values of one of our logs:

Generated Time (mgmt plane): 2020/05/14 20:22:32
Generated Time (data plane): 2020/05/14 20:22:32
Start Time => 2020/05/14 20:22:11
Duration => 4

In Kibana:

event.start => May 14, 2020 @ 20:22:11.000
event.duration => 4,000,000,000
event.end => May 14, 2020 @ 20:22:15.000

event.end is actually already in the raw data field 2 (mgmt) or 7 (data) (not sure which is best, they seem the same in events I checked), as the panw events are generated at the end of the session.

The calculated event.end (May 14, 2020 @ 20:22:15.000) seems to be different from the actual session event end (2020/05/14 20:22:32)..., resulting in more confusion..

Please verify this and fix this too.

Tx

[adriansr]

@willemdh I'm not sure I follow.

First, ECS recommends that event.start + event.duration == event.end (When both start and end are defined).

So our choices are either calculating event.end (current solution), or ignoring the provided duration.

To me, the generated times (data or mgmt) are not necessarily the same as event.end:

event.end contains the date when the event ended or when the activity was last observed.

Those are more suited to @timestamp:

This is the date/time extracted from the event, typically representing when the event was generated by the source.

Am I missing something? Maybe the PANW fields mean something different than what I understand?

[willemdh]

@adriansr Well our panw specialist pointed it out, I'll discuss it with him and let you know.