Revises event.created and expands with event.start and event.end

[robgil]

While trying to map VPC FlowLogs I was finding that event.created didn't really align with the start and end of a flow. This PR adds event.start and event.end.

event.created could still be used with regard to events in general. So for example if there is a threat feed match, we could create an event, and the time the event was created would relate to the event, but not the source of the event itself. So for example, VPC FlowLogs and Netflow for that matter only ever log after the flow is complete. This means that the event.created would land after the start of the flow. While this may not be ideal, this is what I'm trying to delineate between - the time it was determined it was a security event, and when the event actually happened.

A more concrete example.

I have a netflow flow that lasted 36 hours. The time at which it was considered an event was after the flow completed, but the event.start and event.end would map to when the flow started and ended. event.duration would be the following..

event.end - event.start = event.duration.

[ruflin]

Agree, I think there is value in all 3 timestamps.

There is a general requirement to have different timestamps. One proposal I made before the repository was public looked as following:

{
    "event": {
        "timestamp": {
            "created": 2018-01-09T06:12:31.669Z,
            "sent": 2018-01-09T06:14:31.669Z,
            "stored": 2018-01-09T06:15:31.669Z,
            "processed": 2018-01-09T06:16:31.669Z
        }
    }
}

The above would basically be an object to store the different types of timestamps and add their own but I never fully liked the above proposal but thought it's worth sharing here.

As start/end is also something common in APM I looked at what they do for transactions. It seems they track start and duration so start + duration = end.

[ruflin]

For me that is event.created.

[webmat]

I would not explain event.start in terms of the phenomenon of reading log files.

I think it would be more beneficial to specifically explain start/end specifically in terms of the observation of long running phenomena, like flows.

Some points I'd try to hit in this description and/or end:

• start and end usually go hand in hand. Although there may be exceptions (missing field) in the case of partial flows.
• These are to describe potentially long running events. start can actually be hours before @timestamp, if the observed phenomenon started a long time ago and was just reported.
• I would relate this description to event.duration.

To @ruflin's point, having 2/3 fields out of start, end and duration is enough to derive the 3rd fields. I'd rather have 3/3, but we may (or may not) want to mention that 2/3 can be enough?

[ruflin]

You have a nice description in the PR of this with an example. Could you add this here?

[webmat]

Also added a few comments on the descriptions.

[webmat]

"package capturing" => "packet capture" (same correction on line below as well)

[webmat]

I would not explain event.start in terms of the phenomenon of reading log files.

I think it would be more beneficial to specifically explain start/end specifically in terms of the observation of long running phenomena, like flows.

Some points I'd try to hit in this description and/or end:

• start and end usually go hand in hand. Although there may be exceptions (missing field) in the case of partial flows.
• These are to describe potentially long running events. start can actually be hours before @timestamp, if the observed phenomenon started a long time ago and was just reported.
• I would relate this description to event.duration.

To @ruflin's point, having 2/3 fields out of start, end and duration is enough to derive the 3rd fields. I'd rather have 3/3, but we may (or may not) want to mention that 2/3 can be enough?

[webmat]

Closing this, as it was introduced in PR #185 instead. Please let us know if you think the descriptions aren't descriptive enough, or if anything else is missing