Add os.type field

[webmat]

See problem statement in #1110

Closes #1110

[webmat]

@randomuserid Does this satisfy the need?

[randomuserid]

@randomuserid Does this satisfy the need?

It should enable us to test for platform type such as Linux, Mac, or Windows. TBH I think using something simpler and easier to remember like host.os.name - which partially satisfies the need already - would be a better user experience. We have a lot of host.os fields now, some of which contain the same information, and if we could simply them, and consolidate them, I think it would be easier to use and also help reduce the compute and storage costs of processing all of these fields, and would tend to be a better UX then adding an eighth field to the set.

[webmat]

Agreed on the need for simplification. But this will lead to breaking changes and will therefore take more time to implement.

So this new field is the only way we can solve the immediate need and meet your deadline ;-)

[randomuserid]

Agreed on the need for simplification. But this will lead to breaking changes and will therefore take more time to implement.

So this new field is the only way we can solve the immediate need and meet your deadline ;-)

Right - what is the definition of a "breaking change?" - a change to the current de facto usage of a field? Including cases where a field has not defined legal values and is being populated with different unofficial values by different teams?

[randomuserid]

I think what we need to do, in order to avoid the same problem where different modules are placing different values in the field, is to adopt the design principle of normalization for fields that need to work across data streams. This would mean we explicitly define legal and expected values for the field, rather than giving examples or suggestions, and normalized fields would need to have defined legal values when they ship. Then search authors can look up the legal values when writing queries with the expectation they will work everywhere as long as legal values are being used. And data providers can be confident their events will be processed by queries using normalized fields.

[webmat]

@randomuserid I agree with the need to tighten things up for a few of the fields in the OS field set. This is something we're tracking on the list of breaking changes under consideration for the next major release of ECS (#839). We don't have an issue for this yet. But I think we're aligned: we need to clarify how to use these fields for each of the commercial OS family. We could even come out with this in ECS 1.x as a soft guidance, and make it a hard requirement in 8.0.

what is the definition of a "breaking change?"

The reason I'm hesitant to repurpose one of the existing fields for this mandated list of values is that we'll need to have this vetted by too many folks, and we likely wouldn't have much adoption anyway. It's not only about populating the field with these new values in the add_host_metadata processor. There's also potential impact on observability & security UIs, making this change in both user agent processors (LS & ES), there's likely Observability alerts that depend on OS details as well. And I'm sure I'm forgetting a bunch of things still :-)

Adding this new field, on the other hand, will mean it's either absent or adopted the right way when it is, during the 7.x line. So you would actually be able to depend on at the start of your predicate, then fall back to the current workaround e.g. "or redhat or debian" and so on.

[randomuserid]

Reluctantly approved due to time pressure. I'd rather wait and use host.os.name.

Actually, could we please get rid of the commercial_ and simplify it too host.os ?

[webmat]

As discussed out of band, we're changing the proposal to use a much shorter field name: os.type.

I like the symmetry with event.type, another field with a list of allowed value.

[ebeahan]

No issues with pivoting to os.type.

I made notes on a few small points.

[ebeahan]

LGTM 👍