Stage 2 changes for RFC 0009 - data_stream fields

[ebeahan]

Adding the data_stream fields from RFC 0009 (#1367 ) to the schema as beta fields.

ECS docs preview

[ebeahan]

@elasticmachine run elasticsearch-ci/docs

[ruflin]

I'm pretty excited to see this moving forward. @jpountz You were previously concerned about already moving this forward. You think we should still hold back on this or ok to get this shipped and improve in the future on top of it?

[ebeahan]

@ruflin Are we still waiting for any feedback before merging?

[ruflin]

Good to go on my end but I would like to have a 👍 or 👎 from @jpountz as he had initially some concerns.

[ebeahan]

Good to go on my end but I would like to have a 👍 or 👎 from @jpountz as he had initially some concerns.

Ok. The implementation here is based on the design decisions agreed upon in the RFC.

If there are still concerns, maybe we reconsider if the RFC is finished and reopen the discussion?

[jpountz]

The thing that's unclear to me is what GA means for ECS. I'm fine with GA-ing field names and what they mean. However there's an active discussion about changing the way that these fields are mapped to extract them dynamically from the name of the data stream rather than using constant_keyword and requiring data shippers to send these values as part of documents. Should we refrain from making these fields GA until this discussion settles?

[ebeahan]

The thing that's unclear to me is what GA means for ECS. I'm fine with GA-ing field names and what they mean.

Once fields are considered GA, apps/content that produce or consume ECS data can rely on those fields to remain stable with breaking changes only at major versions. ECS data producers can confidently populate, and consumers can operate against those fields.

However there's an active discussion about changing how these fields are mapped to extract them dynamically from the name of the data stream rather than using constant_keyword and requiring data shippers to send these values as part of documents. Should we refrain from making these fields GA until this discussion settles?

I believe the motivation for adding these fields into ECS is their importance in the recommended indexing strategy. If there's a change in direction that'll make those fields obsolete soon, I think we should hold off on GA-ing the data_stream.* fields.

Another option is to add the data_stream.* fields but mark them as beta in the docs (and roll back the RFC to stage 2).

[jpountz]

My perspective is that these fields will stay and that we won't break data consumers (like Kibana), however the changes we are considering might break data producers (like Agent). Adding them as beta first sounds good to me.

[ebeahan]

Adding them as beta first sounds good to me.

@ruflin thoughts on adding these data_stream fields as beta to give us more flexibility while discussion continues around a potentially different, future approach?

[ruflin]

I don't think we will have the option to break the data producers but instead I'm hoping the changes we are planning are somehow backward compatible. In any case, I'm good with putting it in beta as Elastic Agent is still in beta.

@ebeahan Anything I need to do on my end for the above?

[ebeahan]

I've updated the changes here to include the beta header on the fieldset (preview).

@ruflin Can you perform a final review here as well as on #1367?

[ruflin]

Only reviewed the beta stage again. Thanks @ebeahan for pushing this forward.