Adopting new version field type

[ebeahan]

Overview

ECS plans to adopt the upcoming version field type (elastic/elasticsearch#48878). The feature is still under development (elastic/elasticsearch#59773) for Elasticsearch, and this discussion focuses on how to include it into the ECS spec.

Problem

Today ECS captures versions in the .version fields indexed as type keyword. While using semantic versioning has become a widely adopted versioning scheme, there are many various versioning schemes that ECS needs to consider and support. As type keyword, version fields support exact matching and wildcard query support. Also, as a string field, keyword allows for flexibility in capturing various versioning schemes and formats.

Adhering to the semver specification, version numbers follow a convention which is easy to implement and easy for both producers and consumers of software understand how one version relates to another. However, more complicated queries seeking common questions around .version fields are not always straight-forward to devise with .version fields indexed as keyword.

Examples:

• Return documents with .version greater-than or equal to 1.1.0 and less-than 2.0.0
• Query across multiple versions without the need to know in advance ever possible versions in the index (e.g. ecs.version:(1.3.0 OR 1.4.0 OR 1.5.0)
• Return all documents where minor version is 3 (x.3.y) or the major version is considered unstable (0.x.y).

Proposal

Current optimal approach would adopt the new datatype to existing .version fields as a multi-field and silently ignore field values that are not semver. This allows adding the query, agg, and sorting benefits of the new type without introducing a breaking change into the schema.

---
- name: ecs
...
  fields:

    - name: version
      level: core
      type: keyword
...
      example: 1.0.0
      multi_field:
      - type: version
        name: version

[webmat]

Actually my understanding is that version will be in the keyword type family. So similar to wildcard, I think the optimal solution would be to directly change the existing field types to version, rather than adding a multi-field.

The data type will come with a bunch of niceties for version numbers that follow semver (even loosely), and I'm pushing to ensure that event parsing failures will still let users filter for exact matches on strangely formatted values.

[ebeahan]

I hadn't seen that version will be in the keyword family as well. If that's the case, I agree we should swap the existing field type over the multi-field.

[ebeahan]

This was proposed in RFC 0020, but it isn't being actively pursued at this time.