Add field type for version strings #59773

cbuescher · 2020-07-17T13:12:17Z

This PR adds a new 'version' field type that allows indexing string values
representing software versions similar to the ones defined in the Semantic
Versioning definition (semver.org). The field behaves very similar to a
'keyword' field but allows efficient sorting and range queries that take into
accound the special ordering needed for version strings. For example, the main
version parts are sorted numerically (ie 2.0.0 < 11.0.0) whereas this wouldn't
be possible with 'keyword' fields today.

Valid version values are similar to the Semantic Versioning definition, with the
notable exception that in addition to the "main" version consiting of
major.minor.patch, we allow less or more than three numeric identifiers, i.e.
"1.2" or "1.4.6.123.12" are treated as valid too.

Relates to #48878

elasticmachine · 2020-07-17T13:12:19Z

Pinging @elastic/es-search (:Search/Mapping)

This PR adds a new 'version' field type that allows indexing string values representing software versions similar to the ones defined in the Semantic Versioning definition (semver.org). The field behaves very similar to a 'keyword' field but allows efficient sorting and range queries that take into accound the special ordering needed for version strings. For example, the main version parts are sorted numerically (ie 2.0.0 < 11.0.0) whereas this wouldn't be possible with 'keyword' fields today. Valid version values are similar to the Semantic Versioning definition, with the notable exception that in addition to the "main" version consiting of major.minor.patch, we allow less or more than three numeric identifiers, i.e. "1.2" or "1.4.6.123.12" are treated as valid too. Relates to elastic#48878

...ersionfield/src/main/java/org/elasticsearch/xpack/versionfield/VersionStringFieldMapper.java

jtibshirani

I was getting up to speed with the feature so I could help with reviews, and left some small comments. I haven't yet reviewed in detail though.

...k/plugin/versionfield/src/main/java/org/elasticsearch/xpack/versionfield/VersionEncoder.java

...ugin/versionfield/src/main/java/org/elasticsearch/xpack/versionfield/VersionFieldPlugin.java

...gin/versionfield/src/test/java/org/elasticsearch/xpack/versionfield/VersionEncoderTests.java

...ersionfield/src/main/java/org/elasticsearch/xpack/versionfield/VersionStringFieldMapper.java

...k/plugin/versionfield/src/main/java/org/elasticsearch/xpack/versionfield/VersionEncoder.java

...versionfield/src/test/java/org/elasticsearch/xpack/versionfield/VersionStringFieldTests.java

...rsionfield/src/main/java/org/elasticsearch/xpack/versionfield/VersionFieldWildcardQuery.java

cbuescher · 2020-09-16T16:58:50Z

@mayya-sharipova thanks, I hope I adressed your last comments in ce51484

mayya-sharipova

@cbuescher Thanks for all these iterations of addressing feedback and thanks for a great field type, this PR LGTM.

I have just left small comments

.../versionfield/src/main/java/org/elasticsearch/xpack/versionfield/VersionScriptDocValues.java

...ersionfield/src/main/java/org/elasticsearch/xpack/versionfield/VersionStringFieldMapper.java

jimczi

The change looks good to me. I left one comment regarding the new script functions that I think should be discussed separately. I also wonder what family type this field should expose but that can also be decide in a follow up. So +1 to merge the current state since the main functionality we're after is there and consistent (accurate sort and range queries on Semver versions).

docs/reference/mapping/types/version.asciidoc

jimczi · 2020-09-18T13:34:47Z

...ersionfield/src/main/java/org/elasticsearch/xpack/versionfield/VersionStringFieldMapper.java

+
+        @Override
+        public Query prefixQuery(String value, MultiTermQuery.RewriteMethod method, QueryShardContext context) {
+            if (context.allowExpensiveQueries() == false) {


Can you remove the check please

...in/versionfield/src/main/java/org/elasticsearch/xpack/versionfield/ValidationOnSortedDv.java

cbuescher · 2020-09-21T09:10:54Z

@mayya-sharipova @jimczi thanks for the reviews, I will work on backporting and open issues for follow ups where needed.

This PR adds a new 'version' field type that allows indexing string values representing software versions similar to the ones defined in the Semantic Versioning definition (semver.org). The field behaves very similar to a 'keyword' field but allows efficient sorting and range queries that take into accound the special ordering needed for version strings. For example, the main version parts are sorted numerically (ie 2.0.0 < 11.0.0) whereas this wouldn't be possible with 'keyword' fields today. Valid version values are similar to the Semantic Versioning definition, with the notable exception that in addition to the "main" version consiting of major.minor.patch, we allow less or more than three numeric identifiers, i.e. "1.2" or "1.4.6.123.12" are treated as valid too. Relates to elastic#48878

This PR adds a new 'version' field type that allows indexing string values representing software versions similar to the ones defined in the Semantic Versioning definition (semver.org). The field behaves very similar to a 'keyword' field but allows efficient sorting and range queries that take into accound the special ordering needed for version strings. For example, the main version parts are sorted numerically (ie 2.0.0 < 11.0.0) whereas this wouldn't be possible with 'keyword' fields today. Valid version values are similar to the Semantic Versioning definition, with the notable exception that in addition to the "main" version consiting of major.minor.patch, we allow less or more than three numeric identifiers, i.e. "1.2" or "1.4.6.123.12" are treated as valid too. Relates to #48878

cbuescher added >enhancement :Search Foundations/Mapping Index mappings, including merging and defining field types v8.0.0 v7.10.0 labels Jul 17, 2020

elasticmachine added the Team:Search Meta label for search team label Jul 17, 2020

cbuescher mentioned this pull request Jul 17, 2020

WIP First draft for version field #58256

Closed

cbuescher force-pushed the add-version-field branch from 43d470b to 2cd8f1d Compare July 17, 2020 13:41

cbuescher mentioned this pull request Jul 17, 2020

WIP Add 'version_range' field #59786

Closed

This was referenced Jul 17, 2020

Adopting new version field type elastic/ecs#887

Closed

Split ecs.version into parts elastic/ecs#842

Closed

cbuescher force-pushed the add-version-field branch from 2cd8f1d to 343310c Compare July 20, 2020 09:00

Christoph Büscher added 2 commits July 20, 2020 19:26

Add yaml for search, ranges and scripting

397a7c0

Add brute force regex query support

08989e7

cbuescher force-pushed the add-version-field branch from 9b249a1 to 08989e7 Compare July 22, 2020 17:18

Christoph Büscher added 4 commits July 23, 2020 15:13

Add fuzzy query support

8a1acc2

Merge branch 'master' into add-version-field

bc972ad

Remove some left-over commented code

5bac785

Adding tests

8a7b43c

mayya-sharipova self-requested a review July 23, 2020 19:24

mayya-sharipova reviewed Jul 27, 2020

View reviewed changes

...ersionfield/src/main/java/org/elasticsearch/xpack/versionfield/VersionStringFieldMapper.java Outdated Show resolved Hide resolved

mayya-sharipova reviewed Jul 27, 2020

View reviewed changes

...ersionfield/src/main/java/org/elasticsearch/xpack/versionfield/VersionStringFieldMapper.java Show resolved Hide resolved

mayya-sharipova reviewed Jul 27, 2020

View reviewed changes

...ersionfield/src/main/java/org/elasticsearch/xpack/versionfield/VersionStringFieldMapper.java Show resolved Hide resolved

jtibshirani reviewed Jul 27, 2020

View reviewed changes

Christoph Büscher added 2 commits July 28, 2020 15:01

Make string_stats work

910616e

Merge branch 'master' into add-version-field

6d71b52

mayya-sharipova reviewed Jul 28, 2020

View reviewed changes

...k/plugin/versionfield/src/main/java/org/elasticsearch/xpack/versionfield/VersionEncoder.java Show resolved Hide resolved

mayya-sharipova reviewed Jul 28, 2020

View reviewed changes

...k/plugin/versionfield/src/main/java/org/elasticsearch/xpack/versionfield/VersionEncoder.java Show resolved Hide resolved

Adressing review comments

970085a

cbuescher force-pushed the add-version-field branch from f1bf52f to 67df1d3 Compare September 15, 2020 12:46

mayya-sharipova reviewed Sep 15, 2020

View reviewed changes

...k/plugin/versionfield/src/main/java/org/elasticsearch/xpack/versionfield/VersionEncoder.java Outdated Show resolved Hide resolved

alisonelizabeth mentioned this pull request Sep 15, 2020

[Mappings editor] Add UI form for version field type elastic/kibana#77490

Closed

mayya-sharipova reviewed Sep 15, 2020

View reviewed changes

...versionfield/src/test/java/org/elasticsearch/xpack/versionfield/VersionStringFieldTests.java Show resolved Hide resolved

mayya-sharipova reviewed Sep 15, 2020

View reviewed changes

...rsionfield/src/main/java/org/elasticsearch/xpack/versionfield/VersionFieldWildcardQuery.java Outdated Show resolved Hide resolved

Another review iteration

ce51484

mayya-sharipova approved these changes Sep 17, 2020

View reviewed changes

Christoph Büscher added 2 commits September 17, 2020 18:12

Another round of reviews

04ac11d

Moving docs under 'structured'

ccebd6f

cbuescher mentioned this pull request Sep 18, 2020

Investigate ways for faster fuzzy queries on version fields #62608

Closed

jimczi approved these changes Sep 18, 2020

View reviewed changes

Christoph Büscher added 3 commits September 18, 2020 16:21

Merge branch 'master' into add-version-field

b0abc83

iter

4c0b1e9

Removing specialized script doc values functions

6849550

cbuescher merged commit ea2dbd9 into elastic:master Sep 21, 2020

cbuescher added the backport pending label Sep 21, 2020

cbuescher removed the backport pending label Sep 21, 2020

alisonelizabeth mentioned this pull request Sep 22, 2020

[Mappings editor] Add support for version field type elastic/kibana#78206

Merged

This was referenced Sep 28, 2020

Easier access to details from version field #62952

Closed

Adding script support for version parts #62953

Closed

Mpdreamz mentioned this pull request Nov 16, 2020

7.10.1 Meta Ticket elastic/elasticsearch-net#5096

Closed

61 tasks

stevejgordon mentioned this pull request Dec 17, 2020

7.11.0 Meta Ticket elastic/elasticsearch-net#5198

Closed

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

rudolf mentioned this pull request Feb 1, 2022

EQL support for fields of type version #83375

Closed

cbuescher mentioned this pull request Feb 7, 2022

Should we move the "version" field under the "keyword" family type #83594

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add field type for version strings #59773

Add field type for version strings #59773

cbuescher commented Jul 17, 2020

elasticmachine commented Jul 17, 2020

jtibshirani left a comment

cbuescher commented Sep 16, 2020

mayya-sharipova left a comment

jimczi left a comment

jimczi Sep 18, 2020

cbuescher commented Sep 21, 2020

Add field type for version strings #59773

Add field type for version strings #59773

Conversation

cbuescher commented Jul 17, 2020

elasticmachine commented Jul 17, 2020

jtibshirani left a comment

Choose a reason for hiding this comment

cbuescher commented Sep 16, 2020

mayya-sharipova left a comment

Choose a reason for hiding this comment

jimczi left a comment

Choose a reason for hiding this comment

jimczi Sep 18, 2020

Choose a reason for hiding this comment

cbuescher commented Sep 21, 2020