Add prefer_v2_templates flag and index setting #55411

dakrone · 2020-04-17T16:35:47Z

This commit adds a new querystring parameter on the following APIs:

Index
Update
Bulk
Create Index
Rollover

These APIs now support a ?prefer_v2_templates=true|false flag. This flag changes the preference
creation to use either V2 index templates or V1 templates. This flag defaults to false and will be
changed to true for 8.0+ in subsequent work.

Additionally, setting this flag internally sets the index.prefer_v2_templates index-level setting.
This setting is used so that actions that automatically create a new index (things like rollover
initiated by ILM) will inherit the preference from the original index. This setting is dynamic so
that a transition from v1 to v2 templates can occur for long-running indices grouped by an alias
performing periodic rollover.

This also adds support for sending this parameter to the High Level Rest Client.

Relates to #53101

This commit adds a new querystring parameter on the following APIs: - Index - Update - Bulk - Create Index - Rollover These APIs now support a `?prefer_v2_templates=true|false` flag. This flag changes the preference creation to use either V2 index templates or V1 templates. This flag defaults to `false` and will be changed to `true` for 8.0+ in subsequent work. Additionally, setting this flag internally sets the `index.prefer_v2_templates` index-level setting. This setting is used so that actions that automatically create a new index (things like rollover initiated by ILM) will inherit the preference from the original index. This setting is dynamic so that a transition from v1 to v2 templates can occur for longrunning indices grouped by an alias performing periodic rollover. This also adds support for sending this parameter to the High Level Rest Client. Relates to elastic#53101

elasticmachine · 2020-04-17T16:35:50Z

Pinging @elastic/es-core-features (:Core/Features/Indices APIs)

dakrone · 2020-04-17T16:37:49Z

This work does not yet change the default to true (will only be done for 8.0 in a subsequent PR). It also doesn't support reindex yet, as that is more complex to pass a flag through (again, will be done in subsequent work).

dakrone · 2020-04-17T17:46:46Z

@elasticmachine update branch

(will be adjusted back down later)

andreidan

This looks great Lee. Thanks for working on this.

I've just marked a few places where using the @nullable annotation would be a nice readability and IDE aid, super minor though. Nice work!

server/src/main/java/org/elasticsearch/rest/action/admin/indices/RestCreateIndexAction.java

client/rest-high-level/src/main/java/org/elasticsearch/client/indices/CreateIndexRequest.java

...java/org/elasticsearch/action/admin/indices/create/CreateIndexClusterStateUpdateRequest.java

server/src/main/java/org/elasticsearch/action/admin/indices/create/CreateIndexRequest.java

server/src/main/java/org/elasticsearch/action/bulk/BulkRequest.java

server/src/main/java/org/elasticsearch/action/index/IndexRequest.java

server/src/main/java/org/elasticsearch/action/update/UpdateRequest.java

andreidan · 2020-04-20T08:59:42Z

server/src/main/java/org/elasticsearch/cluster/metadata/MetadataCreateIndexService.java

@@ -345,6 +350,11 @@ public ClusterState applyCreateIndexRequest(ClusterState currentState, CreateInd
        }
    }

+    private static boolean resolvePreferV2Templates(CreateIndexClusterStateUpdateRequest request) {


this is pretty cool, aides readability quite a lot

dakrone · 2020-04-20T15:10:41Z

@elasticmachine update branch

This commit adds a new querystring parameter on the following APIs: - Index - Update - Bulk - Create Index - Rollover These APIs now support a `?prefer_v2_templates=true|false` flag. This flag changes the preference creation to use either V2 index templates or V1 templates. This flag defaults to `false` and will be changed to `true` for 8.0+ in subsequent work. Additionally, setting this flag internally sets the `index.prefer_v2_templates` index-level setting. This setting is used so that actions that automatically create a new index (things like rollover initiated by ILM) will inherit the preference from the original index. This setting is dynamic so that a transition from v1 to v2 templates can occur for long-running indices grouped by an alias performing periodic rollover. This also adds support for sending this parameter to the High Level Rest Client. Relates to elastic#53101

This adjusts the minimum version for serialization for elastic#55411. It should only be merged after elastic#55476 has been merged

This disableds BWC for the backport of elastic#55411 Backport: elastic#55476 Version adjustment: elastic#55478

This disables BWC for the backport of #55411 Backport: #55476 Version adjustment: #55478

This commit adds a new querystring parameter on the following APIs: - Index - Update - Bulk - Create Index - Rollover These APIs now support a `?prefer_v2_templates=true|false` flag. This flag changes the preference creation to use either V2 index templates or V1 templates. This flag defaults to `false` and will be changed to `true` for 8.0+ in subsequent work. Additionally, setting this flag internally sets the `index.prefer_v2_templates` index-level setting. This setting is used so that actions that automatically create a new index (things like rollover initiated by ILM) will inherit the preference from the original index. This setting is dynamic so that a transition from v1 to v2 templates can occur for long-running indices grouped by an alias performing periodic rollover. This also adds support for sending this parameter to the High Level Rest Client. Relates to #53101

This adjusts the minimum version for serialization for #55411. It should only be merged after #55476 has been merged

As a followup to elastic#55411, this commit changes the default for the `?prefer_v2_templates` querystring parameter to be `true`. This means that V2 templates will take precedence by default in 8.0+ Relates to elastic#53101

As a followup to #55411, this commit changes the default for the `?prefer_v2_templates` querystring parameter to be `true`. This means that V2 templates will take precedence by default in 8.0+ Relates to #53101

If some of the nodes are pre-7.8 nodes, they may not support the `index.prefer_v2_templates` setting (since it exists only on 7.8+). This commit makes sure that we only add this setting to the index's metadata if all of the nodes are 7.8+. Otherwise we get errors like: ``` [ WARN ][o.e.i.c.IndicesClusterStateService] [v7.7.0-3] [test-snapshot-index][2] marking and sending shard failed due to [failed to create index] java.lang.IllegalArgumentException: unknown setting [index.prefer_v2_templates] please check that any required plugins are installed, or check the breaking changes documentation for removed settings at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:544) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:460) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] at org.elasticsearch.indices.IndicesService.createIndexService(IndicesService.java:595) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:549) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:176) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] at org.elasticsearch.indices.cluster.IndicesClusterStateService.createIndices(IndicesClusterStateService.java:484) [elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:246) [elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$5(ClusterApplierService.java:517) [elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] ``` Relates to elastic#55411 Relates to elastic#53101 Resolves elastic#55539

…#55546) If some of the nodes are pre-7.8 nodes, they may not support the `index.prefer_v2_templates` setting (since it exists only on 7.8+). This commit makes sure that we only add this setting to the index's metadata if all of the nodes are 7.8+. Otherwise we get errors like: ``` [ WARN ][o.e.i.c.IndicesClusterStateService] [v7.7.0-3] [test-snapshot-index][2] marking and sending shard failed due to [failed to create index] java.lang.IllegalArgumentException: unknown setting [index.prefer_v2_templates] please check that any required plugins are installed, or check the breaking changes documentation for removed settings at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:544) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:460) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] at org.elasticsearch.indices.IndicesService.createIndexService(IndicesService.java:595) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:549) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:176) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] at org.elasticsearch.indices.cluster.IndicesClusterStateService.createIndices(IndicesClusterStateService.java:484) [elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:246) [elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$5(ClusterApplierService.java:517) [elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] ``` Relates to #55411 Relates to #53101 Resolves #55539

dakrone added :Data Management/Indices APIs APIs to create and manage indices and templates v8.0.0 v7.8.0 labels Apr 17, 2020

dakrone requested review from andreidan and probakowski April 17, 2020 16:35

martijnvg mentioned this pull request Apr 17, 2020

Composable Templates #53101

Closed

39 tasks

Don't replicate preference setting for CCR

3d50c15

elasticmachine and others added 2 commits April 17, 2020 13:46

Merge branch 'master' into itv2-add-prefer-param

01b7d7a

Adjust version for tests that use the prefer_v2_templates flag

0fd7a8f

(will be adjusted back down later)

ycombinator mentioned this pull request Apr 17, 2020

Support v2 templates in ES indexing calls elastic/beats#17809

Closed

dakrone added 2 commits April 19, 2020 21:02

Merge remote-tracking branch 'origin/master' into itv2-add-prefer-param

02f9cf6

Use setting in test that doesn't get overridden by global template

40d2965

andreidan approved these changes Apr 20, 2020

View reviewed changes

Add @nullable everywhere

944c50d

Merge branch 'master' into itv2-add-prefer-param

9e346ad

dakrone merged commit 0202e1a into elastic:master Apr 20, 2020

dakrone deleted the itv2-add-prefer-param branch April 20, 2020 16:04

dakrone added the backport pending label Apr 20, 2020

dakrone added a commit to dakrone/elasticsearch that referenced this pull request Apr 20, 2020

Adjust serialization versions for prefer_v2_templates flag

c32ba3b

This adjusts the minimum version for serialization for elastic#55411. It should only be merged after elastic#55476 has been merged

dakrone mentioned this pull request Apr 20, 2020

Adjust serialization versions for prefer_v2_templates flag #55478

Merged

dakrone added a commit to dakrone/elasticsearch that referenced this pull request Apr 20, 2020

Disable BWC for backporting prefer_v2_templates change

795daff

This disableds BWC for the backport of elastic#55411 Backport: elastic#55476 Version adjustment: elastic#55478

dakrone mentioned this pull request Apr 20, 2020

Disable BWC for backporting prefer_v2_templates change #55479

Merged

dakrone added a commit that referenced this pull request Apr 20, 2020

Disable BWC for backporting prefer_v2_templates change (#55479)

a407e7e

This disables BWC for the backport of #55411 Backport: #55476 Version adjustment: #55478

dakrone removed the backport pending label Apr 20, 2020

dakrone added a commit that referenced this pull request Apr 20, 2020

Adjust serialization versions for prefer_v2_templates flag (#55478)

93021f7

This adjusts the minimum version for serialization for #55411. It should only be merged after #55476 has been merged

dakrone mentioned this pull request Apr 20, 2020

Change prefer_v2_templates parameter to default to true #55489

Merged

iverase mentioned this pull request Apr 21, 2020

[CI] ':qa:mixed-cluster:v7.X.X#mixedClusterTest' failures #55539

Closed

dakrone mentioned this pull request Apr 21, 2020

Guard adding the index.prefer_v2_templates settings for pre-7.8 nodes #55546

Merged

dakrone mentioned this pull request May 4, 2020

[ILM] ilm-enabled indices use spamming log with messages "applying create index request using v1 templates" #55263

Closed

pugnascotia added the >enhancement label May 6, 2020

dakrone mentioned this pull request May 11, 2020

Remove prefer_v2_templates flag on indexing operations #56528

Closed

russcam mentioned this pull request May 29, 2020

7.8.0 Meta ticket elastic/elasticsearch-net#4718

Closed

17 tasks

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add prefer_v2_templates flag and index setting #55411

Add prefer_v2_templates flag and index setting #55411

dakrone commented Apr 17, 2020

elasticmachine commented Apr 17, 2020

dakrone commented Apr 17, 2020

dakrone commented Apr 17, 2020

andreidan left a comment

andreidan Apr 20, 2020

dakrone commented Apr 20, 2020

Add prefer_v2_templates flag and index setting #55411

Add prefer_v2_templates flag and index setting #55411

Conversation

dakrone commented Apr 17, 2020

elasticmachine commented Apr 17, 2020

dakrone commented Apr 17, 2020

dakrone commented Apr 17, 2020

andreidan left a comment

Choose a reason for hiding this comment

andreidan Apr 20, 2020

Choose a reason for hiding this comment

dakrone commented Apr 20, 2020