Skip to content

Commit

Permalink
Add template simulation API for simulating template composition
Browse files Browse the repository at this point in the history
This adds an API for simulating template composition with or without an index template.

It looks like:

```
POST /_index_template/_simulate/my-template
```

To simulate a template named `my-template` that already exists, or, to simulate a template that does
not already exist:

```
POST /_index_template/_simulate
{
  "index_patterns": ["my-index"]
  "composed_of": ["ct1", "ct2"],
}
```

This is related to elastic#55686, which adds an API to simulate composition based on an index name (hence
the `_simulate_index` vs `_simulate`).

This commit also adds reference documentation for both simulation APIs.

Relates to elastic#53101
Resolves elastic#56390
Resolves elastic#56255
  • Loading branch information
dakrone committed May 15, 2020
1 parent 500a726 commit 581d635
Show file tree
Hide file tree
Showing 11 changed files with 808 additions and 44 deletions.
92 changes: 91 additions & 1 deletion docs/reference/indices/index-templates.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ PUT _index_template/template_1
[source,console]
--------------------------------------------------
DELETE _index_template/template_*
DELETE _index_template/*
DELETE _component_template/*
--------------------------------------------------
// TEARDOWN
Expand Down Expand Up @@ -291,6 +291,96 @@ PUT /_index_template/template_1
In this case, an index matching `t*` will have three primary shards. If the order of composed
templates were reversed, the index would have two primary shards.


[[simulating-templates]]
===== Simulating template composition

Since templates can be composed not only of multiple component templates, but also the index
template itself, there are two simulation APIs to determine what the resulting index settings will
be.

To simulate the settings that would be applied to a matching index name:

[source,console]
--------------------------------------------------
POST /_index_template/_simulate_index/myindex
--------------------------------------------------

To simulate the settings that would be applied from a particular template:

[source,console]
--------------------------------------------------
POST /_index_template/_simulate/template_1
POST /_index_template/_simulate
{
"index_patterns": ["foo"],
"template": {
"settings": {
"number_of_replicas": 0
}
}
}
--------------------------------------------------


Here's an example demonstrating simulating both an index name and template name:

[source,console]
--------------------------------------------------
PUT /_component_template/ct1 <1>
{
"template": {
"settings": {
"index.number_of_shards": 2
}
}
}
PUT /_component_template/ct2 <2>
{
"template": {
"settings": {
"index.number_of_replicas": 0
},
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
}
}
}
}
}
PUT /_index_template/final-template <3>
{
"index_patterns": ["logdata-*"],
"composed_of": ["ct1", "ct2"],
"priority": 5
}
POST /_index_template/_simulate_index/logdata-2019-02-01 <4>
POST /_index_template/_simulate/final-template <5>
POST /_index_template/_simulate <6>
{
"index_patterns": ["mydata-*"],
"composed_of": ["ct2"],
"priority": 10
}
--------------------------------------------------
<1> Creating a component template (ct1) setting the number of shards to two
<2> Creating another component template (ct2) setting the number of replicas to zero with mappings
<3> Creating an index template called "final" template using ct1 and ct2
<4> Simulate the settings that would be applied for a new index "logdata-2019-02-01"
<5> Simulate the settings composed using the "final-template" index template
<6> Simulate the settings composed using a custom specified template

When simulating a template and specifying a template in the body of the request, the simulated
template is not added to the existing templates, it is only used for the simulation.

===== Index template with index aliases

You can include <<indices-aliases,index aliases>> in an index template.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
{
"indices.simulate_template":{
"documentation":{
"url":"https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-templates.html",
"description": "Simulate resolving the given template name or body"
},
"stability":"stable",
"url":{
"paths":[
{
"path":"/_index_template/_simulate",
"methods":[
"POST"
]
},
{
"path":"/_index_template/_simulate/{name}",
"methods":[
"POST"
],
"parts":{
"name":{
"type":"string",
"description":"The name of the index template"
}
}
}
]
},
"params":{
"create":{
"type":"boolean",
"description":"Whether the index template we optionally defined in the body should only be dry-run added if new or can also replace an existing one",
"default":false
},
"cause":{
"type":"string",
"description":"User defined reason for dry-run creating the new template for simulation purposes",
"default":false
},
"master_timeout":{
"type":"time",
"description":"Specify timeout for connection to master"
}
},
"body":{
"description":"New index template definition to be simulated, if no index template name is specified",
"required":false
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@

- match: {template.settings.index.number_of_shards: "1"}
- match: {template.settings.index.number_of_replicas: "0"}
- match: {template.mappings._doc.properties.field.type: "keyword"}
- match: {template.mappings.properties.field.type: "keyword"}
- match: {overlapping: []}

---
Expand Down Expand Up @@ -77,7 +77,7 @@

- match: {template.settings.index.blocks.write: "true"}
- match: {template.settings.index.number_of_replicas: "2"}
- match: {template.mappings._doc.properties.ct_field.type: "keyword"}
- match: {template.mappings.properties.ct_field.type: "keyword"}
- match: {overlapping.0.name: existing_test}
- match: {overlapping.0.index_patterns: ["te*"]}
- length: {template.aliases: 1}
Expand Down Expand Up @@ -170,7 +170,7 @@

- match: {template.settings.index.number_of_shards: "1"}
- match: {template.settings.index.number_of_replicas: "0"}
- match: {template.mappings._doc.properties.field.type: "keyword"}
- match: {template.mappings.properties.field.type: "keyword"}
- match: {overlapping.0.name: v1_template}
- match: {overlapping.0.index_patterns: ["t*", "t1*"]}
- match: {overlapping.1.name: v2_template}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
---
"Simulate template without a template in the body":
- skip:
version: " - 7.99.99"
reason: "not yet backported"
features: ["default_shards"]

- do:
indices.put_index_template:
name: my-template
body:
index_patterns: other
template:
settings:
number_of_shards: 1
number_of_replicas: 0
mappings:
properties:
field:
type: keyword

- do:
indices.simulate_template:
name: my-template

- match: {template.settings.index.number_of_shards: "1"}
- match: {template.settings.index.number_of_replicas: "0"}
- match: {template.mappings.properties.field.type: "keyword"}
- match: {overlapping: []}

---
"Simulate index template specifying a new template":
- skip:
version: " - 7.99.99"
reason: "not yet backported"
features: ["default_shards"]

- do:
indices.put_index_template:
name: existing_test
body:
index_patterns: te*
priority: 10
template:
settings:
number_of_shards: 1
number_of_replicas: 0
mappings:
properties:
field:
type: keyword

- do:
cluster.put_component_template:
name: ct
body:
template:
settings:
index.number_of_replicas: 2
mappings:
properties:
ct_field:
type: keyword

- do:
indices.simulate_template:
body:
index_patterns: te*
priority: 15
template:
settings:
index.blocks.write: true
aliases:
test_alias: {}
composed_of: ["ct"]

- match: {template.settings.index.blocks.write: "true"}
- match: {template.settings.index.number_of_replicas: "2"}
- match: {template.mappings.properties.ct_field.type: "keyword"}
- match: {overlapping.0.name: existing_test}
- match: {overlapping.0.index_patterns: ["te*"]}
- length: {template.aliases: 1}
- is_true: template.aliases.test_alias

---
"Simulate template matches overlapping V1 and V2 templates":
- skip:
version: " - 7.99.99"
reason: "not yet backported"
features: ["allowed_warnings", "default_shards"]

- do:
indices.put_template:
name: v1_template
body:
index_patterns: [t*, t1*]
settings:
number_of_shards: 5

- do:
allowed_warnings:
- "index template [v2_template] has index patterns [te*] matching patterns from existing older templates [v1_template] with patterns
(v1_template => [t*, t1*]); this template [v2_template] will take precedence during new index creation"
indices.put_index_template:
name: v2_template
body:
index_patterns: te*
priority: 10
template:
settings:
number_of_shards: 10
number_of_replicas: 2
mappings:
properties:
field:
type: text

- do:
allowed_warnings:
- "index template [winning_v2_template] has index patterns [te*] matching patterns from existing older templates [v1_template] with patterns
(v1_template => [t*, t1*]); this template [winning_v2_template] will take precedence during new index creation"
indices.put_index_template:
name: winning_v2_template
body:
index_patterns: te*
priority: 20
template:
settings:
number_of_shards: 1
number_of_replicas: 0
mappings:
properties:
field:
type: keyword

- do:
indices.simulate_template:
name: winning_v2_template

- match: {template.settings.index.number_of_shards: "1"}
- match: {template.settings.index.number_of_replicas: "0"}
- match: {template.mappings.properties.field.type: "keyword"}
- match: {overlapping.0.name: v1_template}
- match: {overlapping.0.index_patterns: ["t*", "t1*"]}
- match: {overlapping.1.name: v2_template}
- match: {overlapping.1.index_patterns: ["te*"]}
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,9 @@
import org.elasticsearch.action.admin.indices.template.get.TransportGetIndexTemplateV2Action;
import org.elasticsearch.action.admin.indices.template.get.TransportGetIndexTemplatesAction;
import org.elasticsearch.action.admin.indices.template.post.SimulateIndexTemplateAction;
import org.elasticsearch.action.admin.indices.template.post.SimulateTemplateAction;
import org.elasticsearch.action.admin.indices.template.post.TransportSimulateIndexTemplateAction;
import org.elasticsearch.action.admin.indices.template.post.TransportSimulateTemplateAction;
import org.elasticsearch.action.admin.indices.template.put.PutComponentTemplateAction;
import org.elasticsearch.action.admin.indices.template.put.PutIndexTemplateAction;
import org.elasticsearch.action.admin.indices.template.put.PutIndexTemplateV2Action;
Expand Down Expand Up @@ -313,7 +315,6 @@
import org.elasticsearch.rest.action.admin.indices.RestIndicesShardStoresAction;
import org.elasticsearch.rest.action.admin.indices.RestIndicesStatsAction;
import org.elasticsearch.rest.action.admin.indices.RestOpenIndexAction;
import org.elasticsearch.rest.action.admin.indices.RestSimulateIndexTemplateAction;
import org.elasticsearch.rest.action.admin.indices.RestPutComponentTemplateAction;
import org.elasticsearch.rest.action.admin.indices.RestPutIndexTemplateAction;
import org.elasticsearch.rest.action.admin.indices.RestPutIndexTemplateV2Action;
Expand All @@ -322,6 +323,8 @@
import org.elasticsearch.rest.action.admin.indices.RestRefreshAction;
import org.elasticsearch.rest.action.admin.indices.RestResizeHandler;
import org.elasticsearch.rest.action.admin.indices.RestRolloverIndexAction;
import org.elasticsearch.rest.action.admin.indices.RestSimulateIndexTemplateAction;
import org.elasticsearch.rest.action.admin.indices.RestSimulateTemplateAction;
import org.elasticsearch.rest.action.admin.indices.RestSyncedFlushAction;
import org.elasticsearch.rest.action.admin.indices.RestUpdateSettingsAction;
import org.elasticsearch.rest.action.admin.indices.RestUpgradeActionDeprecated;
Expand Down Expand Up @@ -540,6 +543,7 @@ public <Request extends ActionRequest, Response extends ActionResponse> void reg
actions.register(GetIndexTemplateV2Action.INSTANCE, TransportGetIndexTemplateV2Action.class);
actions.register(DeleteIndexTemplateV2Action.INSTANCE, TransportDeleteIndexTemplateV2Action.class);
actions.register(SimulateIndexTemplateAction.INSTANCE, TransportSimulateIndexTemplateAction.class);
actions.register(SimulateTemplateAction.INSTANCE, TransportSimulateTemplateAction.class);
actions.register(ValidateQueryAction.INSTANCE, TransportValidateQueryAction.class);
actions.register(RefreshAction.INSTANCE, TransportRefreshAction.class);
actions.register(FlushAction.INSTANCE, TransportFlushAction.class);
Expand Down Expand Up @@ -690,6 +694,7 @@ public void initRestHandlers(Supplier<DiscoveryNodes> nodesInCluster) {
registerHandler.accept(new RestGetIndexTemplateV2Action());
registerHandler.accept(new RestDeleteIndexTemplateV2Action());
registerHandler.accept(new RestSimulateIndexTemplateAction());
registerHandler.accept(new RestSimulateTemplateAction());

registerHandler.accept(new RestPutMappingAction());
registerHandler.accept(new RestGetMappingAction());
Expand Down
Loading

0 comments on commit 581d635

Please sign in to comment.