diff --git a/CHANGELOG.next.md b/CHANGELOG.next.md index 3a0c7e7476..58eafff020 100644 --- a/CHANGELOG.next.md +++ b/CHANGELOG.next.md @@ -16,6 +16,7 @@ Thanks, you're awesome :-) --> #### Added +* Add `data_stream` fieldset. #1307 * Add `orchestrator` fieldset as beta fields. #1326 * Extend `threat.*` experimental fields with proposed changes from RFC 0018. #1344, #1351 diff --git a/code/go/ecs/data_stream.go b/code/go/ecs/data_stream.go new file mode 100644 index 0000000000..e641fbb514 --- /dev/null +++ b/code/go/ecs/data_stream.go @@ -0,0 +1,67 @@ +// Licensed to Elasticsearch B.V. under one or more contributor +// license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright +// ownership. Elasticsearch B.V. licenses this file to you under +// the Apache License, Version 2.0 (the "License"); you may +// not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +// Code generated by scripts/gocodegen.go - DO NOT EDIT. + +package ecs + +// The data_stream fields take part in defining the new data stream naming +// scheme. +// In the new data stream naming scheme the value of the data stream fields +// combine to the name of the actual data stream in the following manner: +// `{data_stream.type}-{data_stream.dataset}-{data_stream.namespace}`. This +// means the fields can only contain characters that are valid as part of names +// of data streams. More details about this can be found in this +// https://www.elastic.co/blog/an-introduction-to-the-elastic-data-stream-naming-scheme[blog +// post]. +// An Elasticsearch data stream consists of one or more backing indices, and a +// data stream name forms part of the backing indices names. Due to this +// convention, data streams must also follow index naming restrictions. For +// example, data stream names cannot include `\`, `/`, `*`, `?`, `"`, `<`, `>`, +// `|`, ` ` (space character), `,`, or `#`. Please see the Elasticsearch +// reference for additional +// https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#indices-create-api-path-params[restrictions]. +type DataStream struct { + // An overarching type for the data stream. + // Currently allowed values are "logs" and "metrics". We expect to also add + // "traces" and "synthetics" in the near future. + Type string `ecs:"type"` + + // The field can contain anything that makes sense to signify the source of + // the data. + // Examples include `nginx.access`, `prometheus`, `endpoint` etc. For data + // streams that otherwise fit, but that do not have dataset set we use the + // value "generic" for the dataset value. `event.dataset` should have the + // same value as `data_stream.dataset`. + // Beyond the Elasticsearch data stream naming criteria noted above, the + // `dataset` value has additional restrictions: + // * Must not contain `-` + // * No longer than 100 characters + Dataset string `ecs:"dataset"` + + // A user defined namespace. Namespaces are useful to allow grouping of + // data. + // Many users already organize their indices this way, and the data stream + // naming scheme now provides this best practice as a default. Many users + // will populate this field with `default`. If no value is used, it falls + // back to `default`. + // Beyond the Elasticsearch index naming criteria noted above, `namespace` + // value has the additional restrictions: + // * Must not contain `-` + // * No longer than 100 characters + Namespace string `ecs:"namespace"` +} diff --git a/docs/field-details.asciidoc b/docs/field-details.asciidoc index 2cc663d86e..52ed080c6d 100644 --- a/docs/field-details.asciidoc +++ b/docs/field-details.asciidoc @@ -1017,6 +1017,94 @@ example: `docker` |===== +[[ecs-data_stream]] +=== Data Stream Fields + +The data_stream fields take part in defining the new data stream naming scheme. + +In the new data stream naming scheme the value of the data stream fields combine to the name of the actual data stream in the following manner: `{data_stream.type}-{data_stream.dataset}-{data_stream.namespace}`. This means the fields can only contain characters that are valid as part of names of data streams. More details about this can be found in this https://www.elastic.co/blog/an-introduction-to-the-elastic-data-stream-naming-scheme[blog post]. + +An Elasticsearch data stream consists of one or more backing indices, and a data stream name forms part of the backing indices names. Due to this convention, data streams must also follow index naming restrictions. For example, data stream names cannot include `\`, `/`, `*`, `?`, `"`, `<`, `>`, `|`, ` ` (space character), `,`, or `#`. Please see the Elasticsearch reference for additional https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#indices-create-api-path-params[restrictions]. + +beta::[ These fields are in beta and are subject to change.] + +[discrete] +==== Data Stream Field Details + +[options="header"] +|===== +| Field | Description | Level + +// =============================================================== + +| +[[field-data-stream-dataset]] +<> + +| The field can contain anything that makes sense to signify the source of the data. + +Examples include `nginx.access`, `prometheus`, `endpoint` etc. For data streams that otherwise fit, but that do not have dataset set we use the value "generic" for the dataset value. `event.dataset` should have the same value as `data_stream.dataset`. + +Beyond the Elasticsearch data stream naming criteria noted above, the `dataset` value has additional restrictions: + + * Must not contain `-` + + * No longer than 100 characters + +type: constant_keyword + + + +example: `nginx.access` + +| extended + +// =============================================================== + +| +[[field-data-stream-namespace]] +<> + +| A user defined namespace. Namespaces are useful to allow grouping of data. + +Many users already organize their indices this way, and the data stream naming scheme now provides this best practice as a default. Many users will populate this field with `default`. If no value is used, it falls back to `default`. + +Beyond the Elasticsearch index naming criteria noted above, `namespace` value has the additional restrictions: + + * Must not contain `-` + + * No longer than 100 characters + +type: constant_keyword + + + +example: `production` + +| extended + +// =============================================================== + +| +[[field-data-stream-type]] +<> + +| An overarching type for the data stream. + +Currently allowed values are "logs" and "metrics". We expect to also add "traces" and "synthetics" in the near future. + +type: constant_keyword + + + +example: `logs` + +| extended + +// =============================================================== + +|===== + [[ecs-destination]] === Destination Fields diff --git a/docs/fields.asciidoc b/docs/fields.asciidoc index 7b901e7aa9..a9adf96872 100644 --- a/docs/fields.asciidoc +++ b/docs/fields.asciidoc @@ -32,6 +32,8 @@ all fields are defined. | <> | Fields describing the container that generated this event. +| <> | The data_stream fields take part in defining the new data stream naming scheme. + | <> | Fields about the destination side of a network connection, used with source. | <> | These fields contain information about code libraries dynamically loaded into processes. diff --git a/experimental/generated/beats/fields.ecs.yml b/experimental/generated/beats/fields.ecs.yml index 99d7c8f951..f2c68b783f 100644 --- a/experimental/generated/beats/fields.ecs.yml +++ b/experimental/generated/beats/fields.ecs.yml @@ -632,7 +632,7 @@ naming scheme. In the new data stream naming scheme the value of the data stream fields combine - to the name of the actual data stream in the following manner `{data_stream.type}-{data_stream.dataset}-{data_stream.namespace}`. + to the name of the actual data stream in the following manner: `{data_stream.type}-{data_stream.dataset}-{data_stream.namespace}`. This means the fields can only contain characters that are valid as part of names of data streams. More details about this can be found in this https://www.elastic.co/blog/an-introduction-to-the-elastic-data-stream-naming-scheme[blog post]. @@ -640,8 +640,8 @@ An Elasticsearch data stream consists of one or more backing indices, and a data stream name forms part of the backing indices names. Due to this convention, data streams must also follow index naming restrictions. For example, data stream - names cannot include \, /, *, ?, ", <, >, |, ` `. Please see the Elasticsearch - reference for additional https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#indices-create-api-path-params[restrictions].' + names cannot include `\`, `/`, `*`, `?`, `"`, `<`, `>`, `|`, ` ` (space character), + `,`, or `#`. Please see the Elasticsearch reference for additional https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#indices-create-api-path-params[restrictions].' type: group fields: - name: dataset diff --git a/experimental/generated/ecs/ecs_nested.yml b/experimental/generated/ecs/ecs_nested.yml index ea594fdb32..be5465f8fd 100644 --- a/experimental/generated/ecs/ecs_nested.yml +++ b/experimental/generated/ecs/ecs_nested.yml @@ -1057,11 +1057,12 @@ container: title: Container type: group data_stream: + beta: These fields are in beta and are subject to change. description: 'The data_stream fields take part in defining the new data stream naming scheme. In the new data stream naming scheme the value of the data stream fields combine - to the name of the actual data stream in the following manner `{data_stream.type}-{data_stream.dataset}-{data_stream.namespace}`. + to the name of the actual data stream in the following manner: `{data_stream.type}-{data_stream.dataset}-{data_stream.namespace}`. This means the fields can only contain characters that are valid as part of names of data streams. More details about this can be found in this https://www.elastic.co/blog/an-introduction-to-the-elastic-data-stream-naming-scheme[blog post]. @@ -1069,8 +1070,8 @@ data_stream: An Elasticsearch data stream consists of one or more backing indices, and a data stream name forms part of the backing indices names. Due to this convention, data streams must also follow index naming restrictions. For example, data stream names - cannot include \, /, *, ?, ", <, >, |, ` `. Please see the Elasticsearch reference - for additional https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#indices-create-api-path-params[restrictions].' + cannot include `\`, `/`, `*`, `?`, `"`, `<`, `>`, `|`, ` ` (space character), + `,`, or `#`. Please see the Elasticsearch reference for additional https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#indices-create-api-path-params[restrictions].' fields: data_stream.dataset: dashed_name: data-stream-dataset diff --git a/experimental/generated/elasticsearch/template.json b/experimental/generated/elasticsearch/template.json index e67083a166..5775a87263 100644 --- a/experimental/generated/elasticsearch/template.json +++ b/experimental/generated/elasticsearch/template.json @@ -9,6 +9,7 @@ "ecs_2.0.0-dev-exp_client", "ecs_2.0.0-dev-exp_cloud", "ecs_2.0.0-dev-exp_container", + "ecs_2.0.0-dev-exp_data_stream", "ecs_2.0.0-dev-exp_destination", "ecs_2.0.0-dev-exp_dll", "ecs_2.0.0-dev-exp_dns", @@ -38,8 +39,7 @@ "ecs_2.0.0-dev-exp_url", "ecs_2.0.0-dev-exp_user", "ecs_2.0.0-dev-exp_user_agent", - "ecs_2.0.0-dev-exp_vulnerability", - "ecs_2.0.0-dev-exp_data_stream" + "ecs_2.0.0-dev-exp_vulnerability" ], "index_patterns": [ "try-ecs-*" diff --git a/generated/beats/fields.ecs.yml b/generated/beats/fields.ecs.yml index 76d2fed2f3..3067eab765 100644 --- a/generated/beats/fields.ecs.yml +++ b/generated/beats/fields.ecs.yml @@ -634,6 +634,58 @@ ignore_above: 1024 description: Runtime managing this container. example: docker + - name: data_stream + title: Data Stream + group: 2 + description: 'The data_stream fields take part in defining the new data stream + naming scheme. + + In the new data stream naming scheme the value of the data stream fields combine + to the name of the actual data stream in the following manner: `{data_stream.type}-{data_stream.dataset}-{data_stream.namespace}`. + This means the fields can only contain characters that are valid as part of + names of data streams. More details about this can be found in this https://www.elastic.co/blog/an-introduction-to-the-elastic-data-stream-naming-scheme[blog + post]. + + An Elasticsearch data stream consists of one or more backing indices, and a + data stream name forms part of the backing indices names. Due to this convention, + data streams must also follow index naming restrictions. For example, data stream + names cannot include `\`, `/`, `*`, `?`, `"`, `<`, `>`, `|`, ` ` (space character), + `,`, or `#`. Please see the Elasticsearch reference for additional https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#indices-create-api-path-params[restrictions].' + type: group + fields: + - name: dataset + level: extended + type: constant_keyword + description: "The field can contain anything that makes sense to signify the\ + \ source of the data.\nExamples include `nginx.access`, `prometheus`, `endpoint`\ + \ etc. For data streams that otherwise fit, but that do not have dataset set\ + \ we use the value \"generic\" for the dataset value. `event.dataset` should\ + \ have the same value as `data_stream.dataset`.\nBeyond the Elasticsearch\ + \ data stream naming criteria noted above, the `dataset` value has additional\ + \ restrictions:\n * Must not contain `-`\n * No longer than 100 characters" + example: nginx.access + default_field: false + - name: namespace + level: extended + type: constant_keyword + description: "A user defined namespace. Namespaces are useful to allow grouping\ + \ of data.\nMany users already organize their indices this way, and the data\ + \ stream naming scheme now provides this best practice as a default. Many\ + \ users will populate this field with `default`. If no value is used, it falls\ + \ back to `default`.\nBeyond the Elasticsearch index naming criteria noted\ + \ above, `namespace` value has the additional restrictions:\n * Must not\ + \ contain `-`\n * No longer than 100 characters" + example: production + default_field: false + - name: type + level: extended + type: constant_keyword + description: 'An overarching type for the data stream. + + Currently allowed values are "logs" and "metrics". We expect to also add "traces" + and "synthetics" in the near future.' + example: logs + default_field: false - name: destination title: Destination group: 2 diff --git a/generated/csv/fields.csv b/generated/csv/fields.csv index 9a6affed78..2ba05d75be 100644 --- a/generated/csv/fields.csv +++ b/generated/csv/fields.csv @@ -64,6 +64,9 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description 2.0.0-dev,true,container,container.labels,object,extended,,,Image labels. 2.0.0-dev,true,container,container.name,keyword,extended,,,Container name. 2.0.0-dev,true,container,container.runtime,keyword,extended,,docker,Runtime managing this container. +2.0.0-dev,true,data_stream,data_stream.dataset,constant_keyword,extended,,nginx.access,The field can contain anything that makes sense to signify the source of the data. +2.0.0-dev,true,data_stream,data_stream.namespace,constant_keyword,extended,,production,A user defined namespace. Namespaces are useful to allow grouping of data. +2.0.0-dev,true,data_stream,data_stream.type,constant_keyword,extended,,logs,An overarching type for the data stream. 2.0.0-dev,true,destination,destination.address,keyword,extended,,,Destination network address. 2.0.0-dev,true,destination,destination.as.number,long,extended,,15169,Unique number allocated to the autonomous system. 2.0.0-dev,true,destination,destination.as.organization.name,keyword,extended,,Google LLC,Organization name. diff --git a/generated/ecs/ecs_flat.yml b/generated/ecs/ecs_flat.yml index 66e5e9fb09..3e8772e2b0 100644 --- a/generated/ecs/ecs_flat.yml +++ b/generated/ecs/ecs_flat.yml @@ -756,6 +756,52 @@ container.runtime: normalize: [] short: Runtime managing this container. type: keyword +data_stream.dataset: + dashed_name: data-stream-dataset + description: "The field can contain anything that makes sense to signify the source\ + \ of the data.\nExamples include `nginx.access`, `prometheus`, `endpoint` etc.\ + \ For data streams that otherwise fit, but that do not have dataset set we use\ + \ the value \"generic\" for the dataset value. `event.dataset` should have the\ + \ same value as `data_stream.dataset`.\nBeyond the Elasticsearch data stream naming\ + \ criteria noted above, the `dataset` value has additional restrictions:\n *\ + \ Must not contain `-`\n * No longer than 100 characters" + example: nginx.access + flat_name: data_stream.dataset + level: extended + name: dataset + normalize: [] + short: The field can contain anything that makes sense to signify the source of + the data. + type: constant_keyword +data_stream.namespace: + dashed_name: data-stream-namespace + description: "A user defined namespace. Namespaces are useful to allow grouping\ + \ of data.\nMany users already organize their indices this way, and the data stream\ + \ naming scheme now provides this best practice as a default. Many users will\ + \ populate this field with `default`. If no value is used, it falls back to `default`.\n\ + Beyond the Elasticsearch index naming criteria noted above, `namespace` value\ + \ has the additional restrictions:\n * Must not contain `-`\n * No longer than\ + \ 100 characters" + example: production + flat_name: data_stream.namespace + level: extended + name: namespace + normalize: [] + short: A user defined namespace. Namespaces are useful to allow grouping of data. + type: constant_keyword +data_stream.type: + dashed_name: data-stream-type + description: 'An overarching type for the data stream. + + Currently allowed values are "logs" and "metrics". We expect to also add "traces" + and "synthetics" in the near future.' + example: logs + flat_name: data_stream.type + level: extended + name: type + normalize: [] + short: An overarching type for the data stream. + type: constant_keyword destination.address: dashed_name: destination-address description: 'Some event destination addresses are defined ambiguously. The event diff --git a/generated/ecs/ecs_nested.yml b/generated/ecs/ecs_nested.yml index 9d0461cc38..b0958cccce 100644 --- a/generated/ecs/ecs_nested.yml +++ b/generated/ecs/ecs_nested.yml @@ -1062,6 +1062,76 @@ container: short: Fields describing the container that generated this event. title: Container type: group +data_stream: + beta: These fields are in beta and are subject to change. + description: 'The data_stream fields take part in defining the new data stream naming + scheme. + + In the new data stream naming scheme the value of the data stream fields combine + to the name of the actual data stream in the following manner: `{data_stream.type}-{data_stream.dataset}-{data_stream.namespace}`. + This means the fields can only contain characters that are valid as part of names + of data streams. More details about this can be found in this https://www.elastic.co/blog/an-introduction-to-the-elastic-data-stream-naming-scheme[blog + post]. + + An Elasticsearch data stream consists of one or more backing indices, and a data + stream name forms part of the backing indices names. Due to this convention, data + streams must also follow index naming restrictions. For example, data stream names + cannot include `\`, `/`, `*`, `?`, `"`, `<`, `>`, `|`, ` ` (space character), + `,`, or `#`. Please see the Elasticsearch reference for additional https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#indices-create-api-path-params[restrictions].' + fields: + data_stream.dataset: + dashed_name: data-stream-dataset + description: "The field can contain anything that makes sense to signify the\ + \ source of the data.\nExamples include `nginx.access`, `prometheus`, `endpoint`\ + \ etc. For data streams that otherwise fit, but that do not have dataset set\ + \ we use the value \"generic\" for the dataset value. `event.dataset` should\ + \ have the same value as `data_stream.dataset`.\nBeyond the Elasticsearch\ + \ data stream naming criteria noted above, the `dataset` value has additional\ + \ restrictions:\n * Must not contain `-`\n * No longer than 100 characters" + example: nginx.access + flat_name: data_stream.dataset + level: extended + name: dataset + normalize: [] + short: The field can contain anything that makes sense to signify the source + of the data. + type: constant_keyword + data_stream.namespace: + dashed_name: data-stream-namespace + description: "A user defined namespace. Namespaces are useful to allow grouping\ + \ of data.\nMany users already organize their indices this way, and the data\ + \ stream naming scheme now provides this best practice as a default. Many\ + \ users will populate this field with `default`. If no value is used, it falls\ + \ back to `default`.\nBeyond the Elasticsearch index naming criteria noted\ + \ above, `namespace` value has the additional restrictions:\n * Must not\ + \ contain `-`\n * No longer than 100 characters" + example: production + flat_name: data_stream.namespace + level: extended + name: namespace + normalize: [] + short: A user defined namespace. Namespaces are useful to allow grouping of + data. + type: constant_keyword + data_stream.type: + dashed_name: data-stream-type + description: 'An overarching type for the data stream. + + Currently allowed values are "logs" and "metrics". We expect to also add "traces" + and "synthetics" in the near future.' + example: logs + flat_name: data_stream.type + level: extended + name: type + normalize: [] + short: An overarching type for the data stream. + type: constant_keyword + group: 2 + name: data_stream + prefix: data_stream. + short: The data_stream fields take part in defining the new data stream naming scheme. + title: Data Stream + type: group destination: description: 'Destination fields capture details about the receiver of a network exchange/packet. These fields are populated from a network event, packet, or other diff --git a/generated/elasticsearch/6/template.json b/generated/elasticsearch/6/template.json index 0ad8100c0b..2a964ae302 100644 --- a/generated/elasticsearch/6/template.json +++ b/generated/elasticsearch/6/template.json @@ -332,6 +332,22 @@ } } }, + "data_stream": { + "properties": { + "dataset": { + "ignore_above": 1024, + "type": "keyword" + }, + "namespace": { + "ignore_above": 1024, + "type": "keyword" + }, + "type": { + "ignore_above": 1024, + "type": "keyword" + } + } + }, "destination": { "properties": { "address": { diff --git a/generated/elasticsearch/7/template.json b/generated/elasticsearch/7/template.json index 120dd31501..fea66d43bb 100644 --- a/generated/elasticsearch/7/template.json +++ b/generated/elasticsearch/7/template.json @@ -331,6 +331,19 @@ } } }, + "data_stream": { + "properties": { + "dataset": { + "type": "constant_keyword" + }, + "namespace": { + "type": "constant_keyword" + }, + "type": { + "type": "constant_keyword" + } + } + }, "destination": { "properties": { "address": { diff --git a/generated/elasticsearch/component/data_stream.json b/generated/elasticsearch/component/data_stream.json new file mode 100644 index 0000000000..f1cf96a92b --- /dev/null +++ b/generated/elasticsearch/component/data_stream.json @@ -0,0 +1,25 @@ +{ + "_meta": { + "documentation": "https://www.elastic.co/guide/en/ecs/current/ecs-data_stream.html", + "ecs_version": "2.0.0-dev" + }, + "template": { + "mappings": { + "properties": { + "data_stream": { + "properties": { + "dataset": { + "type": "constant_keyword" + }, + "namespace": { + "type": "constant_keyword" + }, + "type": { + "type": "constant_keyword" + } + } + } + } + } + } +} \ No newline at end of file diff --git a/generated/elasticsearch/template.json b/generated/elasticsearch/template.json index 24224701c2..933091ab10 100644 --- a/generated/elasticsearch/template.json +++ b/generated/elasticsearch/template.json @@ -9,6 +9,7 @@ "ecs_2.0.0-dev_client", "ecs_2.0.0-dev_cloud", "ecs_2.0.0-dev_container", + "ecs_2.0.0-dev_data_stream", "ecs_2.0.0-dev_destination", "ecs_2.0.0-dev_dll", "ecs_2.0.0-dev_dns", diff --git a/experimental/schemas/data_stream.yml b/schemas/data_stream.yml similarity index 92% rename from experimental/schemas/data_stream.yml rename to schemas/data_stream.yml index d651800fa4..a169b0a61c 100644 --- a/experimental/schemas/data_stream.yml +++ b/schemas/data_stream.yml @@ -2,16 +2,18 @@ - name: data_stream title: Data Stream short: The data_stream fields take part in defining the new data stream naming scheme. + beta: > + These fields are in beta and are subject to change. description: > The data_stream fields take part in defining the new data stream naming scheme. In the new data stream naming scheme the value of the data stream fields combine to the name of the actual data - stream in the following manner `{data_stream.type}-{data_stream.dataset}-{data_stream.namespace}`. This means the fields + stream in the following manner: `{data_stream.type}-{data_stream.dataset}-{data_stream.namespace}`. This means the fields can only contain characters that are valid as part of names of data streams. More details about this can be found in this https://www.elastic.co/blog/an-introduction-to-the-elastic-data-stream-naming-scheme[blog post]. An Elasticsearch data stream consists of one or more backing indices, and a data stream name forms part of the backing indices names. - Due to this convention, data streams must also follow index naming restrictions. For example, data stream names cannot include \, /, *, ?, ", <, >, |, ` `. + Due to this convention, data streams must also follow index naming restrictions. For example, data stream names cannot include `\`, `/`, `*`, `?`, `"`, `<`, `>`, `|`, ` ` (space character), `,`, or `#`. Please see the Elasticsearch reference for additional https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#indices-create-api-path-params[restrictions]. fields: