Merge branch 'master' into threat-enrichment-stage-1

* master:
  Stage 2 changes for RFC 0018 - extending the `threat.*` field set (elastic#1438)
  Remove deprecated `host.user.*` fields (elastic#1439)
  Explicitly include user identifiers in `related.user` field description (elastic#1420)
  Set the merge date on RFC 0018 stage 2 (elastic#1429)
  [RFC] Extend Threat Fieldset - Stage 2 Proposal (elastic#1395)
  [Tooling] Add --exclude flag to Generator to support field removal testing (elastic#1411)
  Add `host.user.*` deprecation notice in field reuse description (elastic#1422)
  Stage 2 changes for RFC 0015 - `elf` header (elastic#1410)
  Stage 3 changes for RFC 0012 - `orchestrator` field set (elastic#1417)
  Support `match_only_text` in Go code generator (elastic#1418)
  Stage 3 Orchestrator RFC (elastic#1343)
  moving into folder (elastic#1416)
  removing use-cases (elastic#1405)
  removing --oss (elastic#1404)
  Set the merge date on RFC 0015 stage 2 (elastic#1409)
  Consolidate `Breaking changes` sections in `CHANGELOG.next` (elastic#1408)
  RFC-Stage-0: Proposal to add a "ticket" schema / field definition to ECS (elastic#1383)
  [RFC] `match_only_text` type migration - Stage 0 (elastic#1396)
  Client port is wrongly documented (elastic#1402) (elastic#1406)

CHANGELOG.next.md

@@ -8,28 +8,40 @@ Thanks, you're awesome :-) -->
 
 ## Unreleased
 
-### Schema Changes
+### Breaking Changes
 
-#### Breaking changes
+* Removing deprecated --oss from generator #1404
+* Removing use-cases directory #1405
+* Remove `host.user.*` field reuse. #1439
+
+### Schema Changes
 
 #### Bugfixes
 
 #### Added
 
+* `elf.*` field set added as beta. #1410
+* Remove `beta` from `orchestrator` field set. #1417
+* Extend `threat.*` field set beta. #1438
+
 #### Improvements
 
 * Fix ecs GitHub repo link source branch #1393
+* Add --exclude flag to Generator to support field removal testing #1411
+* Explicitly include user identifiers in `relater.user` description. #1420
 
 #### Deprecated
 
-### Tooling and Artifact Changes
+* Note deprecation of the `host.user.*` field reuse. #1422
 
-#### Breaking changes
+### Tooling and Artifact Changes
 
 #### Bugfixes
 
 #### Added
 
+* Support `match_only_text` data type in Go code generator. #1418
+
 #### Improvements
 
 #### Deprecated

CONTRIBUTING.md

@@ -61,6 +61,7 @@ You need these tools to contribute to the ECS repo:
 * Request feedback about your changes.
   - Create a [Pull Request](https://help.github.com/articles/creating-a-pull-request/) against the ECS repo.
     - (Look for the `Compare & pull request` button on your branch in github.com.)
+  - Add an entry to [CHANGELOG.next.md](CHANGELOG.next.md).
   - Wait for reviews on your PR.
   - Incorporate review comments and push updates if needed.
 * Thank you for your contribution!
@@ -96,15 +97,26 @@ Please follow these guidelines when submitting Issues:
 * Click `New issue`. Provide as many details as possible to help reviewers and other contributors understand your proposal.
 * Add your text, and click `Submit new issue`.
 
-### Backports
+### Branching
 
-ECS maintains multiple release branches in the repo. The `master` branch is where all new contributions should be submitted, and features and bug fixes will be backported into other branches when appropriate. Any backporting needs will be handled by the ECS team.
+ECS follows this branching strategy:
+
+* The `master` is the next major version. It is where all new contributions are first merged. This includes new features and bug fixes, and it may also include breaking changes.
+* The `<major>.x` is the next minor version and gets backports of most non-breaking features and fixes.
+* The `<major>.<minor>` is the next release of a minor version, including patch releases.
+
+### Changelog
+
+ECS maintains two changelog files:
 
-#### Branching
+* [CHANGELOG.md](CHANGELOG.md) contains a list of notable changes for each released version of ECS.
+* [CHANGELOG.next.md](CHANGELOG.next.md) contains a list of unreleased ECS changes.
 
-* The `master` branch is where all new contributions are merged. This includes new features and bug fixes, and it may also include breaking changes.
-* The `1.x` branch gets backports of most non-breaking features and fixes. This branch represents the next `major.minor` release.
-* The `major.minor` branches (e.g. `1.6`, `1.5`, `1.4`, etc.) contain the latest released version of those releases.
+Breaking changes intended for the next major version should be included underneath the `Breaking changes` sections in `CHANGELOG.next.md`.
+
+### Backports
+
+ECS maintains multiple release branches in the repo. The `master` branch is where all new contributions should be submitted, and features and bug fixes will be backported into other branches when appropriate. Any backporting needs will be handled by the ECS team.
 
 #### Tooling
 

Makefile

@@ -35,8 +35,6 @@ check-license-headers:
 .PHONY: clean
 clean:
 	rm -rf build generated/elasticsearch/component experimental/generated/elasticsearch/component
-	# Clean all markdown files for use-cases
-	find ./use-cases -type f -name '*.md' -not -name 'README.md' -print0 | xargs -0 rm --
 
 # Alias to generate source code for all languages.
 .PHONY: codegen
@@ -64,7 +62,7 @@ fmt: ve
 
 # Alias to generate everything.
 .PHONY: generate
-generate: generator legacy_use_cases codegen
+generate: generator codegen
 	$(PYTHON) --version
 
 # Run the new generator
@@ -82,11 +80,6 @@ gocodegen:
 	        -schema=../schemas \
 	        -out=../code/go/ecs
 
-# Generate the Use Cases
-.PHONY: legacy_use_cases
-legacy_use_cases: ve
-	$(PYTHON) scripts/use-cases.py --stdout=true >> /dev/null
-
 # Check Makefile format.
 .PHONY: makelint
 makelint: SHELL:=/bin/bash

USAGE.md

@@ -27,10 +27,10 @@ relevant artifacts for their unique set of data sources.
   * [Generator Options](#generator-options)
     + [Out](#out)
     + [Include](#include)
+    + [Exclude](#exclude)
     + [Subset](#subset)
     + [Ref](#ref)
     + [Mapping & Template Settings](#mapping--template-settings)
-    + [OSS (Deprecated)](#oss-deprecated)
     + [Strict Mode](#strict-mode)
     + [Intermediate-Only](#intermediate-only)
 
@@ -232,6 +232,41 @@ Include can be used together with the `--ref` flag to merge custom fields into a
 
 > NOTE: The `--include` mechanism will not validate custom YAML files prior to merging. This allows for modifying existing ECS fields in a custom schema without having to redefine all the mandatory field attributes.
 
+#### Exclude
+
+Use the `--exclude` flag to generate ephemeral ECS artifacts based on the current ECS schema field definitions minus fields considered for removal, e.g. to assess impact of removing these. Warning! This is not the recommended route to remove a field permanently as it is not intentended to be invoked during the build process. Definitive field removal should be implemented using a custom [Subset](#subset) or via the [RFC process](https://github.com/elastic/ecs/tree/master/rfcs/README.md). Example:
+
+```
+$ python scripts/generator.py --exclude=../my-project/my-exclude-file.yml
+$ python scripts/generator.py --exclude="../my-project/schemas/a*.yml"
+```
+
+The `--exclude` flag expects a path to one or more YAML files using the same [file format](https://github.com/elastic/ecs/tree/master/schemas#fields-supported-in-schemasyml) as the ECS schema files. You can also use a subset, provided that relevant `name` and `fields` fields are preserved.
+
+```
+---
+- name: log
+  fields:
+    - name: original
+```
+
+The root Field Set `name` must always be present and specified with no dots `.`. Subfields may be specified using dot notation, for example:
+
+```
+---
+- name: log
+  fields:
+    - name: syslog.severity.name
+```
+
+Generate artifacts using `--exclude` to load our custom definitions in addition to `--out` to place them in the desired output directory:
+
+```
+$ python scripts/generator.py --exclude ../myproject/exclude-set.yml/ --out ../myproject/out/
+Loading schemas from local files
+Running generator. ECS version 1.11.0
+```
+
 #### Subset
 
 If your indices will never populate particular ECS fields, there's no need to include those field definitions in your index mappings. The `--subset` argument allows for passing a subset definition YAML file which indicates which field sets or specific fields to include in the generated artifacts.
@@ -352,30 +387,6 @@ The `--template-settings` argument defines [index level settings](https://www.el
 
 For `template.json`, the `mappings` object is left empty: `{}`. Likewise the `properties` object remains empty in the `mapping.json` example. This will be filled in automatically by the script.
 
-#### OSS (Deprecated)
-
-**Important**: This feature has been deprecated. An Apache 2.0 distribution of Elasticsearch will not be
-produced starting with the 7.11 release. Learn more about these changes [here](https://www.elastic.co/pricing/faq/licensing).
-
-Users that want to use an Apache 2.0 distribution of Elasticsearch do not have access to the basic data types.
-However some of these types have an OSS replacement that can be used instead, without too much loss of functionality.
-
-This flag performs a best effort fallback, replacing basic data types with their OSS replacement.
-
-Indices using purely OSS types will benefit from the normalization of ECS, but may be missing on some of the added functionality of these basic types.
-
-Current fallbacks applied by this flag are:
-
-- `constant_keyword` => `keyword`
-- `wildcard` => `keyword`
-- `version` => `keyword`
-
-Usage:
-
-```
-$ python scripts/generator.py --oss
-```
-
 #### Strict Mode
 
 The `--strict` argument enables "strict mode". Strict mode performs a stricter validation step against the schema's contents.