diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml new file mode 100644 index 0000000000..f5aad7d831 --- /dev/null +++ b/.github/workflows/test.yml @@ -0,0 +1,19 @@ +name: Tests + +on: [push, pull_request] + +jobs: + tests: + runs-on: ubuntu-20.04 + name: Unit Tests + steps: + - uses: actions/checkout@v2 + - uses: actions/setup-go@v2 + with: + go-version: '1.15.x' + - uses: actions/setup-python@v2 + with: + python-version: '3.x' + - run: git fetch --prune --unshallow --tags + - run: make setup + - run: make check diff --git a/.gitignore b/.gitignore index 20c4de146e..a3cabe1d6a 100644 --- a/.gitignore +++ b/.gitignore @@ -7,3 +7,7 @@ build .idea *.iml .vscode/* + +# experimental exclusions +experimental/generated/elasticsearch/6 +experimental/generated/docs diff --git a/.travis.yml b/.travis.yml deleted file mode 100644 index a56a30a073..0000000000 --- a/.travis.yml +++ /dev/null @@ -1,29 +0,0 @@ -sudo: false - -language: go - -os: -- linux - -dist: bionic - -go: -- 1.13.x - -install: -- git fetch --tags --all -- make setup - -addons: - apt: - update: true - packages: - - libxml2-utils - - python3-venv - - xsltproc - -jobs: - include: - - stage: check - script: - - make check diff --git a/CHANGELOG.md b/CHANGELOG.md index 5510520c9f..83526636a4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,240 @@ # CHANGELOG All notable changes to this project will be documented in this file based on the [Keep a Changelog](http://keepachangelog.com/) Standard. This project adheres to [Semantic Versioning](http://semver.org/). +## [1.9.0](https://github.com/elastic/ecs/compare/v1.8.0...v1.9.0) + +### Schema Changes + +#### Added + +* Added `hash.ssdeep`. #1169 +* Added `cloud.service.name`. #1204 +* Added `http.request.id`. #1208 +* `data_stream.*` fieldset introduced in experimental schema and artifacts. #1215 +* Added `geo.timezone`, `geo.postal_code`, and `geo.continent_code`. #1229 +* Added `beta` host metrics fields. #1248 +* Added `code_signature.team_id`, `code_signature.signing_id`. #1249 +* Extended `pe` fields added to experimental schema. #1256 +* Add `elf` fieldset to experimental schema. #1261 +* Add `threat.indicator` fields to experimental schema. #1268 + +#### Improvements + +* Include formatting guidance and examples for MAC address fields. #456 +* New section in ECS detailing event categorization fields usage. #1242 +* `user.changes.*`, `user.effective.*`, and `user.target.*` field reuses are GA. #1271 + +### Tooling and Artifact Changes + +#### Improvements + +* Update Python dependencies #1310, #1318 +* Adjustments to use terminology that doesn't have negative connotation. #1315 + + +## [1.8.0](https://github.com/elastic/ecs/compare/v1.7.0...v1.8.0) + +### Schema Changes + +#### Bugfixes + +* Clean up `event.reference` description. #1181 +* Go code generator fails if `scaled_float` type is used. #1250 + +#### Added + +* Added `event.category` "registry". #1040 +* Added `event.category` "session". #1049 +* Added usage documentation for `user` fields. #1066 +* Added `user` fields at `user.effective.*`, `user.target.*` and `user.changes.*`. #1066 +* Added `os.type`. #1111 + +#### Improvements + +* Event categorization fields GA. #1067 +* Note `[` and `]` bracket characters may enclose a literal IPv6 address when populating `url.domain`. #1131 +* Reinforce the exclusion of the leading dot from `url.extension`. #1151 + +#### Deprecated + +* Deprecated `host.user.*` fields for removal at the next major. #1066 + +### Tooling and Artifact Changes + +#### Bugfixes + +* `tracing` fields should be at root of Beats `fields.ecs.yml` artifacts. #1164 + +#### Added + +* Added the `path` key when type is `alias`, to support the [alias field type](https://www.elastic.co/guide/en/elasticsearch/reference/current/alias.html). #877 +* Added support for `scaled_float`'s mandatory parameter `scaling_factor`. #1042 +* Added ability for --oss flag to fall back `constant_keyword` to `keyword`. #1046 +* Added support in the generated Go source go for `wildcard`, `version`, and `constant_keyword` data types. #1050 +* Added support for marking fields, field sets, or field reuse as beta in the documentation. #1051 +* Added support for `constant_keyword`'s optional parameter `value`. #1112 +* Added component templates for ECS field sets. #1156, #1186, #1191 +* Added functionality for merging custom and core multi-fields. #982 + +#### Improvements + +* Make all fields linkable directly. #1148 +* Added a notice highlighting that the `tracing` fields are not nested under the + namespace `tracing.` #1162 +* ES 6.x template data types will fallback to supported types. #1171, #1176, #1186 +* Add a documentation page discussing the experimental artifacts. #1189 + +## [1.7.0](https://github.com/elastic/ecs/compare/v1.6.0...v1.7.0) + +### Schema Changes + +#### Bugfixes + +* The `protocol` allowed value under `event.type` should not have the `expected_event_types` defined. #964 +* Clarify the definition of `file.extension` (no dots). #1016 + +#### Added + +* Added Mime Type fields to HTTP request and response. #944 +* Added network directions ingress and egress. #945 +* Added `threat.technique.subtechnique` to capture MITRE ATT&CK® subtechniques. #951 +* Added `configuration` as an allowed `event.category`. #963 +* Added a new directory with experimental artifacts, which includes all changes + from RFCs that have reached stage 2. #993, #1053, #1115, #1117, #1118 + +#### Improvements + +* Expanded field set definitions for `source.*` and `destination.*`. #967 +* Provided better guidance for mapping network events. #969 +* Added the field `.subdomain` under `client`, `destination`, `server`, `source` + and `url`, to match its presence at `dns.question.subdomain`. #981 +* Clarified ambiguity in guidance on how to use x509 fields for connections with + only one certificate. #1114 + +### Tooling and Artifact Changes + +#### Breaking changes + +* Changed the index pattern of the sample Elasticsearch template from `ecs-*` to + `try-ecs-*` to avoid conflicting with Logstash' `ecs-logstash-*`. #1048 + +#### Bugfixes + +* Addressed issue where foreign reuses weren't using the user-supplied `as` value for their destination. #960 +* Experimental artifacts failed to install due to `event.original` index setting. #1053 + +#### Added + +* Introduced `--strict` flag to perform stricter schema validation when running the generator script. #937 +* Added check under `--strict` that ensures composite types in example fields are quoted. #966 +* Added `ignore_above` and `normalizer` support for keyword multi-fields. #971 +* Added ability to supply free-form usage documentation per fieldset. #988 +* Added `--oss` flag for users who want to generate ECS templates for use on OSS clusters. #991 + +#### Improvements + +* Field details Jinja2 template components have been consolidated into one template #897 +* Add `[discrete]` marker before each section header in field details. #989 +* `--ref` now loads `experimental/schemas` based on git ref in addition to `schemas`. #1063 + + +## [1.6.0](https://github.com/elastic/ecs/compare/v1.5.0...v1.6.0) + +### Schema Changes + +#### Bugfixes + +* Field `registry.data.strings` should have been marked as an array field. #790 + +#### Added + +* Added `x509.*` field set. #762 +* Add architecture and imphash for PE field set. #763 +* Added `agent.build.*` for extended agent version information. #764 +* Added `log.file.path` to capture the log file an event came from. #802 +* Added more account and project cloud metadata. #816 +* Added missing field reuse of `pe` at `process.parent.pe` #868 +* Added `span.id` to the tracing fieldset, for additional log correlation #882 +* Added `event.reason` for the reason why an event's outcome or action was taken. #907 +* Added `user.roles` to capture a list of role names that apply to the user. #917 + +#### Improvements + +* Removed misleading pluralization in the description of `user.id`, it should + contain one ID, not many. #801 +* Clarified misleading wording about multiple IPs in src/dst or cli/srv. #804 +* Improved verbiage about the MITRE ATT&CK® framework. #866 +* Removed the default `object_type=keyword` that was being applied to `object` fields. + This attribute is Beats-specific. It's still supported, but needs to be set explicitly + on a case by case basis now. This default being removed affects `dns.answers`, + `log.syslog`, `network.inner`, `observer.egress`, and `observer.ingress`. #871 +* Improved attribute `dashed_name` in `generated/ecs/*.yml` to also + replace `@` with `-`. #871 +* Updated several URLs in the documentation with "example.com" domain. #910 + +#### Deprecated + +* Deprecate guidance to lowercase `http.request.method` #840 + + +### Tooling and Artifact Changes + +#### Breaking changes + +* Removed field definitions at the root of documents for fieldsets that + had `reusable.top_level:false`. This PR affects `ecs_flat.yml`, the csv file + and the sample Elasticsearch templates. #495, #813 +* Removed the `order` attribute from the `ecs_nested.yml` and `ecs_flat.yml` files. #811 +* In `ecs_nested.yml`, the array of strings that used to be in `reusable.expected` + has been replaced by an array of objects with 3 keys: 'as', 'at' and 'full'. #864 +* The subset format now requires `name` and `fields` keys at the top level. #873 + +#### Bugfixes + +* Subsets are created after duplicating reusable fields now so subsets can + be applied to each reused instance independently. #753 +* Quoted the example for `labels` to avoid YAML interpreting it, and having + slightly different results in different situations. #782 +* Fix incorrect listing of where field sets are nested in asciidoc, + when they are nested deep. #784 +* Allow beats output to be generated when using `--include` or `--subset` flags. #814 +* Field parameter `index` is now correctly populated in the Beats field definition file. #824 + +#### Improvements + +* Add support for reusing official fieldsets in custom schemas. #751 +* Add full path names to reused fieldsets in `nestings` array in `ecs_nested.yml`. #803 +* Allow shorthand notation for including all subfields in subsets. #805 +* Add support for Elasticsearch `enabled` field parameter. #824 +* Add `ref` option to generator allowing schemas to be built for a specific ECS version. #851 +* Add `template-settings` and `mapping-settings` options to allow override of defaults in generated ES templates. #856 +* When overriding ECS field sets via the `--include` flag, it's no longer necessary + to duplicate the field set's mandatory attributes. The customizations are merged + before validation. #864 +* Add ability to nest field sets as another name. #864 +* Add ability to nest field sets within themselves (e.g. `process` => `process.parent`). #864 +* New attribute `reused_here` is added in `ecs_nested.yml`. It obsoletes the + previous attribute `nestings`, and is able to fully capture details of other + field sets reused under this one. #864 +* When chained reuses are needed (e.g. `group` => `user`, then `user` => many places), + it's now necessary to force the order with new attribute `reusable.order`. This + attribute is otherwise optional. It's currently only needed for `group`. #864 +* There's a new representation of ECS at `generated/ecs/ecs.yml`, which is a deeply nested + representation of the fields. This file is not in git, as it's only meant for + developers working on the ECS tools. #864 +* Jinja2 templates now define the doc structure for the AsciiDoc generator. #865 +* Intermediate `ecs_flat.yml` and `ecs_nested.yml` files are now generated for each individual subset, + in addition to the intermediate files generated for the combined subset. #873 + +#### Deprecated + +* In `ecs_nested.yml`, we're deprecating the attribute `nestings`. It will be + removed in a future release. The deprecated `nestings` attribute was an array of + flat field names describing where fields are nested within the field set. + This is replaced with the attribute `reused_here`, which is an array of objects. + The new format still lists where the fields are nested via the same flat field name, + but also specifies additional information about each field reuse. #864 + ## [1.5.0](https://github.com/elastic/ecs/compare/v1.4.0...v1.5.0) diff --git a/CHANGELOG.next.md b/CHANGELOG.next.md index ae6e775639..4c99b1e12d 100644 --- a/CHANGELOG.next.md +++ b/CHANGELOG.next.md @@ -10,103 +10,26 @@ Thanks, you're awesome :-) --> ### Schema Changes -* Added `log.file.path` to capture the log file an event came from. #802 - #### Breaking changes #### Bugfixes -* Field `registry.data.strings` should have been marked as an array field. #790 - #### Added -* Add architecture and imphash for PE field set. (#763) -* Added `agent.build.*` for extended agent version information. (#764) -* Added `x509.*` field set. (#762) -* Added more account and project cloud metadata. (#816) -* Added missing field reuse of `pe` at `process.parent.pe` #868 -* Added `span.id` to the tracing fieldset, for additional log correlation (#882) -* Added `event.reason` for the reason why an event's outcome or action was taken. #907 - #### Improvements -* Removed misleading pluralization in the description of `user.id`, it should - contain one ID, not many. #801 -* Clarified misleading wording about multiple IPs in src/dst or cli/srv. #804 -* Improved verbiage about the MITRE ATT&CK® framework. #866 -* Removed the default `object_type=keyword` that was being applied to `object` fields. - This attribute is Beats-specific. It's still supported, but needs to be set explicitly - on a case by case basis now. This default being removed affects `dns.answers`, - `log.syslog`, `network.inner`, `observer.egress`, and `observer.ingress`. #871 -* Improved attribute `dashed_name` in `generated/ecs/*.yml` to also - replace `@` with `-`. #871 -* Updated several URLs in the documentation with "example.com" domain. #910 - -#### Deprecated - -* Deprecate guidance to lowercase `http.request.method` #840 -* In `ecs_nested.yml`, we're deprecating the attribute `nestings`. It will be - removed in a future release. The deprecated `nestings` attribute was an array of - flat field names describing where fields are nested within the field set. - This is replaced with the attribute `reused_here`, which is an array of objects. - The new format still lists where the fields are nested via the same flat field name, - but also specifies additional information about each field reuse. - - ### Tooling and Artifact Changes #### Breaking changes -* Removed field definitions at the root of documents for fieldsets that - had `reusable.top_level:false`. This PR affects `ecs_flat.yml`, the csv file - and the sample Elasticsearch templates. #495, #813 -* Removed the `order` attribute from the `ecs_nested.yml` and `ecs_flat.yml` files. #811 -* In `ecs_nested.yml`, the array of strings that used to be in `reusable.expected` - has been replaced by an array of objects with 3 keys: 'as', 'at' and 'full'. #864 -* The subset format now requires `name` and `fields` keys at the top level. #873 - #### Bugfixes -* Subsets are created after duplicating reusable fields now so subsets can - be applied to each reused instance independently. #753 -* Quoted the example for `labels` to avoid YAML interpreting it, and having - slightly different results in different situations. #782 -* Fix incorrect listing of where field sets are nested in asciidoc, - when they are nested deep. #784 -* Allow beats output to be generated when using `--include` or `--subset` flags. #814 -* Field parameter `index` is now correctly populated in the Beats field definition file. #824 - #### Added #### Improvements -* Add support for reusing official fieldsets in custom schemas. #751 -* Add full path names to reused fieldsets in `nestings` array in `ecs_nested.yml`. #803 -* Allow shorthand notation for including all subfields in subsets. #805 -* Add support for Elasticsearch `enabled` field parameter. #824 -* Add `ref` option to generator allowing schemas to be built for a specific ECS version. #851 -* Add `template-settings` and `mapping-settings` options to allow override of defaults in generated ES templates. #856 -* When overriding ECS field sets via the `--include` flag, it's no longer necessary - to duplicate the field set's mandatory attributes. The customizations are merged - before validation. #864 -* Add ability to nest field sets as another name. #864 -* Add ability to nest field sets within themselves (e.g. `process` => `process.parent`). #864 -* New attribute `reused_here` is added in `ecs_nested.yml`. It obsoletes the - previous attribute `nestings`, and is able to fully capture details of other - field sets reused under this one. #864 -* When chained reuses are needed (e.g. `group` => `user`, then `user` => many places), - it's now necessary to force the order with new attribute `reusable.order`. This - attribute is otherwise optional. It's currently only needed for `group`. #864 -* There's a new representation of ECS at `generated/ecs/ecs.yml`, which is a deeply nested - representation of the fields. This file is not in git, as it's only meant for - developers working on the ECS tools. #864 -* Jinja2 templates now define the doc structure for the AsciiDoc generator. #865 -* Intermediate `ecs_flat.yml` and `ecs_nested.yml` files are now generated for each individual subset, - in addition to the intermediate files generated for the combined subset. #873 - #### Deprecated -