Skip to content

Commit

Permalink
[RFC] 0017 Deprecation of log.original Stage 3 Proposal (#1465)
Browse files Browse the repository at this point in the history
* Stage 3 Finished

Co-authored-by: Eric Beahan <[email protected]>
  • Loading branch information
djptek and ebeahan authored Jun 23, 2021
1 parent 6cdf0c3 commit 6200e56
Showing 1 changed file with 31 additions and 13 deletions.
44 changes: 31 additions & 13 deletions rfcs/text/0017-remove-log-original.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# 0017: Remove log.original

- Stage: **2 (candidate)** <!-- Update to reflect target stage. See https://elastic.github.io/ecs/stages.html -->
- Date: **2021-04-28** <!-- The ECS team sets this date at merge time. This is the date of the latest stage advancement. -->
- Stage: **3 (finished)** <!-- Update to reflect target stage. See https://elastic.github.io/ecs/stages.html -->
- Date: **2021-06-23** <!-- The ECS team sets this date at merge time. This is the date of the latest stage advancement. -->

This RFC supersedes issue [#841](https://github.com/elastic/ecs/issues/841) which implies breaking changes therefore the RFC Process is indicated.

Expand Down Expand Up @@ -45,10 +45,13 @@ Any or all incoming log or event messages when the original value of the event n
dst=2.1.2.2spt=1232"}
{"event.original": "Sep 19 08:26:10 localhost My log"}
{
"event": {
"original": "{\"Node\":\"compute01-hc-xx-local-abcdefgh-com\",\"CheckID\":\"_nomad-check-2724899cc463354bb6336c67421b73f36fb966e2\",\"Name\":\"service: \\\"hello-world\\\" check\",\"Status\":\"passing\",\"Notes\":\"\",\"Output\":\"HTTP GET http://10.100.8.99:8080/: 200 OK Output: Hello world! Your request address: 10.100.8.34:37230\\n\\nRuntime env:\\nPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\\nHOSTNAME=7c9b09a0258c\\nNOMAD_ALLOC_DIR=/alloc\\nNOMAD_ALLOC_ID=4c62244a-6924-4cb6-60d3-a4c574c58085\\nNOMAD_ALLOC_INDEX=0\\nNOMAD_ALLOC_NAME=hello-world.servers[0]\\nNOMAD_CPU_LIMIT=50\\nNOMAD_DC=xx\\nNOMAD_GROUP_NAME=servers\\nNOMAD_JOB_ID=hello-world\\nNOMAD_JOB_NAME=hello-world\\nNOMAD_MEMORY_LIMIT=32\\nNOMAD_NAMESPACE=default\\nNOMAD_REGION=global\\nNOMAD_SECRETS_DIR=/secrets\\nNOMAD_TASK_DIR=/local\\nNOMAD_TASK_NAME=hello-world\\nHOME=/\\n\",\"ServiceID\":\"_nomad-task-4c62244a-6924-4cb6-60d3-a4c574c58085-group-servers-hello-world-8080\",\"ServiceName\":\"hello-world\",\"ServiceTags\":[\"http\",\"traefik.enable=true\",\"traefik.http.routers.hello-world.entrypoints=https\",\"traefik.http.routers.hello-world.tls.certresolver=le\",\"traefik.http.routers.hello-world.tls.domains[0].main=svc.abcdefgh.com\",\"traefik.http.routers.hello-world.tls.domains[0].sans=*.svc.abcdefgh.com\"],\"Type\":\"http\",\"Definition\":{},\"CreateIndex\":4281349\"ModifyIndex\":5749556}"
}
}
```
<!--
Stage 3: Add more real world example source documents so we have at least 2 total, but ideally 3. Format as described in stage 2.
-->

## Scope of impact

Expand All @@ -58,9 +61,13 @@ The removal of `log.original` will be considered a breaking change since the fie

- The [`Beats default fields inclusion list`](https://github.com/elastic/ecs/blob/master/scripts/generators/beats_default_fields_allowlist.yml) list should be updated by removing `log.original` if/when Deprecation progresses to Removal

- The logs UI `message` column currently displays `log.original` in the absence of a `message` field. This should be updated to use `event.original` as the substitute field. See [builtin_rules](https://github.com/elastic/kibana/blob/master/x-pack/plugins/infra/server/services/log_entries/message/builtin_rules/generic.ts) and [associated test](https://github.com/elastic/kibana/blob/master/x-pack/plugins/infra/server/services/log_entries/message/builtin_rules/generic.test.ts).
- The logs UI `message` column currently displays `log.original` in the absence of a `message` field. It should be updated to use `event.original` as the substitute field. See [builtin_rules](https://github.com/elastic/kibana/blob/master/x-pack/plugins/infra/server/services/log_entries/message/builtin_rules/generic.ts) and [associated test](https://github.com/elastic/kibana/blob/master/x-pack/plugins/infra/server/services/log_entries/message/builtin_rules/generic.test.ts). Kibana team is aware and has a [PR awaiting review](https://github.com/elastic/kibana/pull/102236) which addresses this.

- The breakdown of usage of this field in Kibana source shows:
- 88 usages in tests
- 22 usages outside of tests, e.g. `js`, `ts`, `map` files

- References in the [RAC Rule Registry](https://github.com/elastic/kibana/blob/master/x-pack/plugins/rule_registry/common/assets/field_maps/ecs_field_map.ts) will need to be removed - these have `required: false` so hopefully non-breaking change.
- For example, references in the [RAC Rule Registry](https://github.com/elastic/kibana/blob/master/x-pack/plugins/rule_registry/common/assets/field_maps/ecs_field_map.ts) will need to be removed - these have `required: false` so hopefully non-breaking change.

- Multiple tests in Kibana will need to be updated see e.g. [Function Test APM Mapping](https://github.com/elastic/kibana/blob/master/x-pack/test/functional/es_archives/monitoring/setup/collection/detect_apm/mappings.json)

Expand All @@ -72,9 +79,13 @@ The removal of `log.original` will be considered a breaking change since the fie

As a breaking change, this would require timely communication to the Elastic Community.

<!--
Stage 3: Document resolutions for all existing concerns. Any new concerns should be documented along with their resolution. The goal here is to eliminate risk of churn and instability by ensuring all concerns have been addressed.
-->
With respect to internal stakeholders, the following actons have been taken:

- Reached out the Beats + Integrations teams: Filebeat has already taken action, thanks @P1llus

- Reached out to Logs UI team: `logs.original` has now been superceed by `event.original` as fallback when `message`not available, thanks @weltenwort

- Reached out to RAC: the reference [RAC Rule Registry](https://github.com/elastic/kibana/blob/master/x-pack/plugins/rule_registry/common/assets/field_maps/ecs_field_map.ts) has been acknowledged, thanks @jasonrhodes

## People

Expand All @@ -86,11 +97,16 @@ The following are the people that consulted on the contents of this RFC.
* @jasonrhodes | Logging UI & RAC
* @MikePaquette | RAC


## References

* [#841](https://github.com/elastic/ecs/issues/841)
* [#777](https://github.com/elastic/integrations/issues/777)
### Issues
* [ecs #841](https://github.com/elastic/ecs/issues/841)
* [integrations #777](https://github.com/elastic/integrations/issues/777)
* [integrations #994](https://github.com/elastic/integrations/issues/994)
* [logs UI #101307](https://github.com/elastic/kibana/issues/101307)

### Stack PRs
* [kibana PR #102236](https://github.com/elastic/kibana/pull/102236)

### RFC Pull Requests

Expand All @@ -99,3 +115,5 @@ The following are the people that consulted on the contents of this RFC.
* Stage 0: https://github.com/elastic/ecs/pull/1298
* Stage 1: https://github.com/elastic/ecs/pull/1314
* Stage 2: https://github.com/elastic/ecs/pull/1347
* Stage 3: https://github.com/elastic/ecs/pull/1465
* Implementation: https://github.com/elastic/ecs/pull/1469

0 comments on commit 6200e56

Please sign in to comment.