Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable fields added from unknown saved object types #70951

Merged
merged 7 commits into from
Jul 9, 2020

Conversation

rudolf
Copy link
Contributor

@rudolf rudolf commented Jul 7, 2020

Summary

Fixes #67086 so that we don't need workarounds to remove old type mappings like https://github.com/aaronjcaldwell/kibana/blob/8a04349b3c5b5c6cb852cf4cef83b4bb9884e8d3/x-pack/plugins/maps/server/saved_objects/maps_telemetry.ts#L8-L23

When upgrading from 7.7.1 -> 7.8.0 -> master this PR removes 123 fields, but 122 of these are because security_solution is disabled by default in master but was enabled in 7.8 (as the siem plugin).

           "server":{
              "type":"object",
              "dynamic":"false"
           },
           "siem-detection-engine-rule-actions":{
              "type":"object",
              "dynamic":"false"
           },
           "siem-detection-engine-rule-status":{
              "type":"object",
              "dynamic":"false"
           },
           "siem-ui-timeline":{
              "type":"object",
              "dynamic":"false"
           },
           "siem-ui-timeline-note":{
              "type":"object",
              "dynamic":"false"
           },
           "siem-ui-timeline-pinned-event":{
              "type":"object",
              "dynamic":"false"
           },

Limitations

Setting dynamic: false on the top-level index mappings would remove a lot of validation provided by ES so until Saved Objects allow plugins to define their own validation that's not really an option. Since we always retain the documents belonging to unknown saved objects types, we need to keep the minimal mappings to be able to store these. This means we can only disable the fields underneath an unknown type's mappings with dynamic: false.

Although plugins could use a "core field datatype" like text for their mappings from Legacy this is unlikely and most plugins used an object data type.

I don't think this has ever been done, but should we for instance deprecate the namespace field (type: 'keyword') in favour of namespaces then this PR won't clean up that field.

It's also important to note that although we clean up all the fields of a unknown type, the top-level field for that type will still count towards the field count limit.

As an example, the following mapping:

{
  "dynamic": strict,
  "properties": {
    "a": {
      "properties": {
        "b": {
          "properties": {
            "c": {
              "type": "text"
            }
          }
        }
      }
    }
  }
}

Would add fields a, b and c that count towards the field count limit (even though only c is added to the index). We can disable everything below a but with dynamic: false but we would still be left with one field a.

So this would remove one of the two fields left over by server.uuid from #66963 and potentially leave one maps-telemetry field over from #69995 when users upgrade from older versions.

We will eventually completely solve this problem in #66056 by refusing to upgrade with unknown data so then we don't need any mappings for unknown types hanging around.

Checklist

Delete any items that are not applicable to this PR.

For maintainers

@rudolf rudolf added Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc Feature:Saved Objects v8.0.0 v7.9.0 labels Jul 7, 2020
@rudolf rudolf requested a review from a team as a code owner July 7, 2020 12:48
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-platform (Team:Platform)

@rudolf rudolf added the release_note:skip Skip the PR/issue when compiling release notes label Jul 7, 2020
@rudolf rudolf force-pushed the so-mappings-cleanup branch from 4fd945c to 1d6cfc9 Compare July 7, 2020 14:33
@rudolf rudolf requested a review from pgayvallet July 9, 2020 09:51
@kibanamachine
Copy link
Contributor

💔 Build Failed

Failed CI Steps

Build metrics

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@rudolf
Copy link
Contributor Author

rudolf commented Jul 9, 2020

Type check failure comes from master (and has since been fixed) so ignoring.

@rudolf rudolf merged commit 77a97fc into elastic:master Jul 9, 2020
@rudolf rudolf deleted the so-mappings-cleanup branch July 9, 2020 11:45
rudolf added a commit to rudolf/kibana that referenced this pull request Jul 9, 2020
* Allow disabled: false SO field mappings

* Disable fields for unknown SO types

* Update everyone else's docs ;)

* Address review comments

* Add unit tests for disableUnknownTypeMappingFields()
rudolf added a commit that referenced this pull request Jul 9, 2020
* Allow disabled: false SO field mappings

* Disable fields for unknown SO types

* Update everyone else's docs ;)

* Address review comments

* Add unit tests for disableUnknownTypeMappingFields()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:Saved Objects release_note:skip Skip the PR/issue when compiling release notes Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v7.9.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Saved Object Migrations should clean up unused mappings
4 participants