[Discover] Add support for unmapped fields using the fields API

[majagrubic]

Summary

Fixes #88192.
This PR adds support for unmapped fields using the new fields API. The logic we discussed to implement this is as follows:

1. There is a switch in the UI with the deprecation notice; the switch is on by default for existing saved searches; off by default for new saved searches
2. To determine if it's an existing (pre 7.12) saved search, we need a data migration

Current flow:
(notice that for same index pattern unmapped fields don't show if it's a new saved search, but they show up when you load a pre-existing one)

How to test this?

1. Before pulling the branch, create an index in ES with unmapped fields. Eg:

PUT /my-test-index
{
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "properties": {
      "field1": { "type": "text" }
    }
  }
}

PUT /my-test-index/_doc/1
{
  "field1": "Some blob of text",
  "field2": "Some other text"
}

PUT /my-test-index/_doc/2
{
  "field1": "Some blob of text",
  "field2": "Some other text"
}

PUT /my-test-index/_doc/3
{
  "field1": "Some blob of text",
  "field3": 12
}

PUT /my-test-index-2
{
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
     "dynamic": false
  }
}

PUT /my-test-index-2/_doc/1
{
  "field1": "Some blob of text",
  "field2": "Some other text"
}

PUT /my-test-index-2/_doc/2
{
  "field1": "Some blob of text",
  "field2": "Some other text"
}

PUT /my-test-index-2/_doc/3
{
  "field4": "Some blob of text",
  "field5": 12
}


PUT /my-test-index-2/_doc/4
{
  "field4": "Some blob of text",
  "field5": 12
}

2. Create an index pattern from the newly created index
3. Go to Discover, select the index pattern created under 2
4. Save the existing saved search
5. Pull the branch, restart Kibana
6. Observe how unmapped fields are behaving

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
• Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
• If a plugin configuration key changed, check if it needs to be whitelisted in the cloud and added to the docker list
• This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
• This was checked for cross-browser compatibility

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[kertal]

@elasticmachine merge upstream

[kertal]

since you're using

test/functional/fixtures/es_archiver/unmapped_fields

for your functionals, you can delete

test/functional/fixtures/es_archiver/data

[kertal]

typo: funtction -> function

[kertal]

It's looking pretty good 👍 (so far just minor remarks) , while testing I've found one case that I think needs adaptation or discussion. when adding a pre712 saved search, unmapped fields are displayed in Discover, but not when using the saved search on a Dashboard:

saved-search-pre712.mp4

[majagrubic]

@elasticmachine merge upstream

[majagrubic]

@kertal I fixed the embeddable so that it works with the new API now

[kertal]

@elasticmachine merge upstream

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 64282c8

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
discover	413.4KB	417.1KB	+3.7KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
discover	69.1KB	69.1KB	+47.0B

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/development-plugin-saved-objects.html#_mappings

id	before	after	diff
search	9	10	+1

History

• 💚 Build #103326 succeeded 0b8102b
• 💚 Build #103160 succeeded b16e3fd
• 💔 Build #103051 failed d7b2fdf
• 💔 Build #102968 failed abecb21
• 💚 Build #102497 succeeded fda7725

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[kertal]

Code LGTM 👍, embeddable now works. Tested locally using Chrome/Safari, works as expected. One note: using the Angular legacy grid, rendering of records in the table are not updated when there's a refetch when switching the Show unmapped. This is due to the internal Angular caching works. There's no easy workaround here than: switching to EuiDataGrid. There it works, so this should not block this PR.

[kertal]

Nit: another any that could bite the dust

[majagrubic]

Will address this in the next PR