[Discover][Filter by field type] Toggle to hide empty fields has disappeared

[oshmyrko]

Kibana version: 8.8.2

Elasticsearch version: 8.8.2

Server OS version: Ubuntu 20.04.6 LTS

Browser version: Firefox 114.0.2, Chrome 114.0.5735.198

Browser OS version: MacOS 12.5.1 (Monterey)

Original install method (e.g. download page, yum, from source, etc.): ECK operator

Describe the bug:

The toggle to hide empty fields on the left panel of the Discover page has disappeared from Filter by field type after version 8.6.1.

Steps to reproduce:

1. Open the Discover page
2. Try to hide fields from the left panel that have no values (value not set)

Expected behavior:

Filter by field type has the toggle to hide empty fields, so that fields that do not have values (value not set) can be hidden.

Screenshots (if relevant):

• Kibana 8.6.1
  

• Kibana 8.8.2
  

Any additional context:
We are using one general mapping in our index templates to index incoming logs, but since some indices do not contain values for some fields, we would like to hide such fields from the left panel.
Here is an example with IP address field which I would like to hide as well:

[oshmyrko]

Seems related to #147124.

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[davismcphee]

@oshmyrko This is indeed related to #147124, and a little background on this issue:

The change to remove the "Hide empty fields" toggle was intentional as part of an effort to unify the field lists between Discover and Lens as detailed in #135678 (with the actual work completed in #144412).

This was done not only to improve consistency between the apps, but also because the "Hide empty fields" toggle never really functioned as the label would imply. Rather than hiding fields that would be empty based on the query and time range, the "Hide empty fields" feature would simply loop through the 500 document sample returned to Discover and hide any fields for which it couldn't find values within the sample documents. This misled users into believing these fields were empty for all documents returned by their query, when in reality the 501st document could contain values for all "empty" fields, but the fields would still be hidden in the field list.

To account for this we adopted the pattern Lens uses of including an "Empty fields" section into our sidebar which should incude all fields that don't contain values based on the current query and time range. Unfortunately this feature too does not work as intended due to limitations in the upstream Elasticsearch API we rely on -- that's where #147124 comes into play.

I'm going to close this as a duplicate of #147124, but we understand how frustrating not having this functionality can be for users, and we're currently working with the Elasticsearch team directly to figure out how to provide proper support for this feature. Many apologies for the inconvenience here.