[Dashboard] Filter only the relevant panels

[teresaalvarezsoler]

Problem

When users use the global filters, query bar or the controls, the panels that are not built using the dataview return no results.

Solution

The panels should ignore the filter pills, queries and controls when their dataviews don't contain the used fields. We will achieve this by setting the “Ignore filter(s)” in Advance Setting to ON by default. All existing and new dashboards will ignore the filtered fields for the visualizations that don't contain them. Users can always revert this change by setting this to OFF in Advance Settings.

Tour

We should make users aware by adding a tour that meets the following criteria

1. When a user opens a dashboard (in edit or view mode) that contains more than 1 dataview , show a tooltip pointing to the filter pills (@cqliu1 if this is very complex, let's decide another place)
2. In the tooltip, show a checkbox "Don't show again"
3. The tour should appear per user, i.e if another user opens the same dashboard, the tour will appear again

[elasticmachine]

Pinging @elastic/kibana-presentation (Team:Presentation)

[thomasneirynck]

Thanks for creating this. This is a great feature, and really will help to make behavior of a Dashboard with mixed content more legible.

[IanLee1521]

To the discussion question, isn't there already a UX for when accuracy is limited? I've seen something like that in Lens I believe with a little marker on the panel. Perhaps the "this panel isn't using the filters" UX could be something similar?

[teresaalvarezsoler]

Thanks for the input Ian. We think it can be a bit messy to have all these markers visible in the panels e.g., when there are several filters affecting different panels.

[IanLee1521]

Sounds good, agree that extra visual clutter is also an issue.

In other news, I'd also suggest this issue is similar (though different) from #29749 as well. Would love to see both of these features land, there absence is definitely felt strongly! :)

[nreese]

This issue also arises for maps with layers from different data views. Here is a forum question asking for this feature https://discuss.elastic.co/t/kibana-maps-adding-filter-hides-layers-which-dont-have-the-filter-field

[ThomThomson]

One consideration with this new approach: Controls currently don't use searchSource for data fetching. So they won't respect this advanced setting once we turn it on.

As part of this issue we should update Controls to remove filters that aren't compatible with their data views before the request is sent off to the Kibana server.

@markov00 do you know if TSVB respects the ignoreFilterIfFieldNotInIndex setting? If it doesn't, do you think it's necessary to also update it to do so?

[nreese]

Another consideration is functional tests. We should create a functional test suite that sets this value to true and ensure lens, maps, controls, etc respects the value. This is an easy thing to break so some test coverage will be needed if its going to enabled be default

[markov00]

@ThomThomson haven't tried by It looks like TSVB should respect the ignoreFilterIfFieldNotInIndex setting

kibana/src/plugins/vis_types/timeseries/server/lib/vis_data/helpers/get_es_query_uisettings.ts

Lines 15 to 17 in 8a5d327

	const ignoreFilterIfFieldNotInIndex = await uiSettings.get(
	UI_SETTINGS.COURIER_IGNORE_FILTER_IF_FIELD_NOT_IN_INDEX
	);

[jasonrhodes]

Question from observability: how does this setting work with dynamic fields that aren't explicitly called out in an index's mappings? @ThomThomson pointed me to this function, but it's not clear to me when that function is run in a Dashboard context and whether it would be aware of dynamically added fields.

The example we're curious about is something like this:

• Index A1 explicitly maps fields "name" and "size", but does NOT specify "dynamic: false"
• Data view created for Index A1 called DV-A1
• Dashboard D1 is created with visualizations V1, V2, and V3, where V1 queries DV-A1
• Documents are later pushed into index A1 with field "location" (not mapped)
• Dashboard D1 has a filter applied - location: "US"

Would visualization V3 properly filter by location?

I created a sample index, data view, visualization, and dashboard to test this. As far as I can tell, the data view I created was aware of all of the fields in the underlying index, even ones that were not in the mapping at all. As I added documents to that underlying index with new fields, each new field appeared in the data view field list and was available as a filter in the visualization, even when I had this advanced setting ("ignore filters") enabled.

Can someone on the viz team confirm this is expected?

CC: @dgieselaar let me know if there are other dynamic mapping scenarios that I'm not thinking of that could act differently than what I'm describing here.

[dgieselaar]

@jasonrhodes What I had in my head is slightly different. It's not about documents being pushed into the index later and the data view being updated. It's about an index not having data for a field at all, but the field is still relevant. Consider service.environment not being populated for a subset of APM indices (because those have not defined an environment). In this case, the index mapping does not have service.environment mapped, but service.environment still means something. If filtering on service.environment:production still yields results for documents that don't have this value set, I think it can be confusing. The point is, I think we are past the point (with dynamic templates) where we can tell from index metadata that a specific field is or is not relevant to the data set.

Also, I'd argue that it's easier to figure out why there is no data (and possibly add some kind of filter override/ignore setting for this specific panel), than it is for users that the data on the screen is not filtered. Do we plan to somehow indicate to the user that a specific filter was ignored for a certain panel? (It hurts my brain a little to think about the implications here and that's never a good sign).

Can we also name this something more meaningful (and less scary) than "ignore filters"?

[jasonrhodes]

If filtering on service.environment:production still yields results for documents that don't have this value set, I think it can be confusing.

So you're saying in this case, you'd expect that filter to produce no results (i.e., how it works today), because the underlying data hasn't explicitly set this field to the correct value? That sounds like a valid concern to me.

I think the creation of these filters is already tied to data views, so it may not be possible to create this type of UI filter if the field isn't present in the underlying data, but maybe that should be rethought, also.

[thomasneirynck]

I feel the current default behavior is quite wrong and confusing. @dgieselaar how common is the scenario you are describing?

I do agree that ideally surfacing more fine-grained info on a per-panel basis would be very useful UX. This is valuable not just for indicating whether a filter is applied, but in the future this should indicate things like whether data is approximate with random-sampler, or whether it is filtered by tier, ...

For reference, Maps does something like this on a per layer basis. Notice the little icons informing user about how the data is being filtered.

e.g. in this case, by time (the little clock), and search-bar (the little filter icon), and bounding box (the little rectangle)

There would be a lot of value to add similar visual feedback on dashboard panels as well.

---

One last thought; this seems to hang up on whether this is a breaking change or not. I believe it is not. It is a change in the UX, not a change to a programmable API, and we can guide the user to revert the setting (e.g. with the tour)

That said, there is continued uncertainty on
(a) the correct default
(b) whether we need better UX to surface the impact of the value of the setting
(c) whether to make this change at all (?), if we move this to a per dashboard or per panel setting later
(d) perceived magnitude of the change

Providing we agree this is not a breaking change, wondering if we can roll out modifications to this behavior in Serverless first. It would give us some quicker real world feedback and help validate our assumptions.

[dgieselaar]

@thomasneirynck I think it is fairly common to filter down on e.g. dynamically mapped labels. Even when you own the data I think it's really hard to figure out whether the filter you have is relevant to the visualization or not. E.g in the overview cluster we have labels.projectId and labels.project_id. Let's say I add a filter for labels.projectId and one of the visualizations is operating over a dataset that has labels.project_id for whatever reason. It would not be obvious to me that the label is not showing data because the filter is ignored. Obviously per-panel UX (what filters are ignored, ignore/don't ignore for this panel, persist this change, etc) would help here, and IMHO that would be absolutely critical before making this switch.

[thomasneirynck]

thx for feedback @dgieselaar

IMHO that would be absolutely critical before making this switch.

the fact is that today this is not surfaced either, and users today will find themselves in a seemingly "broken" situation (and more frequently, from the other direction, ie. "why are my charts breaking").

It somewhat boils down to the difference between unknown-value and absent-data, which is impossible to make without mapped fields.

[stratoula]

I shared my opinion on this matter on a private channel so I am going to share here too for a broader visibility.
While I think is a change that many users might want, I feel that we should do it with a better planning and possibly a different UI implementation. I find the following confusing behaviors with this setting on:

• The KQL query is not being ignored with this setting, only the filters. This is super confusing
• Vega charts do not respect this setting which means that in a dashboard the users will see data to all their charts regardless the dataview selected but not for vega
• ESQL charts also do not respect this setting
• Do we need an indication that this panel ignores the filters? So for example this Lens chart here has the switch Use global filters set to On which means it should be filtered but the advanced setting is set to ignore causing a big confusion. No?

• @lukasolson mentioned something about alerts not working as expected. Have we discussed with him what he means (Lukas knows this feature better than all of us).

• It also affects how Lens multi layer charts work. Check this screenshot here. I have 2 layers, different indices, setting on. You can see that my filter is ignored by the second layer. Do we want this? Do we inform the users? This setting is a global one, it doesnt affect only dashboards but also other applications. We need to check what it means for each one of them

• The setting has a very low changing rate (our users usually dont discover the advanced settings so this might explain the low rate but still is an important metric. Other advanced settings have much larger changing rate so I woulnd't take this metri slighlty)

My personal opinion is:

• Deprecate the advanced setting
• Create a dashboard setting ignore the filters. Should be ON by default on new dashboards and off on old ones. With off on old ones we are making sure that we are not introducing breaking changes.
• Check with our UX team the warning we have in each panel. It will start being very complicated (if it is not already) what is being ignored, what applies etc). Lens has many settings for ignoring / not ignoring and the presence of this dashboard setting can help simplifying the behavior.
• Add the ignore functionality in vega and esql for consistency. I am expecting all my panels to behave the same and neither Vega or ESQL are considered legacy charts.
• Check how it affects other consumers of unfiied search filters

[stratoula]

About my first bullet some screenshots. Check here I have 2 Lens panels, one is using indexA and the other indexB. The advanced setting is set to Ignore.

This means that:

The filter applied on indexA doesnt affect the panelB

The same KQL query though:

So different behavior in case it is filtered by KQL and different if it is by the filter builder. So as I said above, we need a better planning in order to offer this feature nicely to our users, with consistency on the behavior of all panels and all our filtering options.

[teresaalvarezsoler]

So different behavior in case it is filtered by KQL and different if it is by the filter builder

If this doesn't work, it's a bug of the implementation. This issue says it should behave for all filtering options: filter pills, queries and controls.

[teresaalvarezsoler]

@lukasolson mentioned something about alerts not working as expected. Have we discussed with him what he means (Lukas knows this feature better than all of us).

Yes, it's already describe in the Google doc under "Customer impact" as a "very unlikely edge case regarding Alerts"

[dgieselaar]

A request from our side, can we contain the discussion to a single place? We've got two google docs now and a public issue, preferably there's just one place where keep track of everything (as long as feasible).

[stratoula]

So different behavior in case it is filtered by KQL and different if it is by the filter builder

If this doesn't work, it's a bug of the implementation. This issue says it should behave for all filtering options: filter pills, queries and controls.

My point exactly Teresa! This needs more work, we can't just change the default setting. Btw the advanced setting was designed (poorly) to work with the filter builder only afaik (it was before my time). We actually have an ER for this #131514

[cqliu1]

Closing in favor of #189622. We're no longer planning to change the default value of this advanced setting. We plan to introduce a new dashboard level setting to achieve the same behavior.