Missing Add Data welcome / interstitial page

[alexfrancoeur]

Kibana version:
7.14 - Cloud

Elasticsearch version:
7.14

Describe the bug:
The interstitial / welcome page seems to be missing on Cloud (and potentially self managed)

Jul-28-2021.10-09-04.mp4

You are dropped directly in the home page when you should be seeing this view upon login. If there is no data (which there shouldn't be in a net new deployment), this is meant to be the first screen you see.

These APM indices may be the culprit. Without adding anything to the cluster, regardless of my use case, these indices seem to have been added (cc: @ruflin @kobelb not sure if this is Fleet related)

Directing new users to ingestion is a crucial part of the onboarding experience while we work towards our end state for 7.last #94207. The sooner we can get this resolved the better. This is a regression, though I'm not sure if it occurred earlier than 7.14. My assumption is that this behavior is a result of automatically deploying APM server by default.

Steps to reproduce:

1. Create new cloud deployment
2. Open Kibana

cc: @VijayDoshi @thesmallestduck @osmanis @lukeelmers (I believe Core technically owns this page)

[joshdover]

Regardless of the root cause or solution here, this is a critical path that needs to be tested in the suite we run against the Cloud environment, ideally against each type of deployment template.

cc @liza-mae for guidance on how to best accomplish this in a way that we'll actually see (I'm not current sure how we're monitoring integration test results against Cloud right now).

[joshdover]

Just tested this on 7.12.1 and 7.13.4 on Cloud:

• 7.13.4: same behavior as @alexfrancoeur is seeing on the 7.14 BC
  • logs-* and metrics-* index patterns already exist, though no indices are created yet
  • apm indices exist
  • System package assets (dashboards, etc.) are installed
• 7.12.1: add data interstitial is shown, though maybe a bit broken
  • At first the interstitial wasn't show until I started clicking around, then it appeared
  • Was shown immediately on a refresh, seems like a timing or performance issue
  • No index patterns exist
  • apm indices exist

It seems the root cause is #82851 which got worse in 7.13 because Fleet is automatically included in Cloud deployments that have APM server enabled and it automatically creates the default policy & installs the system package.

[joshdover]

Maybe we could fix this by making the "add data" logic actually check if there is any backing indices for the logs-* and metrics-* index patterns? I think we'd need to exclude metrics-endpoint* indices from that check since it's created by default?

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[liza-mae]

Hi @joshdover can you point me to the test for this? I can check if it is being run on cloud, if it is not being run currently, I can help in getting it added.

[mostlyjason]

@graphaelli @Mpdreamz It looks like these indices are for the standalone version of APM since they are not using the new data stream naming scheme. So I don't think Fleet has anything to do with the example shown in the screenshot.

I think the onboarding flow needs to handle the APM and Fleet use cases through when it detects if data has been added. It'd better to check if data actually exists rather than checking if an index exists?

[graphaelli]

Right, new "APM & Fleet" deployments always use standalone APM Server right now though users may migrate to the integration as of 7.14. These indices have always been created on startup so if this really is a regression I suspect something filtered these out previously.

I expect index existence is used for this check because it's a bit cheaper than inspecting index content though a content check would be ideal.

[joshdover]

The APM indices are irrelevant for the check, sorry for the confusion on that. Here is the current logic, it simply checks if any index-patterns exist in the user's current space:

kibana/src/plugins/home/public/application/components/home.js

Lines 78 to 86 in 9d42d67

	const resp = await this.props.find({
	type: 'index-pattern',
	fields: ['title'],
	search: `*`,
	search_fields: ['title'],
	perPage: 1,
	});
	this.endLoading({ isNewKibanaInstance: resp.total === 0 });

I propose we enhance this with this logic:

• Do any index-patterns exist?
  • If not, show add data interstitial.
  • If so, query each index-pattern with GET /<index-pattern>/_search?size=1&filter_path=hits.total, ignoring cases where the user does not have read permissions for the index-pattern
    • If no documents exist in any index-pattern the user has access to, show the interstitial

[joshdover]

Do any index-patterns exist?

• If not, show add data interstitial.

• If so, query each index-pattern with GET /<index-pattern>/_search?size=1&filter_path=hits.total, ignoring cases where the user does not have read permissions for the index-pattern

  • If no documents exist in any index-pattern the user has access to, show the interstitial

Discussed this idea sync with the Fleet Upgrade Packages group and we discussed tweaking this logic to use the _cat indices API for speed. Here's the new logic:

• Do any index-patterns exist?
  • If not, show add data interstitial.
  • If any index-patterns that are not metrics-* or logs-* exist, do not show the interstitial
  • If only metrics-* and logs-* exist, query each index-pattern with GET /_cat/indices/<index-pattern>
    • If an authorization error is thrown, don't show the interstitial. This can happen in the case where the user has either does not have the monitor cluster privilege or monitor index privilege. In either case, the user is likely not to have privileges to add data.
    • Filter out metrics-elastic_agent* and logs-elastic_agent* indices from the response
    • If no other indices exist, show the interstitial

[mostlyjason]

@joshdover I think metrics-endpoint* is for endpoint security. To filter out elastic agent monitoring data remove metrics-elastic_agent-* and logs-elastic_agent-*

[joshdover]

@mostlyjason When I create a new deployment with APM & Fleet enabled in Cloud, I'm only seeing the metrics-endpoint.metadata_current_default index be created by default (by a transform).It's also worth noting that this endpoint index is empty (no documents). If we need to ignore this index, then we'll only want to ignore this index if there are 0 documents.

I do not see any elastic agent indices. When should I expect to see these and should this check ignore those indices or not?

[mostlyjason]

@joshdover you're right that one is enabled by the security app automatically. the agent metrics and logs are disabled by default on elastic cloud, but not in self-managed clusters. we should probably still ignore them since they are not really user data, which is what we are trying to encourage.

[alvarolobato]

This is something we should handle as a blocker for 7.15, is there anyone working on this already?

cc. @ruflin @jen-huang

[joshdover]

This is something we should handle as a blocker for 7.15, is there anyone working on this already?

cc. @ruflin @jen-huang

Yep, I've started work on this today.

[graphaelli]

In case anyone else is unable to get the interstitial to show at all, @joshdover determined that that localstorage also factors in whether to show it - clearing it out with localStorage.clear() resolves that.

[joshdover]

I have a draft that up that I'm doing some manual testing and so far things seem to be good. One caveat is that there is now a new condition where we'll show this UI where we wouldn't have done so before AND there may already be data in the cluster. This condition only happens when all of the following are true:

• The user has never opened Kibana's home page before
• No index-pattern exists in the current Kibana Space
• The user has monitor cluster privileges
• The user has monitor index privileges on any index BUT not metrics-* and/or logs-*
• Data exists in metrics-* or logs-*

In this case, the _cat/indices API is going to return an empty response on the metrics-* and/or logs-* indices because the user does not have permission, however those indices may already exist. Thus, we may show this "add data" modal even if data already exists (but the index-patterns weren't created and the user hasn't been granted access yet). IMO, this seems like a fairly small edge case that doesn't result in anything bad happening, but it does direct the user to add data when they likely don't have the privileges to do so.

We may be able to add a privilege check here and avoid showing this modal for this case, however we don't yet have any authz code like this in our OSS plugins, so I need to check with the Kibana Security team about the recommended way to add this. Unless I hear any objections, I'm going to proceed with leaving this edge case behavior to expedite a fix for 7.14.1.

cc @VijayDoshi @alexfrancoeur

[joshdover]

While the logic for this has been fixed correctly, it seems this new logic is taking anywhere from 1-5s in my testing on Cloud. We currently have logic that skips showing this interstitial if the request takes more than 500ms to avoid blocking this UI. I propose increasing this timeout to 10s for the following reasons:

• Once a user is shown this UI once, we skip making this query in the first place.
• The case where an index pattern already exists is much faster, which minimizes the risk of increasing this timeout.

cc @alexfrancoeur @VijayDoshi I'm going to put up a PR to make these changes now, but FYI this fix may not make the 7.14.1 release unless we consider it a blocker and ask for a new BC.

[liza-mae]

On the test topic, the new functional test added fails on cloud: #110367

[joshdover]

On the test topic, the new functional test added fails on cloud: #110367

Well, glad to see the test works 😅

[alexfrancoeur]

cc @alexfrancoeur @VijayDoshi I'm going to put up a PR to make these changes now, but FYI this fix may not make the 7.14.1 release unless we consider it a blocker and ask for a new BC.

Thanks Josh. Quick question, does this also impact the no-data check for default Kibana assets from Fleet as referenced in #82851 (comment)? As bad as the interstitial page is, I believe the Kibana assets experience is worse.

[joshdover]

Thanks Josh. Quick question, does this also impact the no-data check for default Kibana assets from Fleet as referenced in #82851 (comment)? As bad as the interstitial page is, I believe the Kibana assets experience is worse.

No, this only affects the interstitial.

[alexfrancoeur]

++ great to hear. Since this regression has been around for a few releases now, I do not think we can treat it as a blocker. I do think we should still backport and hope that it gets picked up by potentially a BC2 (seems unlikely) or 7.14.2 patch release.

[joshdover]

I have confirmed that this is fixed and working on the latest 7.15.0 and 7.14.2 snapshots on Cloud Staging ✅