[EPM] Change index pattern ids or adapt UI

[neptunian]

Currently, EPM creates index patterns with unique ids. Eg. for logs-* it is epm-ip-logs. This causes Kibana dashboard app to throw an error if the id is not logs-*. See if we can change safely change the ids or if its an issue with the UI.

[elasticmachine]

Pinging @elastic/ingest-management (Feature:EPM)

[ruflin]

@neptunian I'm good with changing it for now to see if it solves our problem. But I think we also need to fully understand how name of the index pattern and id related as we reference to it in our dashboards / visualization. We must figure out what must be used there and in which fields must be used what. @mtojek Works on some import scripts and found index patterns referenced in quite a few different places.

[neptunian]

Note: Beats creates index patterns with title and id always the same. I had overlooked this initially.

@ruflin Is there a concern that somewhere else is using the same index pattern id/name?

[ruflin]

One of the concerns could be that we accidentially override and other index pattern created by the user. What is more important here I think is to understand where in visualizations / searches etc. are ids used and where is the name used? Or is it by now always id?

[neptunian]

@ruflin

One of the concerns could be that we accidentally override and other index pattern created by the user.

When creating index patterns through the Management app a unique ID is generated such as 4b6f4350-dcac-11ea-9489-d103615882ab and the index pattern itself is the "title". It would probably be best if we did something similar as we changed the ids to be logs-* and metrics-*. Then we would need to update our asset references to point to the more unique ID.

What is more important here I think is to understand where in visualizations / searches etc. are ids used and where is the name used? Or is it by now always id?

Looking at the assets that reference index patterns looks like this:

This is why we updated the index pattern id to match these references, but we probably should have updated these references to point to the unique id. All other places outside these references that are referencing the saved object type and id, we are most likely going to use the name or "title" as we want the value of the actual index pattern. An example in our assets is: "default_index_pattern": "logs-*" or "series_index_pattern": "logs-*" . These seem to always appear in visState attribute of a visualization.. I tried making my own index pattern with a unique id that doesn't match the title of the index pattern and create a visualization that uses it. I can confirm the index pattern references in the visState is the title and not the id.

We can try changing the ids of the index patterns and updating all references to the saved object type index pattern to this id (in the screenshot).

[ruflin]

@neptunian What are the changes needed if we follow your recommendation above?

BTW: I think it is kind of odd but title is used as reference and not the id 🤔

[neptunian]

@ruflin we would need to change the id of the index pattern saved object from logs-* to whatever unique one we decide. We would need to do some kind of migration for that if we don't want to introduce a breaking change. I'd update the package install code to start creating the index pattern with this id. Then we'd need to have that package script update all "references" objects id to be the new id.

BTW: I think it is kind of odd but title is used as reference and not the id 🤔

Are you referring to the screenshot I posted? The Beats code uses the same id and title when creating the index pattern so perhaps it was expected for the id to be the title or actual index pattern value here. For references like "default_index_pattern": "logs-*", I think anywhere it asks for the actual index pattern or title in the UI because the user does not know the unique id. That is where the references in the visState are from, user input.

[ruflin]

Is there an easy way provided by Kibana to update all references or would it mean we have to understand all SO to make sure we put in the right references? Is there a risk that it not only shows up in references but also other places?

I'm starting to wonder if for us it is just easiest to keep also id and title in sync 🤔

[neptunian]

Since you cannot update a SO id through Kibana, I don't think there would be a way to automatically update references to it. I think we would need to update it ourselves, but I think it would only be in this property references. I found this issue and it seems like modifying SO ids can be tricky: #59598 . It lists some other places the id could be. If the user created some assets using the index pattern, they would need to be updated.

I'm starting to wonder if for us it is just easiest to keep also id and title in sync 🤔

I think it is. Other users seem confused by using id vs title. The saved object API to update an index pattern through PUT will only update with a specified ID and not create if it doesn't exist. POST is used to create and you cannot create an index pattern with the same ID as one that already exists so I don't think we'd need to worry about the user accidentally overwriting ours. Since our ID matches the title, it's a way of letting the user know that index pattern title already exists and they shouldn't create one like it. Because if they are trying to use logs-* as an ID they are probably trying to use it as a title as well. It's not possible to have duplicate index pattern titles through Kibana UI but it is through the API, though I don't see any reason someone would want to.

[ruflin]

@neptunian What is your proposed next step?

[neptunian]

@ruflin I think we should leave the id and title the same, as it is now.

[ruflin]

SGTM. Should we close this one?