Ability to change the index pattern on a visualization

[stacey-gammon]

Similar to how discover does it. Rather than requiring a new visualization to be built, add a field when editing a visualization that lets you change the index pattern on it. We need to figure out how to handle errors when a visualization was built with fields from the original index that don't exist in the new one.

Splitting this particular part out of this issue: #3668 which includes both dashboards and visualizations. Changing the index pattern on a dashboard is a much different problem and is now covered by this issue: #16917

It will be helpful for us to split these issues out and see which are the most important to the community.

also related: #16831

[ReanimationXP]

I believe the visualization index pattern can already be changed manually by changing its' index guid, but it's extremely non user-friendly considering the compounding issues of (a) those index pattern guids are conveniently not available in the object manager / browser, (b) there is no simple dropdown to change it from within the visualization itself, and perhaps most importantly (c) there is no way to bulk-change multiple visualizations.

[conmurray]

I'll add my request here as well. Im a GOLD licence and Ill raise this with my support engineer but I wanted to pop it here aswell.

• User created a visualization against an index called "logstash_test_metrics"
• We then deleted "logstash_test_metrics" and the index pattern for it aswell.
• We needed to modify the logstash conf file and so re ran the logstash conf file to re-add the logstash_test_metrics index
• We then created the index pattern for this "logstash_test_metrics"
• We'd go to the visualization to be told the underlying index is not there. Actually, it is but because the kibanaSavedObjectMeta.searchSourceJSON now references the index by an ID and not by the actual index name, everything is broken. I have to either recreate the visualization or I have to find the ID for the new index and copy it into the kibanaSavedObjectMeta.searchSourceJSON. This isnt really feasible when I've a LOT of visualizations that all reference the same index.

[conmurray]

Why was this closed? Also the Switch Index on Visualization is also closed so what's the story here? I've referenced this in my Internal Elastic Support case as well.

[stacey-gammon]

This issue isn't closed @conmurray. Switch Index on Visualization was closed because it's a duplicate of this one. Am I overlooking something else?

Regarding your use case above, it almost seems like #16831 is a better fit for it. Just being able to change the index pattern on one visualization in the UI would still require you to open each visualization and change it. If we had an export API that included linked objects you would be able to open the file, do a bulk find/replace, then re-import the file.

[conmurray]

Ah yes I see that. Thanks... @stacey-gammon
With respect to #16831 it doesnt seem to support my particular use case. The fact that you're using ID and not the actual index name is the problem here. Ill add this to the #16831

[PritomAhmed]

+1

[Razikus]

+1

[xingfanxia]

+1

[lambdamusic]

+1
As a side note, I noticed the the index ID in the kibanaSavedObjectMeta does not match any of the indexes I have in elasticsearch. Looks like the ID is created by Kibana when an 'index pattern' gets generated.
Is that right? If so how what's the best way to resolve these mappings?

[yami12376]

+1

[smerzlyakov]

+1 really need this feature. Or even manually change index pattern.

[nerophon]

+1

[canturkisci]

+1 on this please! This is tedious to recover when we have repeat deploys and many dashboards stop working due to the mismatch in index patterns. Intuitively we should be able to point a visualization to an index pattern name, and say "try this instead".

[louieamone]

+1. It would be really great to have a more intuitive and efficient way to do this!

[duffar12]

+1

[aviadbi1]

+1

[r33bl]

+1

[rajesh212]

+1

[ranjitmw]

+1

[0asp0]

+1

[jkamdjou]

For those in search of a temporary solution to restore broken visualizations due to deletion of an index pattern, the following worked for me (Kibana 7.4.2):

1. Navigate to a broken visualization, and take note of the broken index pattern id. It will be in the URL string in the form index:'<uuid>'.
2. Create a new index pattern from Management->Kibana->Index Patterns.
3. Export the index pattern:
   curl -X POST 'http://localhost:5601/api/saved_objects/_export' -H "kbn-xsrf: true" -H "Content-Type: application/json" -d '{"type": "index-pattern"}' > new_index_pattern.ndjson
4. Modify this file and change the top level id to your old index pattern id.
5. Delete the new index pattern you just created in Kibana.
6. Import the index pattern, which now specifies your old index pattern id:
   curl -X POST "localhost:5601/api/saved_objects/_import" -H "kbn-xsrf: true" --form file=@new_index_pattern.ndjson

Your visualizations should now work again.

Further documentation on the import/export APIs can be found here.

[craigboman]

I don't think we need to export the index pattern in order to update a broken visualization. From the list of saved objects in advanced settings, inspect the index pattern using the "..." menu. Viewing the index pattern copy the index pattern id from the url string and paste that into the broken visualization edit window. To paste into the visualization, go to advanced settings->saved objects, then inspect the object (... menu), which should open edit search/visualization. Anytime the visualization is referencing the broken index pattern, replace it with the new index pattern id. I've done this loads of times, works great. I would consider this issue request completed.

[jkamdjou]

I tried that before resorting to the steps I outlined above. That did not work for me, got an error.

[arsalankhalid]

Because of how un-intuitive Kibana visual editors out, after some time I was able to figure out how to fetch the ID. Basically, create a "new" visual with the right index, then go to Saved Objects, click on that new visual you created - and go into "edit" or advanced edit. Where you should see a set of jsons, under references you'll see an ID which shows the new index you want. Now go into the visuals that are breaking, and paste the correct ID in the advanced edit for that visual (inside saved objects, under Management)

[csawtelle]

+1

[PieroValdebenito]

+1

[piehld]

An easier way to change the index pattern being used for a visualization (and across a dashboard) would be extremely valuable, especially in my use case of cloning the same dashboard (of 6+ visualizations) for several different index patterns, each representing a different microservice in our architecture.

Regarding a workaround for people experiencing a dashboard not being able to recognize or locate a particular index pattern, what I found to work the easiest was to just export the whole dashboard from the saved objects list (including all child objects), and then drag and drop that file back onto the 'Import' panel of the saved objects list. The import process will actually ask you to select the correct index pattern to use for each visualization if there are any conflicts or non-matches. Note that this will overwrite the saved objects of the "broken" dashboard you just exported.

That said, I wonder why this functionality of the import feature can't be extended to the feature that this issue is trying to address. It essentially allows you to re-set the index pattern for all visualizations of a dashboard by re-importing it, but only if pattern conflicts or non-matches arise. So, if you were to get rid of those conditions and just require that a new index pattern to be specified every time--and perhaps a new suffix to append to every saved object name, so as to not overwrite--that would accomplish what I believe a lot of users here are voting for.

[sebastian-bugajny]

+1

[KrishnaChaitanya-Gopaluni]

From Stack Management > Saved Objects I've replaced index id for my visualization. It worked.

[timroes]

This enhancement is implemented in Lens since the first release. Lens GA’d in 7.10 and going forward Lens is the recommended way in Kibana to create visualizations, with more chart types, data processing and integrations planned (see this Lens meta issue).

Since this feature is not a good fit for the current Visualize Editor architecture and thus would require major refactoring, we decided to not implement this outside of Lens and thus closing this issue with the implementation in Lens now.

[th0ger]

@KrishnaChaitanya-Gopaluni I don't see Saved Objects under Stack Management. Has it been moved, removed, renamed?
Stack Management 7.10.0.

Edit: It was hidden in Manage Spaces > Features.

[KrishnaChaitanya-Gopaluni]

I

@KrishnaChaitanya-Gopaluni I don't see Saved Objects under Stack Management. Has it been moved, removed, renamed?
Stack Management 7.10.0.

@th0ger My Stack Management is 7.9.3. I can see the saved objects at the bottom left. Seems like 7.10 too has one https://www.elastic.co/guide/en/kibana/current/managing-saved-objects.html

[payne747]

+1
Am new to Kibana and just learning, but I've found a need to do this several times already with no easy method. Would very much welcome it.