Saved object import of beats dashboards is not working on 7.0.0 rc1

[bhavyarm]

• Version: 7.0.0 RC1 BC1
• Operating System: darwin_x_86_64
• Steps to Reproduce:

I am not yet sure if this is a bug. Logging it so somebody from the beats team can confirm it.

The exported beats dashboards through saved objects UI in Kibana from 5.6.16 are not working in 7.0.0. I did use the reindex helper in 5.6.16 to get them 6.7.0 BC7. They worked fine but on 7.0.0 - they are not working.

1. Get es/kibana/metricbeat/heartbeat stack going in 5.6.16 (bc1)
2. add the beats dashboards and run both the beats
3. Use the upgrade assistant in 5.6.16 to reindex .kibana
4. export the saved objects json using saved objects UI
5. Shut down 5.6.16 and get the 6.7.0 BC7 stack going for es/kibana/metricbeat/heartbeat
6. run beats but don't add dashboards - create index patterns in Kibana for heartbeat and metricbeat
7. import the exported json from step 4 and associate with the correct index pattern while resolving missing index patterns
8. Open dashboards - everything looks good
9. Check the upgrade assistant - there are no errors
10. Export the beats dashboards using saved objects UI
11. Get the 7.0.0 stack going for es/kibana/metricbeat/heartbeat
12. run beats but don't add dashboards - create index patterns in Kibana for heartbeat and metricbeat
13. import the exported json from step 9 and associate with the correct index pattern while resolving missing index patterns
14. Open one of the beats dashboard
15. Kibana displays could not locate that index pattern field

[bhavyarm]

We seem to be missing information about Add an opt-in BWC layer in 7.x.

[bhavyarm]

tagging @cjcenizal as we went through this issue together!

[ruflin]

@bhavyarm I assume you are referring to using migration.enabled=true in 7.0 to use the old dashboards? When you enabled the flag, did it work?

[bhavyarm]

cc @LeeDr @ruflin this didn't work but I haven't tested it on BC3 which has the fix for this bug: #11271. I am going to take a look at it tomorrow morning if nobody else looked at it. Thanks!

[narph]

@bhavyarm , I have tried reproducing this scenario.
Somehow it worked only once and I assume some caching was involved here. After that, every time I started 6.7.0 stack and import the saved objects I would get the same issues due to the 6.x mapping changes in the heartbeat index.
Let me know if you can reproduce this issue again.

[LeeDr]

@ruflin do you have to set migration.enabled=true in the 6.7.0 *beat.yml before you upgrade to 7.0? Or just in 7.0 before you start the beat?

[ruflin]

@LeeDr No, the flag only exists for 7.0

[ruflin]

I'm stumbling over "Create index pattern for HB / MB in Kibana". Does this mean letting Kibana create the index pattern? This should not be done, always the index pattern created by the Beat should be used.

[ruflin]

After chatting with @narph I think we know have a better understanding on what is happening.

Kibana dashboards and index patterns should also be in sync. So a 5.6 index pattern fits to 5.6 dashboards, 6.5 index pattern to 6.5 dashboards. When exporting in 5.6, it seems the index pattern is not exported so we end up in the situation that users have to generated the index pattern manually (which they shouldn't). So going from 5 to 6, users should just use the dashboards and index pattern shipped with 6.x and load the index pattern again.

As far as I'm aware, the above has not really popped up so far with users migrating from 5 to 6. My assumption is that most users want to use the new dashboards with more features anyway, so export / import is not needed.

For the migration to 7: From 6.7 to 7.0 it seems like the migration is working if we don't have a manually created index pattern because of the 5.6 migration. The part I wonder in general is why do we export / import dashboards for the migration from 6 to 7? Why not just migrate with them in Kibana?

One thing @narph discovered, that when importing dashboards in 7 with a half broken 6.x index pattern and having 7.0 data in Elasticsearch, it seems for some reason the import of the dashboards is successful but some visualisations are missing. The visualisations are in the initial json file which is imported but don't show up in the Save Objects list. Still the UI does not show any errors. Perhaps someone from the Kibana team could shed some lights on this?

[bhavyarm]

I'm stumbling over "Create index pattern for HB / MB in Kibana". Does this mean letting Kibana create the index pattern? This should not be done, always the index pattern created by the Beat should be used.

@ruflin that was a manual step. I assumed that user doesn't want to load the new kibana dashboards so didn't configure my kibana.yml to load dashboards. Which means - I created the indexpattern manually. The docs do not mention that user is not supposed to create index patterns by themselves? Or I am missing it somewhere? Thanks!

[bhavyarm]

@narph @ruflin - this is a different scenario? What does half broken 6.x index pattern means here?
Thanks!

One thing @narph discovered, that when importing dashboards in 7 with a half broken 6.x index pattern and having 7.0 data in Elasticsearch, it seems for some reason the import of the dashboards is successful but some visualisations are missing. The visualisations are in the initial json file which is imported but don't show up in the Save Objects list. Still the UI does not show any errors. Perhaps someone from the Kibana team could shed some lights on this?

[ruflin]

@bhavyarm Good point about the docs, especially as we had in the past also the issue that some users created their own index pattern and had issues. @dedemorton Should we add something around that?

Half broken index pattern in 6.x I mean the self generated index pattern. Sorry for not being specific here. I think it's the same scenario but a different issue. It did not show up in the testing yet, as it broke earlier.

[bhavyarm]

@ruflin we did fix a similar bug during upgrading from 5.6 to 6.0 - elastic/kibana#14592.

[LeeDr]

If we try to list out the main possible cases during upgrade, I think they would be;

1. user upgrades the full stack but wants to keep their existing index patterns, visualizations, and dashboards
2. user upgrades but wants to run setup to install the latest and greatest index patterns, visualizations, and dashboards
3. (this bug) user exports before upgrade, and wants to import those things after upgrade

I would think #1 would be the majority of users. They very likely have customized things and don't want the upgrade to break anything.

@ruflin suggests #2 might be most likely. In this case, does running the beats setup automatically overwrite all the saved objects? Or do they not overlap perfectly and some old save objects are left after setup of new saved objects? Or are we documenting that the user should delete all the previous version saved objects first?

#3 seems least likely, but might be used in the case where a business wants to test their data on the upgrade version on a separate environment before upgrading their production system. In this case they might install a fresh stack of the upgrade version and want to import the saved objects from their earlier version production system.

In the end, I think they all should be supported use cases.

[ruflin]

@LeeDr Agree, all 3 should be supported and be working. Problem with #3 in 5.6 is that the index pattern is not exported which makes it incomplete.

So far I think we only added new visualisations / dashboards, so all should be overwritten. But for example when going from 6.7 to 7.0, there are not Heartbeat dashboards anymore by default. So these would be left overs. But I would also not expect these to affect the import.

I'm worried a bit around #3 as the testing by @narph showed that not everything is imported, but no errors are shown. @LeeDr do you know who from the Kibana team we should pull in here?

@bhavyarm Interesting, seems we hit this from 5 to 6 before which is good to know. Do you know what the expected behaviour in 7.0 during the import is now?

[narph]

hi @bhavyarm , regarding,

@narph @ruflin - this is a different scenario? What does half broken 6.x index pattern means here?
Thanks!

I have followed all the steps in the description (with the specific stack versions) and from step 4 I have the following file export5616.zip and from steps 10 the file export67.zip .
When I import the export67.json file in 7.0 and I either do not have the index patterns manually created or I chose to overwrite everything I am able to import only 107 saved objects. If I have manually created the 2 index patterns and chose to resolve the conflicts then I am able to import 110 objects (the 3 missing saved objects seem to come from the Heartbeat HTTP monitoring dashboard).
The import on these visualizations fails silently, I can attach the kibana debug logs if necessary.
@bhavyarm , I wonder if you can reproduce this behavior as well or the exported files look different in your case.

[bhavyarm]

@narph @ruflin can we please zoom tomorrow to see whats up with this? Thanks!

[bhavyarm]

@narph @ruflin I didn't have any silent failures. I can see that all my saved objects got exported. But some of the vizs are not getting displayed. So I am logging a bug in kibana repo and will tag you. Also heartbeat dashboard doesn't work between 5.6.x to 6.7.0. Will talk to kibana app team about it too. Thanks

[bhavyarm]

elastic/kibana#34129

[narph]

closing issue, PR merged