Upgrade Assistant: Elasticsearch deprecation reindex fails with illegal argument exception for index.mapper

[liza-mae]

Kibana version: 7.16.0

Elasticsearch version: 7.16.0

Server OS version:
Cloud

Browser version:
Chrome

Browser OS version:
Latest
Original install method (e.g. download page, yum, from source, etc.):
Staging
Description of the problem including expected versus actual behavior:
In UA, for upgrade path 5.5.16 -> 6.8.20 -> 7.16.0, under Elasticsearch deprecation issues reindex is failing for .reindexed-v6-kibana-6 index

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Setting index.mapper.dynamic was removed after version 6.0.0"}],"type":"illegal_argument_exception","reason":"Setting index.mapper.dynamic was removed after version 6.0.0"},"status":400}

[elasticmachine]

Pinging @elastic/kibana-stack-management (Team:Stack Management)

[cjcenizal]

@yuliacech or @sabarasaba Would one of you mind looking into this one? Thank you!

[sabarasaba]

Thanks for reporting this @liza-mae! I tried following the same path you did from 5.5.16 to 6.8.20 (using this migration guide) and then from 6.8.20 to 7.16.0 using the old UA. Once it was all up and running, I indeed run into this error when I tried reindexing it the first time. But upon closing inspection of the error returned it seems that another deprecation should be fixed first before addressing this one, in my case I was able to see it in the ES deprecations list from UA:

Finally, once I fixed that deprecation by setting "index.mapper.dynamic": null in the index settings I was able to successfully reindex .reindexed-v6-kibana-6.

Though I see that you don't have that deprecation in your screenshot, did it just never showed up or did you manually solved it before? If it never showed up, I think it might be worth to raise an issue with the ES folks about it.

PS: In case it's of any help to reproduce this again, here's the data snapshot I used in 7.16 to recreate this 5.5-data-snapshot.zip

[liza-mae]

@sabarasaba when I last tried it, I was unable to resolve it, sounds like you did though? How?

[sabarasaba]

Yeah I was able to solve it by following the steps that described in the comment above, basically the deprecation you mention in the main issue depends on fixing another deprecation first (remove index.mapper.dynamic from the index itself) before you can re-index it.

[sabarasaba]

Turns out that within the UA reindexing code there's a list of blacklisted settings that get removed before the reindexing runs and this particular setting, that was removed with elasticsearch#51718, is not there. I created a small PR for addressing that that I hope we can merge soon 🚀

[sabarasaba]

Fixed with #119441