node does not start after upgrade to 6.3 with x-pack installed

[immon]

With version 6.3 x-pack is bundled as a module and you do not need to install it as a plugin. For users already using x-pack an upgrade via OS package manager like yum or apt-get does not remove x-pack plugin before installing ES 6.3. This leads to an issue where node does not start and following exception is logged:

[2018-06-15T10:28:26,252][ERROR][o.e.b.Bootstrap ] Exception
java.lang.IllegalStateException: Duplicate key org.elasticsearch.plugins.PluginsService$Bundle@24a783dd
at java.util.stream.Collectors.lambda$throwingMerger$0(Collectors.java:133) ~[?:1.8.0_171]
at java.util.HashMap.merge(HashMap.java:1254) ~[?:1.8.0_171] 

To recover from it one need to:

• remove x-pack /usr/share/elasticsearch/bin/elasticsearch-plugin remove x-pack
• append log4j config of x-pack to the main one: cat /etc/elasticsearch/x-pack/log4j2.properties >> /etc/elasticsearch/log4j2.properties
• move x-pack config files from /etc/elasticsearch/x-pack to /etc/elasticsearch

Can we have an upgrade document updated to explicitly reflect additional steps when upgrading to 6.3? Or it should be handled in post-upgrade script of rpm/deb package?

[elasticmachine]

Pinging @elastic/es-core-infra

[bleskes]

@rjernst do you mind triaging this and see where it should go?

[rjernst]

Some comments on the proposed additional steps:

remove x-pack /usr/share/elasticsearch/bin/elasticsearch-plugin remove x-pack

This is always necessary when upgrading, for all plugins. It is not special to the upgrade including x-pack.

append log4j config of x-pack to the main one: cat /etc/elasticsearch/x-pack/log4j2.properties >> /etc/elasticsearch/log4j2.properties

The package manager should place the new log4j2.properties that comes with elasticsearch (which includes settings for both oss and x-pack) in a separate file on upgrade, or allow you to choose whether you want to replace it. As with all plugins, you will need to manually remove the x-pack log4j2 file, as it is config and not removed on plugin removal.

move x-pack config files from /etc/elasticsearch/x-pack to /etc/elasticsearch

This is not strictly necessary, yet. Xpack settings fall back to their old location, to allow time (until 7.0) to move the files.

I think someone from the docs team (@lcawl?) would be best to decide how/where to document this notes, if we decide to document them.

[bleskes]

This is always necessary when upgrading, for all plugins. It is not special to the upgrade including x-pack.

Are you saying it is already document somewhere?

Given your reaction and the fact that we do want that the 6.2.0+xpack -> 6.3.0 transition will be smooth as possible, I think we should at the very least document this as part of the "upgrade procedure". @lcawl can you advise on where it needs to go?

[rjernst]

Are you saying it is already document somewhere?

Yes, although it could be more clear that "upgrade" means remove and add.

From the rolling upgrade docs:

6. Upgrade any plugins.

Use the elasticsearch-plugin script to install the upgraded version
of each installed Elasticsearch plugin. All plugins must be upgraded
when you upgrade a node.```

[bleskes]

Thanks for clarifying @rjernst and I agree . The Elasticsearch rolling upgrade docs fall under our responsibility - do you mind updating it with explicit commands? (or something you feel is better). I also think it will be good to add a callout note about upgrading x-pack from 6.2 to the default 6.3.

[rjernst]

I believe this was addressed in #32016.