jvm.options after upgrade experience

[pickypg]

When installing Elasticsearch, regardless of how you choose to install it, the jvm.options comes along and users are expected to modify it to change the heap settings (at least).

Whenever an upgrade happens where the jvm.options has changed, it is then on the end user to merge the two.

This is a straight-forward process: copy/paste your changes into the new file, then save. In practice, even in upgrades where the difference is called out (e.g., .rpmnew) variants, I have noticed end users missing that the file has changed and after discussing it with @jasontedor, he noted that he has observed the issue too.

The result is that the end user may run into OOM issues, DNS issues, or performance issues that they could have avoided if they simply had the "right" JVM settings for their version.

As a very simple example, imagine upgrading to Elasticsearch 6.8 (122 lines) from Elasticsearch 6.2 (99 lines), but not picking up the new jvm.options along the way. You would be missing the DNS changes that came along in 6.6, as well as this critical workaround:

# temporary workaround for C2 bug with JDK 10 on hardware with AVX-512
10-:-XX:UseAVX=2

This gets even worse if you then upgrade that same node to 7.4.

Jason and I discussed it and I brought up the potential of forcing the version into the file so that it does not start if it does not match, thus forcing attention to it changing, and he mentioned the possibility of a system_jvm.options file (or similar).

I'm opening the issue so this does not get lost, but I am not saying either or both of those options are the best.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Core)

[ywelsch]

Relates #47782 (comment)

[jasontedor]

Another possibility would be to move what we consider "system" JVM options into the JVM options parser, so that they're not exposed to the user at all. There are some advantages and disadvantages to this. One advantage is that it prevents us from having to add yet another configuration file (the system_jvm.options that I proposed when we discussed this in person earlier this week). It also removes the temptation that the user might have to mess with these. However, removing that temptation can be seen as both a feature, and a bug in that it also makes where these options come from invisible to the user.

[jpountz]

We agreed to hardcode such system options, while keeping ways to disable them if they prove problematic.

[pcsanwald]

I removed discuss label based on @jpountz summary and the subsequent PR (#48252 )