Make Docker images suitable for ESS/ECE

[pugnascotia]

Cloud currently builds and runs a fully custom Elasticsearch image. The Stack image should provide an image suitable for direct use instead.

User / Group IDs

Cloud currently expects to be able to set the user and group IDs via environment variables. The entry point script changes the founduser user's UID and GID to those provided in the env vars, and performs a recursive chown on some directories in the container.

The Stack image takes a different approach, in that it follows the OKD image guidelines and uses GID 0 for all files.

The different in approaches may well be due to Cloud's multi-tenancy. I don't know how OKD handles this.

(There is a related open issue, "Pin USER in Dockerfile complying with Docker best practices" (#46166).)

setuid flags

Cloud ensures that there are no files with setuid, in order to mitigate "stackclash" attacks. Basically, they do this:

RUN find / -xdev -perm -4000 -exec chmod u-s {} +

This could be done in the Stack image.

Init process

Cloud runs Elasticsearch via a mini-init process in order to avoid zombie processes. There should be no harm in adopting this in the Stack image.

JVM options

Cloud comments out the heap settings and disables the HeapDumpOnOutOfMemoryError flag in jvm.options. Cloud is also passing the relevant heap options on the command line so it's not necessary to comment out the heap settings (this has been the case since 5.0). It does has the advantage that the resulting Elasticsearch command line flags is tidier. I don't know if that's important to Cloud.

Quota-aware filesystem

Cloud changes the default filesystem provider to an implementation that is aware of user quotas. This is achieved by providing a file to the container that contains the quota settings, which the provider then takes into account then answering capacity questions. The file has a default location, which can be changed via a system property. The filesystem provider is changed via a JVM option, set via ES_JAVA_OPTS.

Cloud has suggested that the Stack provide "a native and equivalent way to deal with this". There is a Stack Overflow question that suggests that Java lacks support for this at present. It ought to be possible to perform native calls on Linux (and possibly other operating systems) that return quota information, but it's unclear how feasible / time consuming that will be. Cloud can continue using their current solution without modifying the base image.

Plugins

The Cloud Docker build process downloads and adds plugins to the image, in a "plugin-archive" directory. Then, when an allocator run an Elasticsearch image, it runs the elasticsearch-plugin tool to install the active plugins from the archive directory. User-provided plugins are downloaded and installed dynamically.

There's nothing obvious that the Stack image needs to do here.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Core)

[pugnascotia]

cc @dliappis / @nachogiljaldo for input.

[alpar-t]

@mieciu

[droberts195]

Cloud runs Elasticsearch via a mini-init process in order to avoid zombie processes. There should be no harm in adopting this in the Stack image.

Please can whoever does this work ping me so we make sure the kill order of the "mini-init" doesn't cause problems for ML. This problem has recently affected Cloud and is being fixed for Cloud, but also the identical problem came up as the root cause of #46262, so I'd like to make sure it's not accidentally introduced in a third place.

[pugnascotia]

Talking to @mieciu, we do want the GID 0 approach for the user running ES, which is good. We may need to support UIDs other than 1000 - I'm trying to find out more.

[pugnascotia]

@droberts195 any idea who fixed it in Cloud, or a PR number maybe? Just so I can have a peek.

[droberts195]

any idea who fixed it in Cloud

It was @mieciu. The PR is in a private repo.