Skip to content
This repository has been archived by the owner on Apr 12, 2022. It is now read-only.

Unable to Uninstall X-Pack with elasticsearch:5.2.2 Docker Image #36

Closed
frekele opened this issue Mar 13, 2017 · 11 comments
Closed

Unable to Uninstall X-Pack with elasticsearch:5.2.2 Docker Image #36

frekele opened this issue Mar 13, 2017 · 11 comments

Comments

@frekele
Copy link

frekele commented Mar 13, 2017

When I try to remove x-pack in the official image, I get the following error.

Dockerfile
FROM docker.elastic.co/elasticsearch/elasticsearch:5.2.2
.....

Step 5 : RUN /usr/share/elasticsearch/bin/elasticsearch-plugin remove x-pack

 ---> Running in 3c08ad904d0f

-> Removing x-pack...

[91mException in thread "main" java.nio.file.AtomicMoveNotSupportedException: /usr/share/elasticsearch/plugins/x-pack -> /usr/share/elasticsearch/plugins/.removing-x-pack: Cross-device link
	at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:394)
	at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
	at java.nio.file.Files.move(Files.java:1395)
	at org.elasticsearch.plugins.RemovePluginCommand.execute(RemovePluginCommand.java:88)
	at org.elasticsearch.plugins.RemovePluginCommand.execute(RemovePluginCommand.java:59)
	at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122)
	at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:69)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122)
	at org.elasticsearch.cli.Command.main(Command.java:88)
	at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:47)
[0m
Removing intermediate container 3c08ad904d0f

I created a repository to built into docker hub to demonstrate this problem.
GitHub Repo: https://github.com/frekele/docker-elasticsearch
Docker Hub: https://hub.docker.com/r/frekele/elasticsearch/

You can see inside the build log the error:

Related Issues with deprecated repository:

@rlamarche
Copy link

Hi,

I had the same problem and I fixed it with the following Dockerfile :

FROM docker.elastic.co/elasticsearch/elasticsearch:5.2.2

RUN \
  mv /usr/share/elasticsearch/plugins/x-pack /usr/share/elasticsearch/plugins/.removing-x-pack && \
  mv /usr/share/elasticsearch/plugins/.removing-x-pack /usr/share/elasticsearch/plugins/x-pack && \
  /usr/share/elasticsearch/bin/elasticsearch-plugin remove x-pack

I found this solution here : docker-library/elasticsearch#141

@dliappis
Copy link
Contributor

Hello,

Please see the suggestion here: #35 (comment) for a workaround.

We identified an issue with the elasticsearch-plugin command, now addressed in elasticsearch which will be available starting with the next release.

@tylerjharden
Copy link

@dliappis While this issue is closed, in the latest Elasticsearch 5.4.0, while you can now remove x-pack with the remove command, it still causes the same file system errors the next time you start the container for elasticsearch.

@dliappis
Copy link
Contributor

dliappis commented May 5, 2017

@tylerjharden I've reopened this issue and thanks for reporting it.
Are you creating a customized image based on docker.elastic.co/elasticsearch/elasticsearch:5.4.0? Can you please share your Dockerfile or docker-compose.yml to help us replicate the issue?

@dliappis dliappis reopened this May 5, 2017
@dliappis
Copy link
Contributor

dliappis commented May 5, 2017

@tylerjharden For ref, I tried to replicate (unsuccessfully) building a customized image using the following simple Dockerfile:

$ cat Dockerfile 
FROM docker.elastic.co/elasticsearch/elasticsearch:5.4.0

RUN /usr/share/elasticsearch/bin/elasticsearch-plugin remove x-pack

Then built a customized image using:

$ docker build -t myes .
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM docker.elastic.co/elasticsearch/elasticsearch:5.4.0
 ---> 4c093701c312
Step 2/2 : RUN /usr/share/elasticsearch/bin/elasticsearch-plugin remove x-pack
 ---> Running in b982900fe377
-> removing [x-pack]...
-> preserving plugin config files [/usr/share/elasticsearch/config/x-pack] in case of upgrade; delete manually if not needed
 ---> a6c441173d8d
Removing intermediate container b982900fe377
Successfully built a6c441173d8d

Which ended up running successfully:

$ docker run --rm -e http.host=0 -e transport.host=127.0.0.1 myes
[2017-05-05T14:37:29,499][INFO ][o.e.n.Node               ] [] initializing ...
[2017-05-05T14:37:29,556][INFO ][o.e.e.NodeEnvironment    ] [qQyinaW] using [1] data paths, mounts [[/ (/dev/mapper/docker-253:1-9175146-a1a8d716fd3807fc6472ea4ef784e9a52c8509adfeefa1b9c57277051fc7db07)]], net usable_space [9.5gb], net total_space [9.9gb], spins? [possibly], types [xfs]
[2017-05-05T14:37:29,556][INFO ][o.e.e.NodeEnvironment    ] [qQyinaW] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-05-05T14:37:29,557][INFO ][o.e.n.Node               ] node name [qQyinaW] derived from node ID [qQyinaWjSpCmsgegs-CIag]; set [node.name] to override
[2017-05-05T14:37:29,557][INFO ][o.e.n.Node               ] version[5.4.0], pid[1], build[780f8c4/2017-04-28T17:43:27.229Z], OS[Linux/4.10.13-100.fc24.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_131/25.131-b11]
[2017-05-05T14:37:30,141][INFO ][o.e.p.PluginsService     ] [qQyinaW] loaded module [aggs-matrix-stats]
[2017-05-05T14:37:30,142][INFO ][o.e.p.PluginsService     ] [qQyinaW] loaded module [ingest-common]
[2017-05-05T14:37:30,142][INFO ][o.e.p.PluginsService     ] [qQyinaW] loaded module [lang-expression]
[2017-05-05T14:37:30,142][INFO ][o.e.p.PluginsService     ] [qQyinaW] loaded module [lang-groovy]
[2017-05-05T14:37:30,142][INFO ][o.e.p.PluginsService     ] [qQyinaW] loaded module [lang-mustache]
[2017-05-05T14:37:30,142][INFO ][o.e.p.PluginsService     ] [qQyinaW] loaded module [lang-painless]
[2017-05-05T14:37:30,142][INFO ][o.e.p.PluginsService     ] [qQyinaW] loaded module [percolator]
[2017-05-05T14:37:30,142][INFO ][o.e.p.PluginsService     ] [qQyinaW] loaded module [reindex]
[2017-05-05T14:37:30,142][INFO ][o.e.p.PluginsService     ] [qQyinaW] loaded module [transport-netty3]
[2017-05-05T14:37:30,142][INFO ][o.e.p.PluginsService     ] [qQyinaW] loaded module [transport-netty4]
[2017-05-05T14:37:30,142][INFO ][o.e.p.PluginsService     ] [qQyinaW] loaded plugin [ingest-geoip]
[2017-05-05T14:37:30,143][INFO ][o.e.p.PluginsService     ] [qQyinaW] loaded plugin [ingest-user-agent]
[2017-05-05T14:37:31,356][INFO ][o.e.d.DiscoveryModule    ] [qQyinaW] using discovery type [zen]
[2017-05-05T14:37:31,668][INFO ][o.e.n.Node               ] initialized
[2017-05-05T14:37:31,668][INFO ][o.e.n.Node               ] [qQyinaW] starting ...
[2017-05-05T14:37:31,803][INFO ][o.e.t.TransportService   ] [qQyinaW] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2017-05-05T14:37:34,857][INFO ][o.e.c.s.ClusterService   ] [qQyinaW] new_master {qQyinaW}{qQyinaWjSpCmsgegs-CIag}{CHlkG3CNSeixioBbogSUng}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-05-05T14:37:34,887][INFO ][o.e.h.n.Netty4HttpServerTransport] [qQyinaW] publish_address {172.17.0.2:9200}, bound_addresses {0.0.0.0:9200}
[2017-05-05T14:37:34,890][INFO ][o.e.n.Node               ] [qQyinaW] started
[2017-05-05T14:37:34,904][INFO ][o.e.g.GatewayService     ] [qQyinaW] recovered [0] indices into cluster_state

Looking forward to your replication steps.

@tylerjharden
Copy link

tylerjharden commented May 5, 2017

@dliappis

So I was able to fix the issue, but my fix exposes that what is declared fixed (fallback to non-atomic removal for non top-layer file systems), is not working as expected.

The version of the Dockerfile we had in use that prompted this issue is as follows:

FROM docker.elastic.co/elasticsearch/elasticsearch:5.4.0

COPY docker-healthcheck /usr/local/bin/docker-healthcheck

COPY /config /usr/share/elasticsearch/config

RUN elasticsearch-plugin remove x-pack

# cleanup unused modules to slim image
RUN rm -rf modules/aggs-matrix-stats
RUN rm -rf modules/ingest-common
RUN rm -rf modules/lang-expression
RUN rm -rf modules/lang-groovy
RUN rm -rf modules/lang-mustache
RUN rm -rf modules/percolator
RUN rm -rf modules/transport-netty3

On my own, after tracing several issues around where the root cause (in the Linux kernel) was described in a docker hub GitHub issue that I am now having a hard time finding, it was stated that you can circumvent this by ensuring that the command is run in the top layer of the docker image, and by doing so, I am able to get it working:

FROM docker.elastic.co/elasticsearch/elasticsearch:5.4.0

COPY docker-healthcheck /usr/local/bin/docker-healthcheck

COPY /config /usr/share/elasticsearch/config

# cleanup unused modules to slim image
RUN rm -rf modules/aggs-matrix-stats
RUN rm -rf modules/ingest-common
RUN rm -rf modules/lang-expression
RUN rm -rf modules/lang-groovy
RUN rm -rf modules/lang-mustache
RUN rm -rf modules/percolator
RUN rm -rf modules/transport-netty3

# must remove x-pack as the last command so that it can use atomic move operations as top file system layer
RUN elasticsearch-plugin remove x-pack

So the reason your replication attempt was unsuccessful is because you perform no layers after the x-pack removal, add something simple that touches file system (or maybe even any layer period, I did not try to make it fail multiple ways since it is fixed for us using this 'hack'), but who knows.

@dliappis
Copy link
Contributor

dliappis commented May 5, 2017

@tylerjharden Thank you very much for the detailed steps. This will help us add steps to the current x-pack plugin removal acceptance test.

I did try the Dockerfile you experienced the failures with (only change I used config/ instead of /config and I didn't have too many files under /config, pretty much what there is already now in the image) and it worked fine[1].

However, those layering problems do occasionally surface depending on the storage driver.
In my case, I am using the devicemapper storage driver, with LVM thin pools on 17.04.0-ce on Linux.

I'd definitely be interested to get an output of your docker info, which will show the storage driver, host etc, docker version etc. to try to nail this down.

[1]

$ tree
.
├── config
│   ├── elasticsearch.yml
│   ├── jvm.options
│   └── log4j2.properties
├── Dockerfile
└── docker-healthcheck

1 directory, 5 files
$ cat Dockerfile 
FROM docker.elastic.co/elasticsearch/elasticsearch:5.4.0

COPY docker-healthcheck /usr/local/bin/docker-healthcheck

COPY config /usr/share/elasticsearch/config

RUN elasticsearch-plugin remove x-pack

# cleanup unused modules to slim image
RUN rm -rf modules/aggs-matrix-stats
RUN rm -rf modules/ingest-common
RUN rm -rf modules/lang-expression
RUN rm -rf modules/lang-groovy
RUN rm -rf modules/lang-mustache
RUN rm -rf modules/percolator
RUN rm -rf modules/transport-netty3
$ docker build -t myes .
Sending build context to Docker daemon  9.728kB
Step 1/11 : FROM docker.elastic.co/elasticsearch/elasticsearch:5.4.0
 ---> 4c093701c312
Step 2/11 : COPY docker-healthcheck /usr/local/bin/docker-healthcheck
 ---> 8f69e6de4797
Removing intermediate container 056777fc74a1
Step 3/11 : COPY config /usr/share/elasticsearch/config
 ---> 5d65fa4369e5
Removing intermediate container 6e87fee5d5fe
Step 4/11 : RUN elasticsearch-plugin remove x-pack
 ---> Running in e6ccc7aeba85
-> removing [x-pack]...
-> preserving plugin config files [/usr/share/elasticsearch/config/x-pack] in case of upgrade; delete manually if not needed
 ---> 2113aac47bb2
Removing intermediate container e6ccc7aeba85
Step 5/11 : RUN rm -rf modules/aggs-matrix-stats
 ---> Running in e5a53bb3c12f
 ---> fc0cc2c48154
Removing intermediate container e5a53bb3c12f
Step 6/11 : RUN rm -rf modules/ingest-common
 ---> Running in 05e14babc8db
 ---> 2742c8fb21a6
Removing intermediate container 05e14babc8db
Step 7/11 : RUN rm -rf modules/lang-expression
 ---> Running in b199793bb91d
 ---> 54a74905b11c
Removing intermediate container b199793bb91d
Step 8/11 : RUN rm -rf modules/lang-groovy
 ---> Running in cea68617eff0
 ---> f725a9e9e866
Removing intermediate container cea68617eff0
Step 9/11 : RUN rm -rf modules/lang-mustache
 ---> Running in b03a1ddf2111
 ---> a56a58dadfef
Removing intermediate container b03a1ddf2111
Step 10/11 : RUN rm -rf modules/percolator
 ---> Running in 25a3904c09bf
 ---> 5bc3f05348e8
Removing intermediate container 25a3904c09bf
Step 11/11 : RUN rm -rf modules/transport-netty3
 ---> Running in 9e294d7179c6
 ---> 371763f95741
Removing intermediate container 9e294d7179c6
Successfully built 371763f95741
$ docker run --rm -p 9200:9200  myes
[2017-05-05T16:59:05,979][INFO ][o.e.n.Node               ] [] initializing ...
[2017-05-05T16:59:06,040][INFO ][o.e.e.NodeEnvironment    ] [e97U3I5] using [1] data paths, mounts [[/ (/dev/mapper/docker-253:1-9175146-b0b8f7d8eb8620e88e65a379f36a00f90cf749daae35558d314e8a99fa01462a)]], net usable_space [9.5gb], net total_space [9.9gb], spins? [possibly], types [xfs]
[2017-05-05T16:59:06,041][INFO ][o.e.e.NodeEnvironment    ] [e97U3I5] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-05-05T16:59:06,042][INFO ][o.e.n.Node               ] node name [e97U3I5] derived from node ID [e97U3I5RS7GAGV9_2KAoLg]; set [node.name] to override
[2017-05-05T16:59:06,042][INFO ][o.e.n.Node               ] version[5.4.0], pid[1], build[780f8c4/2017-04-28T17:43:27.229Z], OS[Linux/4.10.13-100.fc24.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_131/25.131-b11]
[2017-05-05T16:59:06,472][INFO ][o.e.p.PluginsService     ] [e97U3I5] loaded module [lang-painless]
[2017-05-05T16:59:06,472][INFO ][o.e.p.PluginsService     ] [e97U3I5] loaded module [reindex]
[2017-05-05T16:59:06,472][INFO ][o.e.p.PluginsService     ] [e97U3I5] loaded module [transport-netty4]
[2017-05-05T16:59:06,473][INFO ][o.e.p.PluginsService     ] [e97U3I5] loaded plugin [ingest-geoip]
[2017-05-05T16:59:06,473][INFO ][o.e.p.PluginsService     ] [e97U3I5] loaded plugin [ingest-user-agent]
[2017-05-05T16:59:07,572][INFO ][o.e.d.DiscoveryModule    ] [e97U3I5] using discovery type [zen]
[2017-05-05T16:59:07,882][INFO ][o.e.n.Node               ] initialized
[2017-05-05T16:59:07,882][INFO ][o.e.n.Node               ] [e97U3I5] starting ...
[2017-05-05T16:59:07,991][INFO ][o.e.t.TransportService   ] [e97U3I5] publish_address {172.17.0.2:9300}, bound_addresses {0.0.0.0:9300}
[2017-05-05T16:59:07,995][INFO ][o.e.b.BootstrapChecks    ] [e97U3I5] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-05-05T16:59:11,036][INFO ][o.e.c.s.ClusterService   ] [e97U3I5] new_master {e97U3I5}{e97U3I5RS7GAGV9_2KAoLg}{B6wJbE-NSQa5znEilUbTNg}{172.17.0.2}{172.17.0.2:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-05-05T16:59:11,060][INFO ][o.e.h.n.Netty4HttpServerTransport] [e97U3I5] publish_address {172.17.0.2:9200}, bound_addresses {0.0.0.0:9200}
[2017-05-05T16:59:11,063][INFO ][o.e.n.Node               ] [e97U3I5] started
[2017-05-05T16:59:11,070][INFO ][o.e.g.GatewayService     ] [e97U3I5] recovered [0] indices into cluster_state

@tylerjharden
Copy link

tylerjharden commented May 5, 2017

@dliappis

This occurs using Docker for Mac on MacOS Sierra 10.12.4

output of docker info:

Containers: 93
 Running: 0
 Paused: 0
 Stopped: 93
Images: 568
Server Version: 17.03.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.13-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.787 GiB
Name: moby
ID: F32P:PITZ:SIH6:FFTE:N7RH:73YB:ZWBH:PELS:7BBQ:NPUN:UDUA:YCWB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 30
 Goroutines: 36
 System Time: 2017-05-05T17:18:02.559913348Z
 EventsListeners: 1
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

@daniloakamine
Copy link

If someone needs to uninstall x-pack from docker-compose, this article will help: https://blog.codeship.com/orchestrate-containers-for-development-with-docker-compose/

@frekele frekele closed this as completed Feb 14, 2018
@science695
Copy link

ElasticSearch now has oss versions of their docker images (without xpack).

They do not however, have them listed on their docker images page: https://www.docker.elastic.co/

However, you can add -oss to their links (worked for the version 6 ones) it gives an image without xpack:

docker.elastic.co/elasticsearch/elasticsearch-oss:6.2.3

@dliappis
Copy link
Contributor

dliappis commented Apr 5, 2018

They do not however, have them listed on their docker images page: https://www.docker.elastic.co/

All images can be seen by expanding the listing as shown below:

image

Just hover over the yellow arrow and click on it:

image

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants