Xms and Xmx specified in ES_JAVA_OPTS variable are not taken into account, overloaded by jvm.options definition

[ghost]

• docker info output:
  [rancher@vct2-rancher-host-1 ~]$ docker info
  Containers: 20
  Running: 19
  Paused: 0
  Stopped: 1
  Images: 19
  Server Version: 1.12.3
  Storage Driver: overlay
  Backing Filesystem: extfs
  Logging Driver: json-file
  Cgroup Driver: cgroupfs
  Plugins:
  Volume: local
  Network: null bridge overlay host
  Swarm: inactive
  Runtimes: runc
  Default Runtime: runc
  Security Options: seccomp
  Kernel Version: 4.4.24-rancher
  Operating System: RancherOS v0.7.1
  OSType: linux
  Architecture: x86_64
  CPUs: 4
  Total Memory: 7.798 GiB
  Name: vct2-rancher-host-1
  ID: YMVO:OVCD:A3G4:4ZNI:A2NY:NGIN:IGRF:OTQ6:37DU:ESH7:WZQG:DL6J
  Docker Root Dir: /var/lib/docker
  Debug Mode (client): false
  Debug Mode (server): false
  Registry: https://index.docker.io/v1/
  Insecure Registries:
  devserver2.corp.skysoft-atm.com:5000
  127.0.0.0/8
• docker-compose version: 1.2.0
• Host OS and version:
  NAME="RancherOS"
  VERSION=v0.7.1

Bug description

I am using the following docker-compose.yml file to instanciate one Elasticsearch node:

version: '2'
services:
  elasticsearch-master-1:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.1.1
    container_name: master-1
    labels:
      type: master-node
    environment:
      - cluster.name=${cluster_name}
      - node.name=master-1
      - bootstrap.memory_lock=true
      - discovery.zen.ping.unicast.hosts=elasticsearch-master-2
      - discovery.zen.minimum_master_nodes=2
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - node.master=true
      - node.data=false
      - node.ingest=false
      - xpack.security.enabled=false
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    mem_limit: 1000000000
    cap_add:
      - IPC_LOCK

My expectation is to have a Xms and Xmx with a value of 512m (overloading. However, it seems that the elasticsearch node is started with the default value defined in the jvm.options file. The container shows 2 different Xms and Xmx values:

ps -aef | grep elasticsearch
    1 elastics 257:21 /usr/lib/jvm/java-1.8-openjdk/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Xms512m -Xmx512m -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-5.1.1.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -Ecluster.name=docker-cluster -Ediscovery.zen.ping.unicast.hosts=master-1,master-2 -Enode.data=true -Enode.master=false -Enode.name=data-3 -Ediscovery.zen.minimum_master_nodes=2 -Ebootstrap.memory_lock=true -Expack.security.enabled=false -Enode.ingest=true

Indeed, the bin/es-docker entrypoint only appends the Xmx and Xms in the ES_JAVA_OPTS variable.

Feature Description

[dliappis]

Thank you for your interest in the docker image!
I couldn't replicate your issue. Using the same docker-compose.yml:

$ cat >docker-compose.yml
version: '2'
services:
  elasticsearch-master-1:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.1.1
    container_name: master-1
    labels:
      type: master-node
    environment:
      - cluster.name=ISSUE22
      - node.name=master-1
      - bootstrap.memory_lock=true
      - discovery.zen.ping.unicast.hosts=elasticsearch-master-2
      - discovery.zen.minimum_master_nodes=2
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - node.master=true
      - node.data=false
      - node.ingest=false
      - xpack.security.enabled=false
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    mem_limit: 1000000000
    cap_add:
      - IPC_LOCK

Heap size reported is 494.9MB, see logs below:

dl@MS-7885 ~/source/elastic/tmp/issue#22 $ docker-compose up
Creating network "issue22_default" with the default driver
Creating master-1
Attaching to master-1
master-1                  | [2017-01-04T17:32:40,341][INFO ][o.e.n.Node               ] [master-1] initializing ...
master-1                  | [2017-01-04T17:32:40,412][INFO ][o.e.e.NodeEnvironment    ] [master-1] using [1] data paths, mounts [[/ (/dev/mapper/docker-253:1-9175146-e10c059dddb00af4d8b7611ea6f02d798d752216617026fd6f45e7158adfbee3)]], net usable_space [9.7gb], net total_space [9.9gb], spins? [possibly], types [xfs]
master-1                  | [2017-01-04T17:32:40,413][INFO ][o.e.e.NodeEnvironment    ] [master-1] heap size [494.9mb], compressed ordinary object pointers [true]
master-1                  | [2017-01-04T17:32:40,413][INFO ][o.e.n.Node               ] [master-1] node name [master-1], node ID [is4Spy2ASTGWIgSfqcGxjg]
master-1                  | [2017-01-04T17:32:40,415][INFO ][o.e.n.Node               ] [master-1] version[5.1.1], pid[1], build[5395e21/2016-12-06T12:36:15.409Z], OS[Linux/4.8.15-200.fc24.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_92-internal/25.92-b14]

This can also be checked with a curl request:

bash-4.3$ curl 'http://localhost:9200/_nodes/stats?pretty=true' | grep heap
          "heap_used_in_bytes" : 113252104,
          "heap_used_percent" : 21,
          "heap_committed_in_bytes" : 518979584,
          "heap_max_in_bytes" : 518979584,
          "non_heap_used_in_bytes" : 68603944,
          "non_heap_committed_in_bytes" : 73326592,

As per the docs it is possible to define the heap size via the ES_JAVA_OPTS variable.

[ghost]

Thanks a lot Dimitrios for your quick reply !

I tried to reproduce in a different environment and it works but still I am not convinced by this configuration. I agree that the docs you mentioned precised you can define the heap size via the ES_JAVA_OPTS but it also precises that it should be done after having commented out the Xms and Xmx settings in the jvm.options file :

It is also possible to set the heap size via an environment variable. This can be done by commenting out the Xms and Xmx settings in the jvm.options file and setting these values via ES_JAVA_OPTS:

Actually, I am really not comfortable by having Xms and Xmx defined twice in the same process:

$ docker exec -ti master-1 ps -aef | grep elastic
    1 elastics   0:17  /usr/lib/jvm/java-1.8-openjdk/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Xms512m -Xmx512m -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-5.1.1.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -Ecluster.name=ISSUE22 -Ediscovery.zen.ping.unicast.hosts=elasticsearch-master-2 -Enode.data=false -Enode.master=true -Enode.name=master-1 -Ediscovery.zen.minimum_master_nodes=2 -Ebootstrap.memory_lock=true -Expack.security.enabled=false -Enode.ingest=false
   80 elastics   0:00 ps -aef

I am not an expert in JVM configuration but I think it's quite hazardous to predict the behavior even in a docker image. That's my feeling and I will probably create a new image extending the elastic image with a proper jvm.options file.

Thanks again for your support.

[dliappis]

@dchambon Doing a customized image with your own jvm.options is definitely a good approrach and there is an example here.

Alternatively, if you can provision the needed jvm.options file on the hosts you can also bind mount that file as shown here. You can either hard code the memory there, or leave it blank and still get your settings honored with ES_JAVA_OPTS.

The double -Xmx -Xms definitions are due to how the startup elasticsearch script works; the params defined in ES_JAVA_OPTS get appended after the ones in jvm.options. Latest openjdk versions (including musl java used in the Alpine image) honor the right most one, so that's why it works:

$ docker run --rm -ti docker.elastic.co/elasticsearch/elasticsearch:5.1.1 /bin/bash
bash-4.3$ java -Xms2g -Xmx2g -XX:+PrintFlagsFinal -Xms512m -Xmx512m 2>/dev/null | grep MaxHeapSize
    uintx MaxHeapSize                              := 536870912                           {product}

However, I do agree that this is not a good practice. I will look into it, possibly addressing it in the documentation for starters.

[jasontedor]

The double -Xmx -Xms definitions are due to how the startup elasticsearch script works; the params defined in ES_JAVA_OPTS get appended after the ones in jvm.options.

This is correct. It it recommend but not necessary to comment out the heap settings in the jvm.options file; we deliberately ordered the parameters first the jvm.options and then the ES_JAVA_OPTS options so that the latter will take precedence. We will maintain this functionality (it's very useful when you're developing and testing Elasticsearch to be able to change JVM options without having to edit the jvm.options file).

[aliok]

I was setting heap size to 512m and was getting a 1g memory usage and was going to reopen this ticket.
But f you check top command inside the container, you will see that the reserved memory is heap size + 500mb.
Writing this down for future reference for other people.

Here is top when Xmx and Xms set to 512m:

top - 13:27:03 up 3 days, 17:24,  0 users,  load average: 7.89, 2.80, 1.80
Tasks:   4 total,   1 running,   3 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.9 us,  1.0 sy,  0.0 ni, 93.6 id,  0.0 wa,  0.5 hi,  0.1 si,  0.0 st
KiB Mem : 32303184 total,  7540620 free, 14733088 used, 10029476 buff/cache
KiB Swap: 16220156 total, 16220156 free,        0 used. 15816752 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND     
    1 elastic+  20   0 6373636 977.1m  26692 S   3.0  3.1   0:35.29 java        
  120 elastic+  20   0   72076   7856   7224 S   0.0  0.0   0:00.00 controller  
  190 root      20   0   11788   3016   2644 S   0.0  0.0   0:00.02 bash        
  207 root      20   0   51892   3748   3244 R   0.0  0.0   0:00.00 top

Same command when Xmx and Xms is 1g:

top - 13:37:30 up 3 days, 17:34,  0 users,  load average: 0.91, 1.35, 1.62
Tasks:   4 total,   1 running,   3 sleeping,   0 stopped,   0 zombie
%Cpu(s):  6.8 us,  1.8 sy,  0.0 ni, 90.5 id,  0.0 wa,  0.7 hi,  0.1 si,  0.0 st
KiB Mem : 32303184 total,  5615808 free, 16648184 used, 10039192 buff/cache
KiB Swap: 16220156 total, 16220156 free,        0 used. 13902388 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND     
    1 elastic+  20   0 8152488 2.646g  26580 S   1.0  8.6   0:42.35 java        
  120 elastic+  20   0   72076   7876   7240 S   0.0  0.0   0:00.00 controller  
  170 root      20   0   11788   2932   2652 S   0.0  0.0   0:00.01 bash        
  184 root      20   0   51892   3596   3096 R   0.0  0.0   0:00.08 top 

I think the confusing part is this line in the logs:

elasticsearch    | [2018-01-24T13:24:04,211][INFO ][o.e.n.Node               ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config]

There is -Xms1g, -Xmx1g, which is default and there is -Des.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m appended.

[dliappis]

Relates #43 ; when in doubt, it's best to query the Elasticsearch API about the heap size as shown in this comment: #43 (comment)

[pulpbill]

I can't make it work with es:latest, I set it at the composer but checking /etc/elasticsearch/jvm.options it's still at 2g, same for top at docker host, jvm is using 2g not 6g as I set it.
I can see running "env" inside ES container the variable having 6g assigned, but jmv.options it's not taking it.

[davidkarlsen]

It would be proper 12-factor to be able to provide all settings as vars. Changing the GC won't work this way as you get "multiple gc" settings.