Need a more comprehensive max_mem calculation

[pulkitsinghal]

On an AWS m1.small instance, the current max_mem calculation of:

max_mem = "#{(node.memory.total.to_i - (node.memory.total.to_i/3) ) / 1024}m"

yields:

rb(main):002:0> ("1697880kB".to_i - ("1697880kB".to_i/3) ) / 1024
=> 1105
irb(main):003:0> 1105*1024
=> 1131520

So -Xmx1105m isn't so bad on a m1.small (1.7GB memory) instance until I run sudo service elasticsearch status -v and see:

elasticsearch 0.19.3 running with PID 1119
...
Memory Total:        1.6gb
...

I know that your setup is perfect for m1.large but I would still like to see a more universal equation than the current one ... so I'll also try and strain my brain to come up with something that factors in permgen etc. before coming up with a max_mem number to use.

Without such a consideration Monit hits the restart if mem_usage > 90% for 15 cycles condition very easily on non m1.large instances like mine.

[pulkitsinghal]

Upon closer inspection of elasticsearch.init.erb, I realized that Memory Total is the machine's total memory and not the total memory being consumed by ES. But the top command really does show 92.6% consumption ... and the following heap_max_in_bytes + non_heap_max_in_bytes ... doesn't add up to more than 80% of the available memory.

$ curl 'http://localhost:9200/_cluster/nodes/elasticsearch-test-1?os&jvm&pretty'
  "jvm" : {
    ...
    "mem" : {
      "heap_init" : "128mb",
      "heap_init_in_bytes" : 134217728,
      "heap_max" : "1gb",
      "heap_max_in_bytes" : 1157562368,
      "non_heap_init" : "23.1mb",
      "non_heap_init_in_bytes" : 24313856,
      "non_heap_max" : "214mb",
      "non_heap_max_in_bytes" : 224395264
    }
  }

Sigh ... sorry for bugging you, I'm closing this until I have a proper suggestion/analysis to present. Too bad that I couldn't get JConsole running on the AWS instance yet.