Why does this cookbook set the es max heap size to 60% of available memory?

[ejsarge-gr]

I may be completely mis-understanding the code but it appears to me that the cookbook sets the heap size to 60% of available memory when normal advice is to allocate 50%.

cookbook-elasticsearch / attributes / default.rb : line 47 says:
allocated_memory = "#{(node.memory.total.to_i * 0.6 ).floor / 1024}m"

Advice: http://asquera.de/opensource/2012/11/25/elasticsearch-pre-flight-checklist/

[karmi]

Hey, nothing too clever going on, it simply gives the ES_HEAP_SIZE a bit more preference then the filesystem cache, which seemed to work nice on smaller testing instances... We could just do true 1/2 there, no reason not to.

[ejsarge-gr]

Great thanks!

[maf23]

Would it not be a good idea to cap the memory at 30G as well?

On Mon, May 12, 2014 at 6:11 PM, ejsarge-gr [email protected]:

Great thanks!

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/209#issuecomment-42852597
.

[karmi]

@maf23 Hmm, capping it sounds a bit excessive (somebody just wants to have a 128GB heap just to try things out), but we could definitely print out an INFO level log...

[maf23]

What about an interval. If 60% of the memory is between 30 and 48G then set
it to 30?

The current cookbook create a bad value if the machine has between 50 and
80 gigs of memory. Like the hi1.4xlarge and i2.2xlarge machines on Amazon.

On Tue, May 13, 2014 at 7:44 AM, Karel Minarik [email protected]:

@maf23 https://github.com/maf23 Hmm, capping it sounds a bit excessive
(somebody just wants to have a 128GB heap just to try things out, but
we could definitely print out an INFO level log...)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/209#issuecomment-42919291
.

[karmi]

@maf23 Hmm, that's an interesting idea, right. I would probably keep it for a refactored, rewritten cookbook...

[maf23]

Then I'll implement it in my wrapper ;)

On Tue, May 13, 2014 at 7:55 AM, Karel Minarik [email protected]:

@maf23 https://github.com/maf23 Hmm, that's an interesting idea, right.
I would probably keep it for a refactored, rewritten cookbook...

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/209#issuecomment-42919867
.

[sciurus]

I came here to note that capping the default value for allocated_memory at 32GB might be desirable. On a c3.8xlarge the current calculations result in a 35GB heap. There's pretty strong warning against going above 32GB in the ES docs. You can still allow people to manually set allocated_memory to whatever they want while ensuring the default value is safe. Something like the following in attributes/default.rb

potential_allocation = (node.memory.total.to_i * 0.6 ).floor / 1024
if potential_allocation > 32768
  allocated_memory = "32768m"
else
  allocated_memory = "#{potential_allocation}m"
end
default.elasticsearch[:allocated_memory] = allocated_memory

[karmi]

@sciurus Yeah, I absolutely agree on this point, and I do agree that people can still set the attribute directly. The only problem is I don't have the bandwidth to properly test changes like this right now -- the testing support of the cookbook is too simple and weak. It's definitely something I'd add to to any future, refactored cookbook, as I've noted above, if that's OK.

[sciurus]

Sounds good.