Remove max_num_segments from _forcemerge

[jpountz]

The _forcemerge API has a max_num_segments whose only useful value is 1. It also has the drawback of potentially conflicting with the maximum merged segment size (index.merge.policy.max_merged_segment).

We could remove the max_num_segments setting and make _forcemerge merge down to the minimum number of segments that honors the maximum merged segment size.

Marking as a blocker because of interactions with https://issues.apache.org/jira/browse/LUCENE-7976.

[elasticmachine]

Pinging @elastic/es-distributed

[danielkasen]

I'm not sure you'd want to move the ability to slowly move down to this. Currently, we use a stepping policy as it's more efficient (fast and less memory utilization). So going from 600 -> 20 -> 10 is quicker then going 600 -> 10

[jpountz]

@danielkasen Are you really having shards that have 600 segments?

We made the decision that forcemerge should keep the max_num_segments parameter in #32291, closing.

[danielkasen]

Sry that's the total primary index for a 750gb index. So was this backported to 6.5 then? Or do I have to upgrade to 6.6.0 at this point?

[danielkasen]

So currently on 6.5.4 when ever I try to do a FM it reduces the entire shard to 1 segment. I've been looking for a fix for this ever since this update and am curious about my options at this time for our production systems.

[jpountz]

Thanks for reporting this, I looked into it and this seems to be an unexpected side-effect of https://issues.apache.org/jira/browse/LUCENE-7679. I just opened https://issues.apache.org/jira/browse/LUCENE-8688 to discuss it on the Lucene side.

[jpountz]

@danielkasen LUCENE-8688 will be fixed in Lucene 8.1. Thanks again for reporting.

[danielkasen]

Any chance this will be back ported? As isn't 8.1 a long way out at this point? Just saw there was a vote for 8.0 the other day and trying to get a gauge of what version this would actually get released in and what month. Thank you for fixing this.