[BUG] OpenSearch should clean up temporary files created during segment merge incase segment merge fails

[RS146BIJAY]

Describe the bug
Currently, during segment merge (both during auto merge and force merge), OpenSearch creates multiple temporary files while merging individual segment components. Once individual components of the segments are merged, IndexWriter creates a Compound file (*.cfs) containing all the Lucene components for a segment. At the end, OpenSearch deletes these temporary files.

Now, in case these temporary files consume all the space on the node (because enough amount of space is unavailable for segment merge to go through), segment merge will fail. Also, the files which got created during segment merge do not get cleaned up and it continues to occupy space on the node. This can cause FSHealthService checks to fail for this node, ultimately causing these nodes to be removed from cluster.

Expected behavior
In case Force segment merge fails, OpenSearch should remove the temporary files created during segment merge.

[anasalkouz]

Hi @RS146BIJAY, this Seems to be an important bug, are you actively working on it?

[RS146BIJAY]

@anasalkouz Yes we are actively working on all this issue (all included under the parent META issue)

[anasalkouz]

@RS146BIJAY Any plan to fix again?

[RS146BIJAY]

Yeah. We are analysing where this fix should go in. We started observing few race condtion scenario with above change therefore had to revert the change. Will send the fix for this issue in few weeks time.

[RS146BIJAY]

Created a github issue in Lucene as this issue seems to be introduced in this Lucene commit.

[RS146BIJAY]

Created a RFC document listing possible approaches to gain feedback: #8024