Temporary TSM files not cleaned up when compaction rename fails

[stuartcarnie]

As a result of analyzing #11339, it was determined there is an issue with the final stage of compaction during the exchange of old with new TSM files. Specifically, if an error occurs during the FileStore.replace call, which is called indirectly via ReplaceWithCallback in the compactGroup API:

influxdb/tsdb/engine/tsm1/engine.go

Lines 2209 to 2214 in aa3dfc0

	if err := s.fileStore.ReplaceWithCallback(group, files, nil); err != nil {
	log.Info("Error replacing new TSM files", zap.Error(err))
	atomic.AddInt64(s.errorStat, 1)
	time.Sleep(time.Second)
	return
	}

Note that the error is simply logged.

Alternatively, when a level-0 (cache snapshot) is written and Replace fails, the temporary files are cleaned up:

influxdb/tsdb/engine/tsm1/engine.go

Lines 1925 to 1933 in aa3dfc0

	log.Info("Error adding new TSM files from snapshot. Removing temp files.", zap.Error(err))
	// Remove the new snapshot files. We will try again.
	for _, file := range newFiles {
	if err := os.Remove(file); err != nil {
	log.Info("Unable to remove file", zap.String("path", file), zap.Error(err))
	}
	}
	return err

Per #11339, FileStore.replace failed because the server was low on memory and was unable to mmap the new TSM files.