Fixing Event Log file cleanup issue #36
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Add latency and failure metrics for Publish Cluster State Metrics from master's perspective * Update Javadoc * Add new line at the end * Empty Queue before running each test * Addressed comments * Metrics for disabled collector * Fixing bug in MasterClusterStateUpdateStatsCollector Co-authored-by: Arpita Mathur <[email protected]>
yojs
reviewed
Jul 13, 2021
| // In case files deletion takes longer/fails, we are okay with eventQueue reaching | ||
| // its max size (100000), post that {@link PerformanceAnalyzerMetrics#emitMetric()} | ||
| // will emit metric {@link WriterMetrics#METRICS_WRITE_ERROR} and return. | ||
| cleanup(); |
There was a problem hiding this comment.
Because the cleanup is called after the files are written, the first time this thread runs, it will delete even the most recently written files because the call to cleanup will not have the lastCleanupTimeBucket and hence deleteAllFiles will be called deleting everything but the .tmp files.
Because the full cleanup is just a one time cleanup and happens at the start, let's do it in the constructor itself ?
There was a problem hiding this comment.
Agreed. This would cause to lose the first timebucket metrics. I would address this as part of separate PR I will have for StatCollector refactoring.
yojs
approved these changes
Jul 13, 2021
sruti1312
approved these changes
Jul 13, 2021
5 tasks
sruti1312
pushed a commit
that referenced
this pull request
Aug 25, 2021
Fixing Event Log file cleanup issue
sruti1312
added a commit
that referenced
this pull request
Aug 25, 2021
* Create writer file if metrics are available (#31) Signed-off-by: Sruti Parthiban <[email protected]> * Add tests to check for writer file only if metrics are present (#35) Signed-off-by: Sruti Parthiban <[email protected]> * Merge pull request #36 from opensearch-project/khushbr-writer-purge-fix Fixing Event Log file cleanup issue * Moving deleteAllFiles() to inside scheduleExecutor() * Fixing the Link Checker errors, updating the official documentation * nit: Fixing spotlessJava indentation issue * Merge pull request #37 from khushbr/feature/purge-fix Handling purging of lingering files before scheduleExecutor start. * Fix failing file handler test (#38) Signed-off-by: Sruti Parthiban <[email protected]> * Remove dependency on main branch when running spotless. (#47) Signed-off-by: Marc Handalian <[email protected]> * Updates to gradle build file (#48) * Updates to gradle build file Signed-off-by: Sruti Parthiban <[email protected]> * Add ability to specify RCA branch Signed-off-by: Sruti Parthiban <[email protected]> * Fix build when opensearch_version flag is provided. (#52) Signed-off-by: Marc Handalian <[email protected]> * Update the version to 1.0.1 Signed-off-by: Sruti Parthiban <[email protected]> Co-authored-by: Khushboo Rajput <[email protected]> Co-authored-by: Khushboo Rajput <[email protected]> Co-authored-by: Marc Handalian <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Is your feature request related to a problem?
Issue : #33
[performance-analyzer-rca] PR: opensearch-project/performance-analyzer-rca#27
Previous PR [now closed] : #34
Describe the solution you are proposing
purgeQueueAndPersist()invokesdeleteFiles()every filesCleanupPeriod (default to 60s), to cleanup the older event log files and then writes the latest event log files withwriteAndRotate()MetricsPurgeActivityclass instantiation.Describe alternatives you've considered
Another approach was to launch a new thread and invoke 'MetricsPurgeActivity' within it. We will again run into the same issue if this thread dies, thus to keep the cleanup and write within same thread was better.
Testing
Tested by spinning up a docker container. Manually copied 100 dummy files to
/dev/shm/performanceanalyzer/.Enabled DEBUG logs to verify cleanup is working as expected
Metrics:
Metric
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.