More aggressive cleanup of failed delayed jobs #3346
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kathap
force-pushed
the
more-aggressive-cleanup-of-failed-delayed-jobs
branch
6 times, most recently
from
e1d545a to
d6e61e1
Compare
kathap
force-pushed
the
more-aggressive-cleanup-of-failed-delayed-jobs
branch
4 times, most recently
from
aa6a656 to
ec136ff
Compare
Currently, failed delayed_jobs are deleted after 14d (configurable) to keep some info about failed jobs that helps debugging: https://github.com/cloudfoundry/cloud_controller_ng/blob/main/app/jobs/runtime/failed_jobs_cleanup.rb This can still lead to very large number of delayed_jobs records that slow down DB queries working on delayed_jobs (also addressed by an index, ccng cloudfoundry#3324). Idea was to have an additional absolute limit of failed jobs so that they get deleted even before the 14d failed_jobs.cutoff_age_in_days. The limit should be configurable. change the start_frequent_jobs method to always use all configured parameters except frequency_in_seconds, change expiration_in_seconds from positional to keyword parameter
kathap
force-pushed
the
more-aggressive-cleanup-of-failed-delayed-jobs
branch
from
e6d929b to
68ab9b0
Compare
philippthun
reviewed
Jul 28, 2023
3 tasks
johha
approved these changes
Jul 31, 2023
Merged
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The idea was to have an additional absolute limit of failed jobs so that they get deleted even before the 14d failed_jobs.cutoff_age_in_days. The limit should be configurable.
Therefore we implement a new parameter for maximum number of failed delayed_jobs that should be in the delayed_jobs table. If the number gets exceeded, entries will be deleted keeping the ones with the higher ids. The failed_jobs cleanup job will now run more often than once per day.
Currently, failed delayed_jobs are deleted after 14d (configurable) to keep some info about failed jobs that helps debugging: https://github.com/cloudfoundry/cloud_controller_ng/blob/main/app/jobs/runtime/failed_jobs_cleanup.rb.
This can still lead to very large number of delayed_jobs records that slow down DB queries working on delayed_jobs (also addressed by an index, ccng #3324).
Related to cloudfoundry/capi-release#328
I have reviewed the contributing guide
I have viewed, signed, and submitted the Contributor License Agreement
I have made this pull request to the
mainbranchI have run all the unit tests using
bundle exec rakeI have run CF Acceptance Tests