Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Monitor worker job runtime #13

Closed
wants to merge 9 commits into from
Closed

Monitor worker job runtime #13

wants to merge 9 commits into from

Conversation

dsalahutdinov
Copy link
Member

Hi, useful metrics to keep track of the current running job runtime (see the "busy" section in the sidekiq admin panel).
This is monstly for alerting to determine time-consuming jobs with the following alert rule:

For example publication* job running more than 10 minutes:

max(sidekiq_worker_runtime{queue=~"publication.*"}) by (queue) > 600

Envek
Envek requested changes Jul 17, 2020
View reviewed changes
Copy link
Member

@Envek Envek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! This is good addition to Sidekiq metrics! However, I found some problems with your implementation. Let's discuss.

lib/yabeda/sidekiq.rb Outdated Show resolved Hide resolved
lib/yabeda/sidekiq.rb Outdated Show resolved Hide resolved
lib/yabeda/sidekiq.rb Outdated Show resolved Hide resolved
lib/yabeda/sidekiq.rb Outdated Show resolved Hide resolved
Envek
Envek requested changes Nov 6, 2020
View reviewed changes
Copy link
Member

@Envek Envek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rubocop and some error-generating typos should be fixed

lib/yabeda/sidekiq.rb Outdated Show resolved Hide resolved
lib/yabeda/sidekiq.rb Outdated Show resolved Hide resolved
lib/yabeda/sidekiq.rb Outdated Show resolved Hide resolved
lib/yabeda/sidekiq.rb Outdated Show resolved Hide resolved
lib/yabeda/sidekiq.rb Outdated Show resolved Hide resolved
lib/yabeda/sidekiq.rb Outdated Show resolved Hide resolved
@Envek
Fix typos and rubocop offences
1f9a535
@Envek
Merge branch 'master' into feat/worker_runtime
a4c7624 
* master:
  Migrate from Travis CI to GitHub Actions for tests and releases
  0.7.0: Apply tags from `yabeda_tags` to all metrics inside a job
  Switch from current time to monotonic time for measuring job runtime
  Add basic specs
  Add Yabeda configuration instruction into README. Fixes yabeda-rb#10
@Envek
Cleanup previous observations
9632341
@Envek
Simplify cleanup logic
ebbdea1
@Envek
Fix specs when Redis isn't available
cf9c2dc
@Envek
Fix rubocop
c906a27
@dsalahutdinov
Copy link
Member Author

@Envek, thanks for getting it done! Let's merge and roll it out to production 🚀

@Envek Envek mentioned this pull request Apr 19, 2021
Merged
@Envek
Copy link
Member

Envek commented Apr 21, 2021

There is also alternative implementation in #17. @dsalahutdinov please take a look.

@Envek Envek closed this in 91c8f4e Apr 28, 2021
@Envek
Copy link
Member

Envek commented Apr 28, 2021

Closed in favor of #17

@Envek Envek mentioned this pull request Apr 30, 2021
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Envek Envek Envek requested changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dsalahutdinov @Envek