chore: Add specific worker backend notes and table with collected met…

…rics for each backend (#228) We needed a place to document a gotcha recently discovered with rescue-scheduler that wasn't loading our latency extension, so I added another sub-section under "Worker adapters" for that. This also takes the opportunity to document what each adapter collects specifically, so it's easier to visualize where queue latency vs queue depth is available, and which can support busy (if enabled).
judoscale · Nov 5, 2024 · 8c13482 · 8c13482
1 parent 6fd131c
commit 8c13482
Show file tree

Hide file tree

Showing 11 changed files with 297 additions and 266 deletions.
diff --git a/README.md b/README.md
@@ -72,6 +72,12 @@ For most apps, no additional configuration is needed. See the [configuration](#c
 
 Note that if you aren't using Rails, you'll need to start the reporter manually. See below.
 
+### Specific worker backend notes
+
+#### Resque
+
+If you're using `resque-scheduler` and their [standalone executable](https://github.com/resque/resque-scheduler?tab=readme-ov-file#standalone-executable) approach, add a `require "judoscale-resque"` to your executable, or require your entire Rails application. This ensures the Judoscale extension that stores latency for each job gets properly loaded within the scheduler process, otherwise metrics may not be reported appropriately from the scheduler.
+
 ## Worker-only apps
 
 If your app doesn't have a web process, you don't _have_ to include the "judoscale-rails" gem. If you omit it, you'll need to start the reporter manually:
@@ -93,10 +99,23 @@ The reporter runs in its own thread so your web requests and background jobs are
 - Judoscale gem versions
 - Dyno name (example: web.1)
 - PID
-- Collection of queue time metrics (time and milliseconds)
+- Collection of queue time metrics (time and milliseconds) for web
+- Collection of queue time and/or queue depth metrics, and busy metrics (if enabled), for workers (see below)
 
 Judoscale aggregates and stores this information to power the autoscaler algorithm and dashboard visualizations.
 
+### What data is collected for each worker adapter?
+
+| adapter               | queue time | queue depth | busy (if enabled) |
+|-----------------------|------------|-------------|-------------------|
+| judoscale-sidekiq     | ✅         | ✅          | ✅               |
+| judoscale-solid_queue | ✅         | ❌          | ✅               |
+| judoscale-resque      | ✅         | ✅          | ✅               |
+| judoscale-delayed_job | ✅         | ❌          | ✅               |
+| judoscale-good_job    | ✅         | ❌          | ✅               |
+| judoscale-que         | ✅         | ❌          | ✅               |
+| judoscale-shoryuken   | ❌         | ✅          | ❌               |
+
 ## Migrating from `rails_autoscale_agent`
 
 The migration from `rails_autoscale_agent` to `judoscale-rails` (+ your job framework gem) is typically a single step: replace the `gem "rails_autoscale_agent"` in your Gemfile with `gem "judoscale-rails"` _and_ the appropriate `judoscale-*` package for your back-end job framework (`sidekiq`, `resque`, `delayed_job`, `good_job`, or `que`) or see the [Installation](#installation) section above for further specifics. If you previously had any custom configuration for the `rails_autoscale_agent`, please note that we now use a `configure` block as shown below.

diff --git a/sample-apps/delayed_job-sample/Gemfile.lock b/sample-apps/delayed_job-sample/Gemfile.lock
@@ -1,44 +1,44 @@
 PATH
   remote: ../../judoscale-delayed_job
   specs:
-    judoscale-delayed_job (1.8.1)
+    judoscale-delayed_job (1.8.2)
       delayed_job_active_record (>= 4.0)
-      judoscale-ruby (= 1.8.1)
+      judoscale-ruby (= 1.8.2)
 
 PATH
   remote: ../../judoscale-rails
   specs:
-    judoscale-rails (1.8.1)
-      judoscale-ruby (= 1.8.1)
+    judoscale-rails (1.8.2)
+      judoscale-ruby (= 1.8.2)
       railties
 
 PATH
   remote: ../../judoscale-ruby
   specs:
-    judoscale-ruby (1.8.1)
+    judoscale-ruby (1.8.2)
 
 GEM
   remote: https://rubygems.org/
   specs:
-    actionpack (7.0.8.5)
-      actionview (= 7.0.8.5)
-      activesupport (= 7.0.8.5)
+    actionpack (7.0.8.6)
+      actionview (= 7.0.8.6)
+      activesupport (= 7.0.8.6)
       rack (~> 2.0, >= 2.2.4)
       rack-test (>= 0.6.3)
       rails-dom-testing (~> 2.0)
       rails-html-sanitizer (~> 1.0, >= 1.2.0)
-    actionview (7.0.8.5)
-      activesupport (= 7.0.8.5)
+    actionview (7.0.8.6)
+      activesupport (= 7.0.8.6)
       builder (~> 3.1)
       erubi (~> 1.4)
       rails-dom-testing (~> 2.0)
       rails-html-sanitizer (~> 1.1, >= 1.2.0)
-    activemodel (7.0.8.5)
-      activesupport (= 7.0.8.5)
-    activerecord (7.0.8.5)
-      activemodel (= 7.0.8.5)
-      activesupport (= 7.0.8.5)
-    activesupport (7.0.8.5)
+    activemodel (7.0.8.6)
+      activesupport (= 7.0.8.6)
+    activerecord (7.0.8.6)
+      activemodel (= 7.0.8.6)
+      activesupport (= 7.0.8.6)
+    activesupport (7.0.8.6)
       concurrent-ruby (~> 1.0, >= 1.0.2)
       i18n (>= 1.6, < 2)
       minitest (>= 5.1)
@@ -60,21 +60,21 @@ GEM
     erubi (1.13.0)
     i18n (1.14.6)
       concurrent-ruby (~> 1.0)
-    loofah (2.22.0)
+    loofah (2.23.1)
       crass (~> 1.0.2)
       nokogiri (>= 1.12.0)
     method_source (1.1.0)
     minitest (5.25.1)
     mustermann (3.0.3)
       ruby2_keywords (~> 0.0.1)
-    nio4r (2.7.3)
+    nio4r (2.7.4)
     nokogiri (1.16.7-arm64-darwin)
       racc (~> 1.4)
     nokogiri (1.16.7-x86_64-darwin)
       racc (~> 1.4)
     nokogiri (1.16.7-x86_64-linux)
       racc (~> 1.4)
-    pg (1.5.8)
+    pg (1.5.9)
     puma (5.6.9)
       nio4r (~> 2.0)
     racc (1.8.1)
@@ -91,9 +91,9 @@ GEM
     rails-html-sanitizer (1.6.0)
       loofah (~> 2.21)
       nokogiri (~> 1.14)
-    railties (7.0.8.5)
-      actionpack (= 7.0.8.5)
-      activesupport (= 7.0.8.5)
+    railties (7.0.8.6)
+      actionpack (= 7.0.8.6)
+      activesupport (= 7.0.8.6)
       method_source
       rake (>= 12.2)
       thor (~> 1.0)
@@ -109,13 +109,14 @@ GEM
     tilt (2.4.0)
     tzinfo (2.0.6)
       concurrent-ruby (~> 1.0)
-    zeitwerk (2.7.0)
+    zeitwerk (2.7.1)
 
 PLATFORMS
   arm64-darwin-20
   arm64-darwin-21
   arm64-darwin-22
   arm64-darwin-23
+  arm64-darwin-24
   x86_64-darwin-21
   x86_64-linux
 

diff --git a/sample-apps/good_job-multi-db-sample/Gemfile.lock b/sample-apps/good_job-multi-db-sample/Gemfile.lock
@@ -1,98 +1,98 @@
 PATH
   remote: ../../judoscale-good_job
   specs:
-    judoscale-good_job (1.8.1)
+    judoscale-good_job (1.8.2)
       good_job (>= 3.0, < 5.0)
-      judoscale-ruby (= 1.8.1)
+      judoscale-ruby (= 1.8.2)
 
 PATH
   remote: ../../judoscale-rails
   specs:
-    judoscale-rails (1.8.1)
-      judoscale-ruby (= 1.8.1)
+    judoscale-rails (1.8.2)
+      judoscale-ruby (= 1.8.2)
       railties
 
 PATH
   remote: ../../judoscale-ruby
   specs:
-    judoscale-ruby (1.8.1)
+    judoscale-ruby (1.8.2)
 
 GEM
   remote: https://rubygems.org/
   specs:
-    actionpack (7.0.8.4)
-      actionview (= 7.0.8.4)
-      activesupport (= 7.0.8.4)
+    actionpack (7.0.8.6)
+      actionview (= 7.0.8.6)
+      activesupport (= 7.0.8.6)
       rack (~> 2.0, >= 2.2.4)
       rack-test (>= 0.6.3)
       rails-dom-testing (~> 2.0)
       rails-html-sanitizer (~> 1.0, >= 1.2.0)
-    actionview (7.0.8.4)
-      activesupport (= 7.0.8.4)
+    actionview (7.0.8.6)
+      activesupport (= 7.0.8.6)
       builder (~> 3.1)
       erubi (~> 1.4)
       rails-dom-testing (~> 2.0)
       rails-html-sanitizer (~> 1.1, >= 1.2.0)
-    activejob (7.0.8.4)
-      activesupport (= 7.0.8.4)
+    activejob (7.0.8.6)
+      activesupport (= 7.0.8.6)
       globalid (>= 0.3.6)
-    activemodel (7.0.8.4)
-      activesupport (= 7.0.8.4)
-    activerecord (7.0.8.4)
-      activemodel (= 7.0.8.4)
-      activesupport (= 7.0.8.4)
-    activesupport (7.0.8.4)
+    activemodel (7.0.8.6)
+      activesupport (= 7.0.8.6)
+    activerecord (7.0.8.6)
+      activemodel (= 7.0.8.6)
+      activesupport (= 7.0.8.6)
+    activesupport (7.0.8.6)
       concurrent-ruby (~> 1.0, >= 1.0.2)
       i18n (>= 1.6, < 2)
       minitest (>= 5.1)
       tzinfo (~> 2.0)
     builder (3.3.0)
-    concurrent-ruby (1.3.3)
+    concurrent-ruby (1.3.4)
     crass (1.0.6)
     debug (1.9.2)
       irb (~> 1.10)
       reline (>= 0.3.8)
     erubi (1.13.0)
     et-orbi (1.2.11)
       tzinfo
-    fugit (1.11.0)
+    fugit (1.11.1)
       et-orbi (~> 1, >= 1.2.11)
       raabro (~> 1.4)
     globalid (1.2.1)
       activesupport (>= 6.1)
-    good_job (3.99.0)
+    good_job (3.99.1)
       activejob (>= 6.0.0)
       activerecord (>= 6.0.0)
       concurrent-ruby (>= 1.0.2)
       fugit (>= 1.1)
       railties (>= 6.0.0)
       thor (>= 0.14.1)
-    i18n (1.14.5)
+    i18n (1.14.6)
       concurrent-ruby (~> 1.0)
     io-console (0.7.2)
-    irb (1.12.0)
-      rdoc
+    irb (1.14.1)
+      rdoc (>= 4.0.0)
       reline (>= 0.4.2)
-    loofah (2.22.0)
+    loofah (2.23.1)
       crass (~> 1.0.2)
       nokogiri (>= 1.12.0)
     method_source (1.1.0)
-    minitest (5.24.1)
-    nio4r (2.7.1)
-    nokogiri (1.16.6-arm64-darwin)
+    minitest (5.25.1)
+    nio4r (2.7.4)
+    nokogiri (1.16.7-arm64-darwin)
       racc (~> 1.4)
-    nokogiri (1.16.6-x86_64-darwin)
+    nokogiri (1.16.7-x86_64-darwin)
       racc (~> 1.4)
-    nokogiri (1.16.6-x86_64-linux)
+    nokogiri (1.16.7-x86_64-linux)
       racc (~> 1.4)
-    pg (1.5.6)
+    pg (1.5.9)
     psych (5.1.2)
       stringio
-    puma (5.6.8)
+    puma (5.6.9)
       nio4r (~> 2.0)
     raabro (1.4.0)
-    racc (1.8.0)
-    rack (2.2.9)
+    racc (1.8.1)
+    rack (2.2.10)
     rack-test (2.1.0)
       rack (>= 1.3)
     rails-dom-testing (2.2.0)
@@ -102,29 +102,30 @@ GEM
     rails-html-sanitizer (1.6.0)
       loofah (~> 2.21)
       nokogiri (~> 1.14)
-    railties (7.0.8.4)
-      actionpack (= 7.0.8.4)
-      activesupport (= 7.0.8.4)
+    railties (7.0.8.6)
+      actionpack (= 7.0.8.6)
+      activesupport (= 7.0.8.6)
       method_source
       rake (>= 12.2)
       thor (~> 1.0)
       zeitwerk (~> 2.5)
     rake (13.2.1)
-    rdoc (6.6.3.1)
+    rdoc (6.7.0)
       psych (>= 4.0.0)
-    reline (0.5.4)
+    reline (0.5.10)
       io-console (~> 0.5)
-    stringio (3.1.0)
-    thor (1.3.1)
+    stringio (3.1.1)
+    thor (1.3.2)
     tzinfo (2.0.6)
       concurrent-ruby (~> 1.0)
-    zeitwerk (2.6.16)
+    zeitwerk (2.7.1)
 
 PLATFORMS
   arm64-darwin-20
   arm64-darwin-21
   arm64-darwin-22
   arm64-darwin-23
+  arm64-darwin-24
   x86_64-darwin-21
   x86_64-linux
 
@@ -144,4 +145,4 @@ DEPENDENCIES
   railties (~> 7.0.1)
 
 BUNDLED WITH
-   2.5.11
+   2.5.22