Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sidekiq Pro, Redis::CommandError: ERR wrong number of arguments for 'brpop' command, when pausing queue #4825

Comments

@thibaudgg
Copy link
Contributor

thibaudgg commented Feb 23, 2021

Ruby version: 2.6.6
Rails version: 6.0.3.5
Sidekiq / Pro / Enterprise version(s): 6.1.2 / 5.2.0 / 2.1.2

Hi,

We are getting this Redis::CommandError: ERR wrong number of arguments for 'brpop' command error, when pausing a queue from the Web UI queues list.

This issue is easily reproducible and is only happening when the paused queue is the only one processed by the sidekiq command, i.e. sidekiq-q default. When two queues are processed (i.e.sidekiq -q default -q low) this error isn't triggered unless the two queues are paused together. To resume, this is triggered when the last active queue is paused.

We found that this issue is coming from the basic_fetch strategy overwritten in Sidekiq Pro as we aren't using the super_fetch strategy.
Sidekiq Pro overwrittes the Sidekiq::Pro::BasicFetch::queues_cmd method and left the @queues variable empty when only one queue is processed (see queues = (@original - @paused.to_a).map { |q| "queue:#{q}" }), which then triggers this error from here: https://github.com/mperham/sidekiq/blob/3b5ae30c4e5e9e760268243ab5c14664a2f8d236/lib/sidekiq/fetch.rb#L39

Backtrace:

Redis::CommandError: ERR wrong number of arguments for 'brpop' command
 [GEM_ROOT]/gems/redis-4.2.2/lib/redis/client.rb:127 :in `call`
 [GEM_ROOT]/gems/redis-4.2.2/lib/redis/client.rb:221 :in `block in call_with_timeout`
 [GEM_ROOT]/gems/redis-4.2.2/lib/redis/client.rb:295 :in `with_socket_timeout`
 [GEM_ROOT]/gems/redis-4.2.2/lib/redis/client.rb:220 :in `call_with_timeout`
218	
219	    def call_with_timeout(command, timeout, &blk)
220	      with_socket_timeout(timeout) do
221	        call(command, &blk)
222	      end
 [GEM_ROOT]/gems/redis-4.2.2/lib/redis.rb:1224 :in `block in _bpop`
 [GEM_ROOT]/gems/redis-4.2.2/lib/redis.rb:69 :in `block in synchronize`
[PROJECT_ROOT]/vendor/ruby-2.6.6/lib/ruby/2.6.0/monitor.rb:235 :in `mon_synchronize`
 [GEM_ROOT]/gems/redis-4.2.2/lib/redis.rb:69 :in `synchronize`
 [GEM_ROOT]/gems/redis-4.2.2/lib/redis.rb:1221 :in `_bpop`
 [GEM_ROOT]/gems/redis-4.2.2/lib/redis.rb:1266 :in `brpop`
 [GEM_ROOT]/gems/sidekiq-6.1.2/lib/sidekiq/fetch.rb:39 :in `block in retrieve_work`
 [GEM_ROOT]/gems/sidekiq-6.1.2/lib/sidekiq.rb:98 :in `block in redis`
 [GEM_ROOT]/gems/connection_pool-2.2.3/lib/connection_pool.rb:63 :in `block (2 levels) in with`
 [GEM_ROOT]/gems/connection_pool-2.2.3/lib/connection_pool.rb:62 :in `handle_interrupt`
 [GEM_ROOT]/gems/connection_pool-2.2.3/lib/connection_pool.rb:62 :in `block in with`
 [GEM_ROOT]/gems/connection_pool-2.2.3/lib/connection_pool.rb:59 :in `handle_interrupt`
 [GEM_ROOT]/gems/connection_pool-2.2.3/lib/connection_pool.rb:59 :in `with`
 [GEM_ROOT]/gems/sidekiq-6.1.2/lib/sidekiq.rb:95 :in `redis`
 [GEM_ROOT]/gems/sidekiq-6.1.2/lib/sidekiq/fetch.rb:39 :in `retrieve_work`
 [GEM_ROOT]/gems/sidekiq-6.1.2/lib/sidekiq/processor.rb:83 :in `get_one`
 [GEM_ROOT]/gems/sidekiq-6.1.2/lib/sidekiq/processor.rb:95 :in `fetch`
 [GEM_ROOT]/gems/sidekiq-6.1.2/lib/sidekiq/processor.rb:77 :in `process_one`
 [GEM_ROOT]/gems/sidekiq-6.1.2/lib/sidekiq/processor.rb:68 :in `run`
 [GEM_ROOT]/gems/sidekiq-6.1.2/lib/sidekiq/util.rb:15 :in `watchdog`
 [GEM_ROOT]/gems/sidekiq-6.1.2/lib/sidekiq/util.rb:24 :in `block in safe_thread`
@thibaudgg
Copy link
Contributor Author

Thanks for the quick fix @mperham! 👍

This was referenced Mar 15, 2021
This was referenced Mar 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment