upgrade_v6_lock : ERR wrong number of arguments for 'hmset' command

[Axxiss]

Describe the bug

Immediately after upgrade from 6.0.20 to 7.0.7 I started seeing the following error ERR wrong number of arguments for 'hmset' command. Unfortunately had to rollback as we have a big amount of old locks.

Expected behavior
Upgrade v6 locks converts v6 locks into v7 format.

Current behavior
Upgrade v6 locks throws ERR wrong number of arguments for 'hmset' command (stacktrace at the bottom)

Worker class

The error is not happening in any specific worker. It comes from the v6 upgrade lock procedure.

Bonus questions

Is there any safe way to remove old locks through rails console/small script? That would allow us to get rid of old stuff so we can move forward with the gem upgrade.

Additional context

• total unique jobs keys scan 'uniquejobs:*' is about 2M
• oldest one I saw while running the v7 was 11 months old (could be that we have even older ones)

Stacktrace

Redis::CommandError: ERR wrong number of arguments for 'hmset' command

1 File "/usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb" line 216 in call_pipelined
2 File "/usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb" line 170 in block in call_pipeline
3 File "/usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb" line 313 in with_reconnect
4 File "/usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb" line 168 in call_pipeline
5 File "/usr/local/bundle/gems/ddtrace-0.46.0/lib/ddtrace/contrib/redis/patcher.rb" line 68 in block in call_pipeline
6 File "/usr/local/bundle/gems/ddtrace-0.46.0/lib/ddtrace/tracer.rb" line 284 in trace
7 File "/usr/local/bundle/gems/ddtrace-0.46.0/lib/ddtrace/contrib/redis/patcher.rb" line 60 in call_pipeline
8 File "/usr/local/bundle/gems/redis-4.2.5/lib/redis.rb" line 2445 in block in pipelined
9 File "/usr/local/bundle/gems/redis-4.2.5/lib/redis.rb" line 69 in block in synchronize
10 File "/usr/local/lib/ruby/2.7.0/monitor.rb" line 202 in synchronize
11 File "/usr/local/lib/ruby/2.7.0/monitor.rb" line 202 in mon_synchronize
12 File "/usr/local/bundle/gems/redis-4.2.5/lib/redis.rb" line 69 in synchronize
13 File "/usr/local/bundle/gems/redis-4.2.5/lib/redis.rb" line 2441 in pipelined
14 File "/usr/local/bundle/gems/sidekiq-unique-jobs-7.0.7/lib/sidekiq_unique_jobs/upgrade_locks.rb" line 90 in upgrade_v6_lock
15 File "/usr/local/bundle/gems/sidekiq-unique-jobs-7.0.7/lib/sidekiq_unique_jobs/upgrade_locks.rb" line 79 in block in upgrade_v6_locks
16 File "/usr/local/bundle/gems/redis-4.2.5/lib/redis.rb" line 2691 in each
17 File "/usr/local/bundle/gems/redis-4.2.5/lib/redis.rb" line 2691 in block in scan_each
18 File "/usr/local/bundle/gems/redis-4.2.5/lib/redis.rb" line 2689 in loop
19 File "/usr/local/bundle/gems/redis-4.2.5/lib/redis.rb" line 2689 in scan_each
20 File "/usr/local/bundle/gems/sidekiq-unique-jobs-7.0.7/lib/sidekiq_unique_jobs/upgrade_locks.rb" line 78 in upgrade_v6_locks
21 File "/usr/local/bundle/gems/sidekiq-unique-jobs-7.0.7/lib/sidekiq_unique_jobs/upgrade_locks.rb" line 59 in block in call
22 File "/usr/local/bundle/gems/sidekiq-unique-jobs-7.0.7/lib/sidekiq_unique_jobs/logging.rb" line 103 in block in with_logging_context
23 File "/usr/local/bundle/gems/sidekiq-5.2.8/lib/sidekiq/logging.rb" line 48 in with_context
24 File "/usr/local/bundle/gems/sidekiq-unique-jobs-7.0.7/lib/sidekiq_unique_jobs/logging.rb" line 118 in call
25 File "/usr/local/bundle/gems/sidekiq-unique-jobs-7.0.7/lib/sidekiq_unique_jobs/logging.rb" line 118 in with_configured_loggers_context
26 File "/usr/local/bundle/gems/sidekiq-unique-jobs-7.0.7/lib/sidekiq_unique_jobs/logging.rb" line 102 in with_logging_context
27 File "/usr/local/bundle/gems/sidekiq-unique-jobs-7.0.7/lib/sidekiq_unique_jobs/upgrade_locks.rb" line 52 in call
28 File "/usr/local/bundle/gems/sidekiq-unique-jobs-7.0.7/lib/sidekiq_unique_jobs/upgrade_locks.rb" line 33 in block in call
29 File "/usr/local/bundle/gems/sidekiq-5.2.8/lib/sidekiq.rb" line 97 in block in redis
30 File "/usr/local/bundle/gems/connection_pool-2.2.3/lib/connection_pool.rb" line 63 in block (2 levels) in with
31 File "/usr/local/bundle/gems/connection_pool-2.2.3/lib/connection_pool.rb" line 62 in handle_interrupt
32 File "/usr/local/bundle/gems/connection_pool-2.2.3/lib/connection_pool.rb" line 62 in block in with
33 File "/usr/local/bundle/gems/connection_pool-2.2.3/lib/connection_pool.rb" line 59 in handle_interrupt
34 File "/usr/local/bundle/gems/connection_pool-2.2.3/lib/connection_pool.rb" line 59 in with
35 File "/usr/local/bundle/gems/sidekiq-5.2.8/lib/sidekiq.rb" line 94 in redis
36 File "/usr/local/bundle/gems/sidekiq-unique-jobs-7.0.7/lib/sidekiq_unique_jobs/connection.rb" line 19 in redis
37 File "/usr/local/bundle/gems/sidekiq-unique-jobs-7.0.7/lib/sidekiq_unique_jobs/upgrade_locks.rb" line 32 in call
38 File "/usr/local/bundle/gems/sidekiq-unique-jobs-7.0.7/lib/sidekiq_unique_jobs/server.rb" line 30 in start
39 File "/usr/local/bundle/gems/sidekiq-unique-jobs-7.0.7/lib/sidekiq_unique_jobs/server.rb" line 20 in block in configure
40 File "/usr/local/bundle/gems/sidekiq-5.2.8/lib/sidekiq/util.rb" line 57 in block in fire_event
41 File "/usr/local/bundle/gems/sidekiq-5.2.8/lib/sidekiq/util.rb" line 55 in each
42 File "/usr/local/bundle/gems/sidekiq-5.2.8/lib/sidekiq/util.rb" line 55 in fire_event
43 File "/usr/local/bundle/gems/sidekiq-5.2.8/lib/sidekiq/cli.rb" line 91 in run
44 File "/usr/local/bundle/gems/sidekiq-5.2.8/bin/sidekiq" line 12 in <top (required)>
45 File "/usr/local/bundle/bin/sidekiq" line 23 in load
46 File "/usr/local/bundle/bin/sidekiq" line 23 in <top (required)>
47 File "/usr/local/bundle/gems/bundler-2.2.9/lib/bundler/cli/exec.rb" line 63 in load
48 File "/usr/local/bundle/gems/bundler-2.2.9/lib/bundler/cli/exec.rb" line 63 in kernel_load
49 File "/usr/local/bundle/gems/bundler-2.2.9/lib/bundler/cli/exec.rb" line 28 in run
50 File "/usr/local/bundle/gems/bundler-2.2.9/lib/bundler/cli.rb" line 494 in exec
51 File "/usr/local/bundle/gems/bundler-2.2.9/lib/bundler/vendor/thor/lib/thor/command.rb" line 27 in run
52 File "/usr/local/bundle/gems/bundler-2.2.9/lib/bundler/vendor/thor/lib/thor/invocation.rb" line 127 in invoke_command
53 File "/usr/local/bundle/gems/bundler-2.2.9/lib/bundler/vendor/thor/lib/thor.rb" line 392 in dispatch
54 File "/usr/local/bundle/gems/bundler-2.2.9/lib/bundler/cli.rb" line 30 in dispatch
55 File "/usr/local/bundle/gems/bundler-2.2.9/lib/bundler/vendor/thor/lib/thor/base.rb" line 485 in start
56 File "/usr/local/bundle/gems/bundler-2.2.9/lib/bundler/cli.rb" line 24 in start
57 File "/usr/local/bundle/gems/bundler-2.2.9/exe/bundle" line 49 in block in <top (required)>
58 File "/usr/local/bundle/gems/bundler-2.2.9/lib/bundler/friendly_errors.rb" line 130 in with_friendly_errors
59 File "/usr/local/bundle/gems/bundler-2.2.9/exe/bundle" line 37 in <top (required)>
60 File "/usr/local/bundle/bin/bundle" line 23 in load
61 File "/usr/local/bundle/bin/bundle" line 23 in <main>

[mhenrixon]

Hi @Axxiss! Sorry about the trouble, the upgrade procedure can be extremely slow in some cases when there are a lot of locks.

I see no reason for this error as I have tests for newer versions of redis-rb. It looks like you might have some locks that don't have the values needed for an upgrade.

What I am about to write is hypothetical so please don't take it with a grain of salt:

I think that you could basically delete all the locks in your system for the sole reason that the locks would be created each time a job is picked up.

What I mean is that no matter which lock type you have, the procedure would lock from where it is. Meaning, if your job is on the queue, awaiting a worker to pick it up; it would create the lock as the server starts processing it. You could potentially run into some weird situations if some lock is pushed or delayed before the worker picks up the job but in that case, an error should be raised and the job retried.

If you have no locks to upgrade, the server should start pretty fast.

The alternative would be to run the upgrader manually before starting the sidekiq server bin/sidekiq -c config/sidekiq.yml but that might not be feasible depending on how you deploy.

Just let me know how you'd prefer to proceed and I can help you with the details.

[Axxiss]

Hi @mhenrixon thanks for the quick reply, much appreciated! 😄

I think that you could basically delete all the locks in your system for the sole reason that the locks would be created each time a job is picked up.

Yeah, something like that is what I have in mind. If it's possible to know the age of a lock, delete the oldest ones. Something like "delete everything older than a week". In case that's not possible, probably deleting all locks and starting from a clean state would be ideal.

This are the locks we are currently using

:until_executing
:until_and_while_executing
:until_executed

[mhenrixon]

@Axxiss, unfortunately, v6 doesn't keep track of anything in regards to timestamps.

The best you can do is delete all locks and start everything up (including the reaper). The reaper should remove locks that end up in a weird state, and eventually, everything should be working fine again.

The :until_and_while_executing was improved and should be less flaky in the future, but those would still be the biggest risk for something going wrong.

I am 9x% certain that apart from a very few potential issues, that hopefully go away on its own. It should be fairly safe.

[Axxiss]

One last thing @mhenrixon I assume that it's safe to delete all keys matching uniquejobs:* or should I use a different pattern?

[mhenrixon]

@Axxiss, that would be the pattern that I'd use and that's the reason for the prefix.

[Axxiss]

After deleting all locks and deploying latest version error is gone. Thank you!