"IndexError: string not matched" when job is replaced on client

[chaimann]

Describe the bug
My use case is fairly simple, I schedule a job upon request to be performed in 15 minutes. If another request received, previous job should be replaced with a new one if the arguments match.
After upgrading from v.7.0.12 to 7.1.5 the error IndexError: string not matched started happening. After some debugging it appeared that normally Sidekiq::Client#process_single returns an item hash, but sometimes it returns a string - job ID, and then fails in Sidekiq::Client#atomic_push:201

to_push = payloads.map do |entry|
  entry['enqueued_at'] = now
  Sidekiq.dump_json(entry)
end

where entry is a String rather than a Hash.

Worker class

class TestJob
  include Sidekiq::Worker
  sidekiq_options queue: :default,
                  retry: 3,
                  lock: :until_executed,
                  on_conflict: { client: :replace, server: :reject },
                  lock_args_method: :lock_args

  def self.lock_args(args)
    args.first
  end
end

Additional context
sidekiq version 5.2.7

To reproduce

[1] pry(main)> TestJob.perform_in(1.minute, 'abc')
=> "71686ac81fa366824b9f738b"
[2] pry(main)> TestJob.perform_in(1.minute, 'abc')
2021-09-03T10:17:12.266Z 58301 TID-2q9x uniquejobs-client DIG-uniquejobs:88b1f6d7ecdb1ead5e068611564de21b INFO: Deleted job: {"class":"TestJob","args":["abc"],"retry":3,"queue":"default","lock":"until_executed","on_conflict":{"client":"replace","server":"reject"},"lock_args_method":"lock_args","jid":"71686ac81fa366824b9f738b","created_at":1630664231.421179,"lock_timeout":0,"lock_ttl":null,"lock_prefix":"uniquejobs","lock_args":"abc","lock_digest":"uniquejobs:88b1f6d7ecdb1ead5e068611564de21b"}
2021-09-03T10:17:12.271Z 58301 TID-2q9x uniquejobs-client DIG-uniquejobs:88b1f6d7ecdb1ead5e068611564de21b INFO: delete_by_digest(uniquejobs:88b1f6d7ecdb1ead5e068611564de21b) completed in 5ms
2021-09-03T10:17:12.271Z 58301 TID-2q9x uniquejobs-client DIG-uniquejobs:88b1f6d7ecdb1ead5e068611564de21b INFO: Deleted `2` keys for uniquejobs:88b1f6d7ecdb1ead5e068611564de21b
IndexError: string not matched
from /Users/chaimann/.rvm/gems/ruby-2.7.4/gems/sidekiq-5.2.7/lib/sidekiq/client.rb:201:in `[]='

[mhenrixon]

I have a sneaky suspicion what this could be but I have haven't had time to investigate. Down with the flu at the moment, having a hard time to focus.

[night91]

I would like to aggregate that with sidekiq 6.2.2. The error is different, but root cause seems the same as @chaimann said.
undefined method key?' for "46b43d4bc6b9896be7bfa228":Stringatlib/sidekiq/client.rb:197 atomic_push`

Using: lock: :until_executed and on_conflict: :replace

This happens when a job is enqueued for future and is enqueued again before the first runs, replacement.

[mhenrixon]

This was fixed in v7.1.7 @night91

[mhenrixon]

Let me know if it ain't fixed and I will reopen the issue n

[chaimann]

Thanks @mhenrixon I'll check asap and will let you know 👍

[chaimann]

@mhenrixon looks like the issue has been resolved, thanks a lot!