Run 1 job and queue 1

[jeremybdk]

Hello,

I wondered if it's possible to setup your gem so that I can run one job and when it's running queue the next same exact job while discarding any other job when one is already in queue.

thanks a lot for your work.

[mhenrixon]

That is one idea I had with "UntilExecuted". I'll investigate that.

[jeremybdk]

With UntilExecuted I cannot add a job a queue while the same one is being processed.

[mhenrixon]

I know :) I just said it was an idea I had with the lock.

[jeremybdk]

Ok I am sorry, I am a little lost.

Right now, we only figured out that until_executed which seems to be the same behavior as v3: a job is unique, whether it's in queue or executing.

With until_executing, if add 10 jobs with the same arguments, 3 jobs (my concurrency limit in sidekiq.yml) run at the same time, the 7 others are added to the queue.

With while_executing, all jobs (3jobs with the same arguments) are started according to Sidekiq but they are not run at the same time. The issue here is that If I queue 4 jobs, ( 3 with one id, and the 4th one with another id) the 4th job will start when the first one is done.
It seems that jobs are not kept in queue but start even if they are the same.

Another question about Redis, you say Redis 2.6.2 is required but is it Redis >= 2.6.2 or only Redis 2.6.2 ?

Thanks,

[mhenrixon]

Great questions; thanks for taking the time! I'll update the documentation during the day to address your questions.

[mhenrixon]

Redis >= 2.6.2 thanks for pointing that out.

[jeremybdk]

Just to be sure, did we understand everything correctly ?

[mhenrixon]

@jaybbb maybe; it looks right but to be completely sure.... Could you provide some code examples for your situation? It looks like you got it right I just want to make sure that I did understand it correctly.

[jeremybdk]

class UpdateLastUpdatesForUserWorker
  include Sidekiq::Worker
  sidekiq_options queue: :visibility, unique: true, unique_lock: :until_executing

   def perform(store_id, user_id)
     sleep 7
   end
end

This is the code we used to test the concurrency and the uniqueness of a job.

• until_executed jobs are unique, whether the job is in queue or processing, if I try to add a new job with the same arguments it returns nil instead of the job id.
• until_executing, job are not unique, they all run at the same time. We might have missed something but it does not seem to be working as intended.
• while_executing, jobs are not unique, they all start at the same time, but are locked during processing so that they are not running at the same time even if they are not in queue anymore.

[jeremybdk]

Thanks a lot, it works as expected. Just to understand, when I add the first job, it goes to queue then processing. When I add the first job it goes to queue then processing but wait for the first one to be done.

The other jobs are dropped until the first one is done.

Would it be possible to keep the second job in queue until the first one is done ?

Anyway, thanks a lot for this amazing support !