Chunk worker cannot process jobs enqueued with a regular worker

[vlymar]

Environment

• Oban Version: 2.17.12
• Oban Pro Version: 1.14.10
• PostgreSQL Version: 17
• Elixir & Erlang/OTP Versions (elixir --version): 1.163-otp25

Current Behavior

1. Insert regular Oban.Worker jobs into a paused queue
2. Deploy new code where the worker has had its type changed to a Oban.Pro.Workers.Chunk worker (but has the same module name).
3. Resume queue
4. Jobs start erroring out with:

** (Protocol.UndefinedError) protocol Enumerable not implemented for nil of type Atom. This protocol is implemented for the following type(s): Ch.Stream, DBConnection.PrepareStream, DBConnection.Stream, Date.Range, Ecto.Adapters.SQL.Stream, File.Stream, Floki.HTMLTree, Function, GenEvent.Stream, [...truncated]

lib/enum.ex in Enumerable.impl_for!/1 at line 1
lib/enum.ex in Enumerable.reduce/3 at line 166
lib/enum.ex in Enum.reduce/3 at line 4396
lib/oban/pro/workers/chunk.ex in Oban.Pro.Workers.Chunk.fetch_last_ts/2 at line 296
lib/oban/pro/workers/chunk.ex in Oban.Pro.Workers.Chunk.maybe_process/2 at line 273
lib/switchboard/users/workers/recipient_deletion_worker.ex in Switchboard.Users.RecipientDeletionWorker.perform/1 at line 11

lib/oban/queue/executor.ex in Oban.Queue.Executor.perform/1 at line 129
lib/oban/queue/executor.ex in Oban.Queue.Executor.call/1 at line 74

I haven't fully figured out why this is happening, but I reproduced it locally and it seems to be caused by the fact that oban_jobs.meta is not getting the chunk_* parameters set in the job records. Manually updating the meta field to populate those params in my dev environment allowed the job to be processed by the Chunk worker.

Expected Behavior

This behavior is somewhat surprising as I expect the options passed to the chunk worker in the use (e.g. size: 100) to determine the runtime behavior of the worker. I'm sure there's a good reason the args are written to the meta field, but it's unfortunate that there's a hard dependency on them being present as it prevents us from easily migrating a workload to a Chunk worker.

Apologies if I'm missing something or off-base with my assumption of what's causing the error.

If i'm correct in understanding the problem, would it be safe to backfill the meta column to enable processing of these jobs via the chunk worker?

Thanks!

[sorentwo]

Apologies if I'm missing something or off-base with my assumption of what's causing the error.

You're diagnoses is correct, and I agree that the behavior is surprising.

If i'm correct in understanding the problem, would it be safe to backfill the meta column to enable processing of these jobs via the chunk worker?

It's entirely safe, and that's the best approach for now. I've pushed a patch that handles your situation gracefully in the future. We'll also be revisiting the chunk worker for Pro v1.6 and smoothing over these little issues.

[vlymar]

awesome, thanks!