New queue fails in edge case max_batch_size=max_entries with assertion error

[JensErat]

Is there an existing issue for this?

• I have searched the existing issues

Kong version ($ kong version)

Kong 3.3.1

Current Behavior

When defining a very special edge case configuration having max_batch_size=max_entries, the queue can fail with an assertion error when removing the frontmost element. This happens especially when the callback repeatedly fails (eg. an unavailable backend system receiving data).

What happens:

1. we add max_batch_size elements, all of which "post" resources

2. the batch queue consumes all of those resources in process_once by wait()ing for them, but gets stuck processing/sending the batch

3. as process_once is stuck until max_retry_time passed, the function does not run delete_frontmost_entry() and thus actually moves the front reference

4. when enqueuing the next item, it tries to drop the oldest entry, but triggers the assertion in queue.lua as no resources are left:

    /kong-plugin/kong/tools/queue.lua:215: assertion failed!
   

Potential thoughts for fixing the issue:

• enforce max_batch_size<max_entries, which is not really a reasonable configuration anyway
• increase max_entries by 1 when both configuration values are the same (our current workaround)
• further refactor the queue to remove the race condition

Kudos to @27ascii for discovering the edge case configuration.

Expected Behavior

Configuration either not allowed or assertion does not fail worker

Steps To Reproduce

Pull request containing unit test reproducing the issue: #11378

Anything else?

No response

^{Jens Erat <jens.erat@mercedes-benz.com>, Mercedes-Benz Tech Innovation GmbH, imprint}