TaskQueue appears to be spinning on same task

[johnaohara]

Describe the bug

Log is full of messages TaskQueue messages, spanning multiple hours;

...
2023-01-16 16:29:41,565 DEBUG [io.hyp.too.hor.bus.TaskQueue] (executor-thread-7567) This thread is going to execute tasks (1) for test 48778, lock level 1
2023-01-16 16:29:41,566 DEBUG [io.hyp.too.hor.bus.TaskQueue] (executor-thread-7567) Finished executing tasks for test 48778
2023-01-16 16:29:42,568 DEBUG [io.hyp.too.hor.bus.TaskQueue] (executor-thread-7567) This thread is going to execute tasks (1) for test 48778, lock level 1
...
2023-01-16 17:43:07,493 DEBUG [io.hyp.too.hor.bus.TaskQueue] (executor-thread-7646) This thread is going to execute tasks (1) for test 48778, lock level 1
2023-01-16 17:43:07,494 DEBUG [io.hyp.too.hor.bus.TaskQueue] (executor-thread-7646) Finished executing tasks for test 48778
2023-01-16 17:43:08,495 DEBUG [io.hyp.too.hor.bus.TaskQueue] (executor-thread-7646) This thread is going to execute tasks (1) for test 48778, lock level 1
2023-01-16 17:43:08,497 DEBUG [io.hyp.too.hor.bus.TaskQueue] (executor-thread-7646) Finished executing tasks for test 48778
2023-01-16 17:43:09,498 DEBUG [io.hyp.too.hor.bus.TaskQueue] (executor-thread-7646) This thread is going to execute tasks (1) for test 48778, lock level 1
2023-01-16 17:43:09,499 DEBUG [io.hyp.too.hor.bus.TaskQueue] (executor-thread-7646) Finished executing tasks for test 48778
...

I would not expect to see this behavior.

[johnaohara]

This behavior has been ongoing for a number of days (at least 4)

[johnaohara]

This occurs when a label update events are processed and the dataset is not found in the data. This can occur in 2 situations, when a dataSet has just been created and is not yet visible in the AlertingServiceImpl.onLabelsUpdated() transaction. It can also occur if a user clicks "Re-transform datasets" multiple times in the Run view and creates a race condition.

The behaviour at the moment is to retry updating dataset labels every 1000ms. For a new data upload, this will eventually succeed, for the retransform case, the dataset has been deleted, so horreum get stuck in a loop, expecting a dataset that has already been deleted

[stalep]

With regards to the two situations you describe. For the first situation, I would assume that would not cause it to go in a never ending loop or?

[johnaohara]

@stalep yes, the loop is probably there to capture the first scenario. Wrt 2nd scenario, we should fix the race cond. but it does not hurt to limit the number of retries and log a warning if an endless loop is detected