Decentralise duplicate detection

[michaelweiser]

Previously we detected duplicate samples upon job submission. This was a
very intricate process that covered two stages of detection: local
duplicates and other Peekaboo instances in a cluster analysing the same
sample concurrently. Apart from being hard to understand and maintain
this was inefficient for analyses which didn't involve any expensive
operations such as offloading a job to Cuckoo. This degraded into a
downright throughput bottleneck for analyses of large numbers (> 10000)
of nonidentical samples which are eventually ignored.

This change moves duplicate handling out of the queueing into a new
duplicate toolbox module. Duplicate detection is moved into individual
rules. Resubmission of withheld samples is done in the worker at the end
of ruleset processing after the processing result is saved to the
database.

Handling of local and cluster duplicates is stricly separated. While
that makes the actual code not much easier to understand and maintain,
the underlying concepts at least are somewhat untangled.

The cluster duplicate handler stays mostly the same, primarily
consisting of a coroutine which periodically tries to lock samples from
its backlog and then submit it to the local queue.

The local duplicate handler is now a distinct module very similar to the
cluster duplicate handler but doesn't need any repeated polling. Instead
potential duplicates are still resubmitted once a sample finishes
processing.

The cluster duplicate handler no longer directly interacts with the
local duplicate handler by putting samples from its backlog into their
backlog. Instead cluster duplicates are submitted to the local queue in
bulk and the duplicate handler is expected to either never come into
play again (because of the known rule and its cached previous analysis
result) or detect the local duplicates and put all but one of them into
its own backlog automatically.

This new design highlighted an additional point for optimisation: If a
sample can be locked by the cluster duplicate handler (i.e. is not
currently being processed by another instance) but we find siblings of
it in our own cluster duplicate backlog, then obviously this sample was
at an earlier point in time a cluster duplicate and withheld samples are
waiting for the next polling run to be resubmitted. In this case we
short-circuit the process from the cluster duplicate detection and
submit them to the job queue immediately.

This depends on #219 to get back to an uncached Known rule.
Therefore the first three commits here are exactly the same.

[michaelweiser]

With #219 merged this should be good to go as well.

[michaelweiser]

BTW: It appears the contaner CI pipeline error would be fixed by scVENUS/PeekabooAV-Installer#79 as alpine edge/testing (3.18.something) has apparently moved too far away from 3.15 so dependencies no longer match.

[michaelweiser]

Merging unreviewed on my own recognisance.