Penciller work backlog

[martinsumner]

As part of the recent refactoring of the Penciller, the behaviour on clerk prompt is that L0 work (either writing a L0 file because the cache is full, or clearing the L0 file to L1 because a L0 file is present) will always take priority.

What though if PUTs are received very fast? If the cache is full, but by the time the L0 file is written, and the L0 is written to L1 - the cache is already full again.

In this case, we keep writing and merging down L0 files. The L1 will eventually get into a backlog state - this will slow down the process of L0 writing to L1, and make it more likely that the cache will be full by the time of the L0 write will finish.

This has caused a situation during a transfer where there was a huge backlog at L1 of > 700 files.

If there is a backlog, the backlog should be resolved before new L0 files are written.

[martinsumner]

The potential workaround is to increase the backend pause.

[martinsumner]

#394