Dynamic bucket allocation & miscellaneous volume improvements

[philippotto]

• increase speed of compression for volume buckets by using two workers instead of one for that job (adds a new WorkerPool abstraction for that)
• adds a tooltip (only visible on hover) to the save button in case there are currently buckets waiting for compression (when annotating large amounts the compression can take some seconds which might irritate the user since the progress was not really visible)
• adapts the bucket allocation logic so that there is no hard limit anymore. instead, there is an initial limit which can be exceeded if necessary. if the limit is exceeded by a factor of 2 (this corresponds to the hotfix of bumping the limit to 10000 for segmentation layers, the user is warned about that and advised to save&reload). I didn't built a shrinking logic, since I don't consider the advantages to be worth it. Happy to discuss this, though!
• fixes that PushQueue.push could starve when the user never does a break longer than 1 second (unrealistic, but when testing, one can easily run into this)
• refactor: rename some important variables (such as queue) and add some comments

URL of deployed dev instance (used for testing):

• https://dynamicbucketallocation.webknossos.xyz

Steps to test:

• I adapted the code to output some periodic logging about the bucket counts and use a lower maximum bucket limit, so that the critical cases can be tested easier
• open a new annotation and open the devtools
• brush large areas over multiple viewports to quickly build up a load of changed buckets
• shortly before it reaches 500, stop brushing and hit save
• hovering over the save-button should show some information about how many buckets need to be compressed
• when everything is saved, brush again (in moderate speed)
• ideally, the allocated bucket limit should not exceed 500
• now, brush very fast across all viewports and jump into new regions to force wk to allocate new buckets
• --> the bucket count should grow over 500
• continue so that the bucket count grows over 1000 --> a toast should appear
• save & wait
• reload the page --> everything should look as before

Issues:

• contributes to Refactor/harden volume annotation architecture/code #5973

---

(Please delete unneeded items, merge only when none are left open)

• Updated (unreleased) changelog
• Updated (unreleased) migration guide if applicable
• Updated documentation if applicable
• Adapted wk-connect if datastore API changes
• Adapted wk-libs python client if relevant API parts change
• Needs datastore update after deployment
• Ready for review

[daniel-wer]

Great changes, code LGTM 👍

During testing I encountered some issues. I'm not sure whether they are related to the temporary changes to ease testing, but I would think not. Maybe buckets get collected, although they shouldn't be?

• Something seems to be fishy with the data loading. When in mag 1 some buckets are not loaded/shown in mag 1, see for example the right side of the XY viewport here (loading had finished):
  

• The loading pattern seems spotty and slower than before.
  Before:
  

  With this PR:
  

• Also, I wasn't able to get a tooltip for the save button yet. I tried hovering before clicking the save button or afterwards. Is this very timing-dependent?

[daniel-wer]

Suggested change

	if (this.buckets.length > (2 * constants.MAXIMUM_BUCKET_COUNT_PER_LAYER) / 10) {
	if (this.buckets.length > 2 * this.BUCKET_COUNT_SOFT_LIMIT) {

[philippotto]

Thanks for testing and raising these oddities 👍 I guess, I got a little bit betriebsblind and didn't notice this myself.

• Something seems to be fishy with the data loading. When in mag 1 some buckets are not loaded/shown in mag 1, see for example the right side of the XY viewport here (loading had finished):

I finally found the issue for this and fixed it in 5f2f7bf. The code which marked a bucket as active happened to late which allowed the bucket collector to collect a bucket while allocating another one in consumeBucketsFromArrayBuffer. Through the artificial low bucket limit, this issue became present + frequent, but I assume this fixes also the rare disappearing buckets I saw from time to time in prod 🎉

• The loading pattern seems spotty and slower than before.

The spottiness should be solved with the above bug fix, too. The slowness is probably a consequence of the low bucket limit which makes GC more frequent (and this slows wk down). Due to the fix from above, wk recognizes that it's necessary to exceed the bucket limit, anyway (otherwise, not everything would be rendered). Therefore, garbage-collections and repeated re-downloads are now avoided.

Could you please test again to see whether both issues are resolved now? (for me it looks good, but I didn't really notice a big speed difference before, either). Afterwards, I'll lift the artificial limits and we can do another testing round (however, I'm assuming that to be way more robust). I just deployed the newest version: https://dynamicbucketallocation.webknossos.xyz/

• Also, I wasn't able to get a tooltip for the save button yet. I tried hovering before clicking the save button or afterwards. Is this very timing-dependent?

Yes, kind of. One needs a really big brush size (i.e., 300) and draw quickly (2 seconds should be enough). Then, hovering over the save button will briefly show the tooltip until all buckets are compressed. Depending on how many buckets were labeled, the tooltip might show progress for several seconds.

[daniel-wer]

I finally found the issue for this and fixed it in 5f2f7bf. The code which marked a bucket as active happened to late which allowed the bucket collector to collect a bucket while allocating another one in consumeBucketsFromArrayBuffer. Through the artificial low bucket limit, this issue became present + frequent, but I assume this fixes also the rare disappearing buckets I saw from time to time in prod tada

Ah nice find! Good thing you introduced these strict limits for testing which made the issue much more apparent 👍

Could you please test again to see whether both issues are resolved now? (for me it looks good, but I didn't really notice a big speed difference before, either).

Both issues are solved for me and also was able to get a quick glance at the save button tooltip 👍

[daniel-wer]

One thing that happened during testing when I was brushing with a large brush size was that the sagas crashed with "Maximum call stack size exceeded". Not sure if that's known. I don't think it has high priority since I brushed unrealistically fast using the maximum brush size. Still I wanted to mention it, stack trace attached.

call_stack_exceeded_stack_trace.txt

[philippotto]

Thanks for testing! I now cleaned up the debugging code and think this PR is ready to merge if you agree :)

One thing that happened during testing when I was brushing with a large brush size was that the sagas crashed with "Maximum call stack size exceeded". Not sure if that's known. I don't think it has high priority since I brushed unrealistically fast using the maximum brush size. Still I wanted to mention it, stack trace attached.

call_stack_exceeded_stack_trace.txt

Yes, I also noticed this error from time to time (only in unrealistic annotation scenarios and already before this PR) and also spent some time debugging it some time ago (unfortunately, without much success). Interestingly, your stack trace looks way bigger than mine (I "only" had 200 function calls on the stack) whereas you have more than 10000. This gave me the idea that redux-saga might do some sort of tail recursion which leads to this problem. I think I've got a fix for this, but I'd like to do this in a separate PR to unblock the current one.

/edit: I opened #6074 for the above issue.

[daniel-wer]

LGTM, see my one small suggestion.

Let's do a quick round of testing once the CI ran through and then this should be ready to be merged 👍

[philippotto]

LGTM, see my one small suggestion.

Let's do a quick round of testing once the CI ran through and then this should be ready to be merged +1

Sounds great! I just tested a bit on https://dynamicbucketallocation.webknossos.xyz. Feel free to do so, too, and let me know whether I should merge :)

[daniel-wer]

I also tested a bit, LGTM 👍