Volume annotation download: zip with BEST_SPEED

[fm3]

Use Deflater.BEST_SPEED (level 1) instead of level 6 when writing data.zip containing volume buckets and also when adding that file to the outer annotation zip.

My experiments showed a few interesting insights

• For volume annotation buckets, there is no huge difference in zip size between level 1 (BEST_SPEED) and 6 (DEFAULT), but there is a ~40% speedup.
• Level 0 (NO_COMPRESSION) has an even greater speedup (another 20% compared to 1, possibly more for smaller annotations) but produces significantly larger zip files (12 times as large in my test case).
• The file paths are a significant part of the archive (lz4 compressed buckets are small, but 500k files lead to large amount of paths)
• It appears zip does not (very well?) compress its file path listing
• This is why it is possible to compress the already-compressed data.zip file again on the wk-side with actual size reduction (getting 60% reduction for large annotation)
• Note that this varies heavily with the zip implementation. The zip command line tool also offers levels from 0 to 9 but got very different results both in size and timing. I measured with the same java implementation that our backend uses
• See full experiments below

URL of deployed dev instance (used for testing):

• https://zippartialcompression.webknossos.xyz

Steps to test:

• Create hybrid annotation, download it
• should still be a valid zip with valid inner data.zip

Issues:

• fixes Optimize Performance for downloading annotations #6002

---

• Updated (unreleased) changelog
• Needs tracingstore update after deployment
• Ready for review

Measurements

2022-02-10 measure performance of zipping (already lz4-compressed) volume annotation buckets.
All files were in memory before, as Tuple[Path,Array[Byte]]. Paths are relative. level -1 means default, I believe it is 6

Huge soma volume annotation, 510501 files, total size (without paths): 486,876,075 bytes
compressing buckets into data.zip:
Level  0, ratio 114.3 % took  3270 ms  # size increase because the file paths are in the archive too
Level  0, ratio 114.3 % took  2651 ms
Level  0, ratio 114.3 % took  2466 ms
Level  0, ratio 114.3 % took  3151 ms
Level  1, ratio  23.8 % took  5927 ms
Level  1, ratio  23.8 % took  5061 ms
Level  1, ratio  23.8 % took  5204 ms
Level  1, ratio  23.8 % took  6942 ms
Level  2, ratio  23.7 % took  6658 ms
Level  2, ratio  23.7 % took  6731 ms
Level  2, ratio  23.7 % took  7533 ms
Level  2, ratio  23.7 % took  7649 ms
Level -1, ratio  22.9 % took  9102 ms
Level -1, ratio  22.9 % took  9034 ms
Level -1, ratio  22.9 % took  8982 ms
Level -1, ratio  22.9 % took  9065 ms
Level  9, ratio  22.9 % took  9990 ms
Level  9, ratio  22.9 % took 10078 ms
Level  9, ratio  22.9 % took 10175 ms
Level  9, ratio  22.9 % took 10075 ms


Smallish dense volume annotation, 688 files, total size (without paths): 4,265,446 bytes
compressing buckets into data.zip:

Level  0, ratio 102.1 % took   6 ms
Level  0, ratio 102.1 % took   7 ms
Level  0, ratio 102.1 % took   6 ms
Level  0, ratio 102.1 % took   6 ms
Level  1, ratio  52.7 % took  79 ms
Level  1, ratio  52.7 % took  75 ms
Level  1, ratio  52.7 % took  76 ms
Level  1, ratio  52.7 % took  75 ms
Level  2, ratio  52.0 % took  85 ms
Level  2, ratio  52.0 % took  82 ms
Level  2, ratio  52.0 % took  81 ms
Level  2, ratio  52.0 % took  83 ms
Level -1, ratio  49.7 % took 147 ms
Level -1, ratio  49.7 % took 147 ms
Level -1, ratio  49.7 % took 148 ms
Level -1, ratio  49.7 % took 147 ms
Level  9, ratio  49.7 % took 152 ms
Level  9, ratio  49.7 % took 151 ms
Level  9, ratio  49.7 % took 148 ms
Level  9, ratio  49.7 % took 148 ms



Compressing the inner (soma annotation, see above) data.zip (level-1, 116 MB, 500k files) again:

Level  0, ratio 100.0 % took  157 ms
Level  0, ratio 100.0 % took  183 ms
Level  0, ratio 100.0 % took  133 ms
Level  0, ratio 100.0 % took  131 ms
Level  1, ratio  38.2 % took 1318 ms
Level  1, ratio  38.2 % took 1253 ms
Level  1, ratio  38.2 % took 1272 ms
Level  1, ratio  38.2 % took 1296 ms
Level  2, ratio  38.1 % took 1367 ms
Level  2, ratio  38.1 % took 1323 ms
Level  2, ratio  38.1 % took 1328 ms
Level  2, ratio  38.1 % took 1309 ms
Level -1, ratio  36.9 % took 2046 ms
Level -1, ratio  36.9 % took 2054 ms
Level -1, ratio  36.9 % took 2056 ms
Level -1, ratio  36.9 % took 2335 ms
Level  9, ratio  36.5 % took 7614 ms
Level  9, ratio  36.5 % took 7050 ms
Level  9, ratio  36.5 % took 6499 ms
Level  9, ratio  36.5 % took 6415 ms

[jstriebel]

Great findings, LGTM! 👍

It appears zip does not (very well?) compress its file path listing
This is why it is possible to compress the already-compressed data.zip file again on the wk-side with actual size reduction (getting 60% reduction for large annotation)

Interesting that this is a bottleneck in our case. This might be an indicator that we want to have larger file-lengths, resulting in fewer shards. But that surely is a different issue.