Allow batch reading of mesh chunks

[frcroth]

URL of deployed dev instance (used for testing):

• https://optimizepremeshesoverseg.webknossos.xyz/

Steps to test:

• open l4dense_motta_et_al_dev_v2 in view mode
• enable mapping 80
• load a precomputed mesh for a segment (e.g., at 2905, 4566, 1786)

TODO

• Adapt front end
• Use new format on other route
• Tweak performance so that it generalizes over different mesh files
• [ ] Use uint32 instead of uint64 for jump table so that Uint32Array can be used in JS
• ensure proper error handling when decoding fails (independent of this PR, strictly speaking)
• debug why some requests cannot be decoded (independent of this PR, strictly speaking)

Issues:

• fixes Optimize loading of precomputed meshes of oversegmentation #6752

---

(Please delete unneeded items, merge only when none are left open)

• Updated changelog
• Updated migration guide if applicable
• Updated documentation if applicable
• Adapted wk-libs python client if relevant API parts change
• Removed dev-only changes like prints and application.conf edits
• Considered common edge cases
• Needs datastore update after deployment

[frcroth]

@philippotto Backend should be ready (though not completely tested)

[philippotto]

I just noticed that THREE.DRACOLoader: Unexpected geometry type. is thrown on the dev instance for some chunk (most of the mesh loads fine). This is already the case on master, too. See the testing steps in the PR description which trigger this.

The front-end doesn't handle this correctly currently (the spinner just keeps spinning). I'll add proper error handling and then we should be able to debug this further.

[philippotto]

I think, the PR is ready for review. @daniel-wer It can make sense to review the PR by commit range (I did some refactoring of the isosurface saga were I simply extracted/moved some code). Also, I upgraded the draco libs.

@frcroth @fm3 See my previous comment regarding an error where a mesh chunk cannot be decoded. I added better error handling and reporting, but I couldn't really figure out what's happening. I only found one segment where one mesh request fails and that is:

Requested chunk: {position: [1024, 5120, 1024], byteOffset: 3920109208, byteSize: 1818}
Error: {type: 'error', id: 9592, error: 'THREE.DRACOLoader: Unexpected geometry type.'}

It happens with l4dense_motta_et_al_dev_v2/segmentation/meshes/meshfile_mapping0.hdf5 with mesh id 355 and mapping 80. The mesh file is 13 GB big..

Maybe you have an idea what could be causing this. However, I think, it's not blocking this PR as this is already an issue on master.

[philippotto]

Bug can be seen isolated on this dev instance: https://provokemeshdecodingerror.webknossos.xyz/datasets/sample_organization/l4dense_motta_et_al_dev_v2/view#2816,4353,1792,0,1.3 (enable mapping 80 and load a mesh).

[fm3]

Works for me :) Great speed increase! I added a note on parallelization.

@philippotto could you write an issue for the draco decoding problem?

[philippotto]

@philippotto could you write an issue for the draco decoding problem?

Done! #7044

[daniel-wer]

Works very well, awesome speed boost!

[philippotto]

@frcroth just a small ping in case this PR slipped through your inbox :)

[frcroth]

@frcroth just a small ping in case this PR slipped through your inbox :)

@philippotto I'm a bit out of the loop on this one. What should I do?

[philippotto]

@frcroth just a small ping in case this PR slipped through your inbox :)

@philippotto I'm a bit out of the loop on this one. What should I do?

Only the PR feedback was missing, but I guess this is good now 👍 @fm3 Do you want to leave a final approval?

[fm3]

Boom 🚀