Introduce data vault as storage backend abstraction

[frcroth]

URL of deployed dev instance (used for testing):

• https://newfilesystem.webknossos.xyz

Steps to test:

• Explore different datasets:
  • Via Google Cloud Storage
    • gs://neuroglancer-fafb-data/fafb_v14/fafb_v14_orig/
    • (Datasets with authentication?)
  • Via HTTPS:
    • https://storage.googleapis.com/neuroglancer-fafb-data/fafb_v14/fafb_v14_orig/
    • https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0047A/4496763.zarr (with authentication)
  • Via s3
    • s3://s3.amazonaws.com/webknossos-zarr/demodata/6001251.zarr/0/ (with authentication)
    • (Datasets without authentication?)
• View datasets
  • s3, https, gcs, local, as well as precomputed, zarr, n5

TODOS

• GCS url handling inconsistencies (trailing slash)
• Local files
• Remove file systems that are now no longer needed
• Fix name of layers when exploring
• New name remote file system
• Delete FileSystemStore

Name

• DataVault: This name suggests that the interface provides a secure place to store and retrieve data. It implies that the data is protected and can only be accessed by authorized parties.

Questions

• GoogleCloudDataVault or GoogleCloudStorageDataVault

Issues:

• fixes Replace NIO FileSystems by custom abstraction, allowing range requests #6896

---

(Please delete unneeded items, merge only when none are left open)

• Removed dev-only changes like prints and application.conf edits
• Considered common edge cases
• Needs datastore update after deployment

[fm3]

Amazing stuff, props for getting through this so quickly, and even including tests :) Very much enjoying the diff line count as well 😆 This already feels a lot more designed and less patched together, compared to using bits and pieces of the NIO providers.

I added some suggestions. Most are fairly small, but one thing I think is interesting: whether or not to pass an additional key to the actual read calls, or whether the key can itself be mapped to the VaultPath instances. Maybe you could read through my coments and have a look at that. I’m open to discussing this, it is well possible that I missed some cases that make the current version necessary.

[fm3]

Very nice cleanup! We are getting very close :)

[fm3]

Looks good and works for me :)

I see that the ”Removed dev-only changes” checkbox is still open, is there something left? If so, could you clear that up? Then this should be ready to go