Multiscale labeled images #45

joshmoore · 2020-08-19T13:33:53Z

ome-zarr now expects a multiscale zgroup rather than a single zarray as the content of a label. Along with the previous round of changes (#44 (comment)) this means to modify an existing ome-zarr with masks you need to:

rename masks/ to labels/
run ome_zarr scale 6001240.py/labels/{0,1}
merge the metadata from labels/0/.zattrs into labels/1/.zattrs
add 1 to the labels/.zattrs "names" element, optionally removing 0

(This will be put into a script to be run against s3.embassy.ebi.ac.uk)

Interestingly, with the current changes, loading is still successful without the changes above since the the mask is treated as a raw zarr array.

Summary of other user changes:

It's now possible to pass the labels/ directory to napari. The mask is opened visible, and the image is loaded invisible. This is the first usage of the link {"image": { "array" .... See Fix masks.[].image.array to match spec omero-cli-zarr#19 . Future work will likely make it possible to point at the label itself and have the images loaded, too.
New ome_zarr create command for creating sample data. Note: colors are not working correctly yet.
New ome_zarr scale command
ome_zarr info now displays different output.
ome_zarr download now downloads recursively.

Summary of other developer changes:

Split main script into napari.py, cli.py, utils.py, reader.py (beginning of refactoring for code reuse; multiple napari plugins may also be a good path forward).
Splitting of BaseZarr into BaseZarrLocation and Layer/Spec types (significantly simplifies different starting points)
Enforce mypy in all method definitions (aided with refactoring).

ome_zarr/reader.py

The introduction of multiple types per node of the Zarr hierarchy (e.g. multiscales AND labeled image) made the existing code flow difficult to extend and test. This reworks the previous BaseZarr type into multiple hierarches: - ome_zarr.io: Locations hierarchy for choosing Remote or Local - ome_zarr.reader: Spec hierarchy for choosing metadata types The general code flow consists of: - ome_zarr.napari.napari_get_reader() - ome_zarr.io.parse_url() - ome_zarr.reader.Reader.__call__() - ome_zarr.reader.Layer() -ome_zarr.reader.Spec() [recursion] - ome_zarr.napari.transform

- replace test lookup for labels/coins - don't write labels group if no label exists

Provides the `ome_zarr scale` CLI command as well as the `ome_zarr.scale.Scaler` class for downsampling arrays as multiscales. originally: ome/omero-ms-zarr#61

New tests and code require a substantial number of requirements like opencv and Qt. Rather than try to encapsulate that configuration directly in .travis, we copy the napari-omero strategy of using conda in GH actions.

mtbc · 2020-08-31T08:40:53Z

ome_zarr/data.py

+    if pyramid[0].shape[CHANNEL_DIMENSION] == 1:
+        image_data = {
+            "channels": [{"window": {"start": 0, "end": 1}}],
+            "rdefs": {"model": "grayscale"},


In the OMERO metadata JSON I think we previously had "greyscale" matching RenderingModel. I don't know if this matters or what it should be.

Happy to defer to @will-moore, but I agree at least the three zarr related repos should use uniform spelling.

Yup. OMERO has greyscale everywhere. napari grayscale. I'll update to use the former and if we need to, we can transform in ome_zarr.napari.

NB: ome-model primarily uses "e", "a" appears only twice in docs. Bio-Formats leans more towards "a" including the CLI argument.

ome_zarr/io.py

ome_zarr/reader.py

Co-authored-by: Mark Carroll <[email protected]>

will-moore · 2020-09-02T12:28:05Z

Testing as above with latest commits from here and omero-cli-zarr is working:

$ omero zarr export Image:139615
$ omero zarr masks Image:139615
$ napari 139615.zarr

I noticed the labels layer is visible initially. Is that a change of behaviour?
Also, I wonder if we want to rename masks -> labels for:

$ omero zarr masks Image:139615

?

will-moore · 2020-09-02T12:35:47Z

I tried downloading:

$ ome_zarr download https://s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001240.zarr/
WARNING:ome_zarr.io:unreachable: https:/s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001240.zarr/labels/.zarray -- details logged at debug
WARNING:ome_zarr.io:unreachable: https:/s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001240.zarr/labels/.zgroup -- details logged at debug
ERROR:ome_zarr.cli:. already exists!

I guess the s3 data hasn't been updated from masks -> labels yet, but even if labels are missing, should the image data itself still download OK?
e.g. When I try napari 'https://s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001240.zarr/' even though there are no labels, this opens the image fine.

joshmoore · 2020-09-02T12:58:07Z

I noticed the labels layer is visible initially. Is that a change of behaviour?

Yeah, I noticed that as well. One of my bug fixes must have reverted. I assume it's not overly critical, but I'll look into it.

I tried downloading ... I guess the s3 data hasn't been updated from masks -> labels yet

Definitely. omero zarr masks is the only source of test data at the moment. Once these PRs go in, I can update the public data. Or we hold off for the color changes.

will-moore · 2020-09-02T13:11:59Z

My point was not so much that the 'masks -> labels` updates need to happen on s3, but that any image fails to download if it doesn't have any labels.
Or, maybe it's not the lack of labels that is breaking download, but something else.
Trying another ID from https://downloads.openmicroscopy.org/presentations/2020/Dundee/Workshops/NGFF/zarr_diagram/ which I don't think has any masks/labels

$ ome_zarr download https://s3.embassy.ebi.ac.uk/idr/zarr/v0.1/179706.zarr/
WARNING:ome_zarr.io:unreachable: https:/s3.embassy.ebi.ac.uk/idr/zarr/v0.1/179706.zarr/labels/.zarray -- details logged at debug
WARNING:ome_zarr.io:unreachable: https:/s3.embassy.ebi.ac.uk/idr/zarr/v0.1/179706.zarr/labels/.zgroup -- details logged at debug
ERROR:ome_zarr.cli:. already exists!

will-moore · 2020-09-02T13:14:07Z

Ah, this was a lack of output dir.
This works:

$ ome_zarr download https://s3.embassy.ebi.ac.uk/idr/zarr/v0.1/179706.zarr/ --output test

Co-authored-by: Mark Carroll <[email protected]>

ome_zarr/reader.py

mtbc · 2020-09-03T08:57:02Z

ome_zarr/reader.py

+        node which will likely be turned into a lower layer for display.
+
+        By unsetting visible, the node (and in turn the layer) can be
+        deactivated for initial display. By default, the value of the current


Slightly confused: "by default" suggests a means of override but I am not seeing what that is.

Passing a value would override the None. I'm pushing a commit to rework the wording. Let me know if you think it's better.

will-moore · 2020-09-03T16:26:17Z

The labels layer is not visible when opened in napari now. 👍

For $ ome_zarr download https://s3.embassy.ebi.ac.uk/idr/zarr/v0.1/179706.zarr/, should --output be a required parameter (since it will fail with the default output dir of .)?

sbesson

👍 for the overall refactoring of this class into more tractable and testable modules. Are we effectively approaching an 0.1.0?

sbesson · 2020-09-03T21:07:02Z

ome_zarr/reader.py

+        self.pre_nodes: List[Node] = []
+        self.post_nodes: List[Node] = []
+
+        # TODO: this should be some form of plugin infra over subclasses


I assume another todo might be to add some validation for incompatible specs i.e. I assume a node cannot be both a Label and Labels

If the Labels pointed to itself with . it might just work...

ome_zarr/data.py

joshmoore · 2020-09-04T06:42:40Z

should --output be a required parameter (since it will fail with the default output dir of .)?

Ah thanks. I've fixed this to work again without the flag.

joshmoore · 2020-09-04T09:22:30Z

Non-Windows tasks are all now passing. It's going to be fiddly to get Windows passing, so I'd like to do that in a new PR.

Any objections to merging?

mtbc · 2020-09-04T09:26:56Z

GFI. I think any fixes will be much smaller increments than this.

joshmoore added 6 commits August 19, 2020 15:23

Open images for a mask via the array link

c45f450

Move python code to a module

47d8563

Fix mypy errors

aafd5ed

Refactor to re-use multiscale loading

185157f

Have mypy check all API methods

266b357

Enfore mypy annotations on all definitions

fc8df8f

mtbc reviewed Aug 19, 2020

View reviewed changes

ome_zarr/reader.py Outdated Show resolved Hide resolved

joshmoore mentioned this pull request Aug 19, 2020

Labeled multiscales ome/omero-ms-zarr#70

Merged

Fix 'bool(false)' in color test

db43af5

joshmoore mentioned this pull request Aug 19, 2020

Fix masks.[].image.array to match spec ome/omero-cli-zarr#19

Closed

joshmoore added 10 commits August 21, 2020 14:59

move create_test_data to ome_zarr.data

967b36c

Add isort as the first pre-commit step

c146101

refactor create_zarr method for use from CLI

1c50a26

More tests and bug fixes

7c8cff1

Deal with empty labels from astronauts

b57d439

- replace test lookup for labels/coins - don't write labels group if no label exists

ome_zarr.scale: migrate scale.py to ome_zarr

9665c4c

Provides the `ome_zarr scale` CLI command as well as the `ome_zarr.scale.Scaler` class for downsampling arrays as multiscales. originally: ome/omero-ms-zarr#61

Enable make_test_viewer from napari

6b18581

Fix recursive reading, info, and downloading

4fb4213

Fix number of channels in coins()

c4ef738

joshmoore mentioned this pull request Aug 27, 2020

scripts: add scale.py ome/omero-ms-zarr#61

Closed

joshmoore added 2 commits August 31, 2020 08:29

Fix CLI tools incl. prefix stripping

77e631a

Update style and fill out well-formed docs

7a4f865

joshmoore force-pushed the multiscale branch from 6de3d30 to 461979f Compare August 31, 2020 08:01

Use GH Actions & Conda for tests

56f7ef5

New tests and code require a substantial number of requirements like opencv and Qt. Rather than try to encapsulate that configuration directly in .travis, we copy the napari-omero strategy of using conda in GH actions.

joshmoore force-pushed the multiscale branch from 461979f to 56f7ef5 Compare August 31, 2020 08:18

mtbc reviewed Aug 31, 2020

View reviewed changes

ome_zarr/io.py Show resolved Hide resolved

mtbc reviewed Aug 31, 2020

View reviewed changes

ome_zarr/reader.py Outdated Show resolved Hide resolved

Fix "loaded as" doc sentence

12b9493

Co-authored-by: Mark Carroll <[email protected]>

Change 'Layer' to 'Node'

8125dd4

joshmoore and others added 5 commits September 3, 2020 08:42

Remove is_zarr() in favor of exists()

31082bc

Fix double negative

0a95ff2

Co-authored-by: Mark Carroll <[email protected]>

Remove extra 'of'

f2cff53

Rename test_layer to test_node

c0806ec

Re-instate visiblility via a property

6cc1c25

mtbc reviewed Sep 3, 2020

View reviewed changes

ome_zarr/reader.py Outdated Show resolved Hide resolved

mtbc reviewed Sep 3, 2020

View reviewed changes

sbesson reviewed Sep 3, 2020

View reviewed changes

Use 'greyscale'

171e122

joshmoore force-pushed the multiscale branch from 3c428c8 to fee5d7a Compare September 4, 2020 06:54

joshmoore added 3 commits September 4, 2020 09:10

Restore downloading without --output

ee73e06

Rework Node.add documentation

d2f9f1c

Disable napari viewer test on non-OSX platforms

fbfa932

joshmoore force-pushed the multiscale branch from fee5d7a to fbfa932 Compare September 4, 2020 07:10

Use bash script with activated conda environment on Windows

4ed3078

joshmoore force-pushed the multiscale branch from a211f82 to 4ed3078 Compare September 4, 2020 08:05

sbesson mentioned this pull request Sep 4, 2020

Multiscale support ome/omero-cli-zarr#25

Merged

joshmoore merged commit 8199ed6 into ome:master Sep 4, 2020

joshmoore deleted the multiscale branch September 4, 2020 10:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multiscale labeled images #45

Multiscale labeled images #45

joshmoore commented Aug 19, 2020 •

edited

Loading

mtbc Aug 31, 2020

joshmoore Aug 31, 2020

joshmoore Sep 4, 2020

will-moore commented Sep 2, 2020

will-moore commented Sep 2, 2020

joshmoore commented Sep 2, 2020

will-moore commented Sep 2, 2020

will-moore commented Sep 2, 2020

mtbc Sep 3, 2020

joshmoore Sep 4, 2020

will-moore commented Sep 3, 2020

sbesson left a comment

sbesson Sep 3, 2020

joshmoore Sep 4, 2020

joshmoore commented Sep 4, 2020

joshmoore commented Sep 4, 2020

mtbc commented Sep 4, 2020

Multiscale labeled images #45

Multiscale labeled images #45

Conversation

joshmoore commented Aug 19, 2020 • edited Loading

mtbc Aug 31, 2020

Choose a reason for hiding this comment

joshmoore Aug 31, 2020

Choose a reason for hiding this comment

joshmoore Sep 4, 2020

Choose a reason for hiding this comment

will-moore commented Sep 2, 2020

will-moore commented Sep 2, 2020

joshmoore commented Sep 2, 2020

will-moore commented Sep 2, 2020

will-moore commented Sep 2, 2020

mtbc Sep 3, 2020

Choose a reason for hiding this comment

joshmoore Sep 4, 2020

Choose a reason for hiding this comment

will-moore commented Sep 3, 2020

sbesson left a comment

Choose a reason for hiding this comment

sbesson Sep 3, 2020

Choose a reason for hiding this comment

joshmoore Sep 4, 2020

Choose a reason for hiding this comment

joshmoore commented Sep 4, 2020

joshmoore commented Sep 4, 2020

mtbc commented Sep 4, 2020

joshmoore commented Aug 19, 2020 •

edited

Loading