Patch v1.3.2 #198

pattonw · 2024-01-03T04:12:44Z

Merging patch branch into master and making new release

Change Log:
bugs fixed:

torch nodes forward hooks moved to start method of nodes to work better with the spawn_subprocess flag
torch train/predict work when setting multiprocessing start method = "spawn"
fixed bug in rasterize graph due to usage of the old api for accessing graph nodes

new features:

torch nodes support defining specific cuda devices via e.g. "cuda:0"
added support for "reflect" mode padding in the pad node
improved error printing.
- Reversed the order of exceptions so the cause of the error is printed at the bottom, and you can scroll up to trace the requests and batches through the tree. Now we no longer need to scroll to the top to see the cause of the error,
- PreCache and other multiprocessing nodes no print repeats of the same error, it is simply printed once
- removed unhelpful tracebacks (and many repititions) for the try, except blocks in node superclasses

general improvements:

moved many tests from unittest to pytest, adding parametrization to simplify them and cover more cases
improved the dependency version bounds
improved docs, formatting, and pass more ci/cd workflows
removed unused imports and f-strings

fix documentation to be more accurate around nonspatial arrays

allow None roi/voxel size for spatial arrays

no longer assumes a deformed label will still exist in an array

This is to get around local functions (i.e. the hooks) not being pickle-able which we need for the "spawn" start function (spawn is the default on windows and recent macs)

We want to test with both fork and spawn start methods, but this seems to interfere with the torch tests

numpy is no longer releasing updates for python 3.7, they are on 1.24 but the last release for 3.7 was 1.21. I don't think we need to support it either, but we should test on 3.11

if using start method = "spawn" and the "start_subprocess" flag for the predict node, we now pass our test.

if using the start method "spawn", and the "spawn_subprocess" flag for the train node, we now pass our test

Now prints the errors in reverse order of execution so the initial pipeline error is printed first

commit 1686b949766b76960534ede1105751591fd91c9f Author: William Patton <[email protected]> Date: Tue Dec 19 08:43:11 2023 -0700 black reformatting commit 26d2c7cfff3f2702f56a5bb4249a0811f54b45ef Author: Mohinta2892 <[email protected]> Date: Thu Nov 2 19:09:15 2023 +0000 Revert "black reformatted" This reverts commit 66dd69b. Only format changed files, since black does not consider formatting history commit a273fd3813fc16b516c2438ad5af0c4ee3f0686b Author: Samia Mohinta <[email protected]> Date: Thu Nov 2 17:12:26 2023 +0000 black reformatted commit bb37769eec33af5921386f283e2579055bb34e6d Author: Samia Mohinta <[email protected]> Date: Thu Nov 2 16:40:32 2023 +0000 add device arg Allow passing cuda device to Predict. Issue #188 commit a3b3588a1406d609ae95370cf2c5339872616011 Author: Samia Mohinta <[email protected]> Date: Thu Nov 2 16:39:09 2023 +0000 add device arg allow passing cuda device to Train

currently failing a few of them, some are expected failures.

Squashed commit of the following: commit 0fb29c8 Author: William Patton <[email protected]> Date: Tue Jan 2 08:54:17 2024 -0800 replace custom padding code with np.pad commit c6928bd Author: William Patton <[email protected]> Date: Tue Jan 2 08:54:06 2024 -0800 simplify/expand padding test test padding on both sides commit 3782525 Author: William Patton <[email protected]> Date: Tue Dec 19 11:30:31 2023 -0700 pass the fixed tests commit a7027c6 Author: William Patton <[email protected]> Date: Tue Dec 19 10:37:48 2023 -0700 fix the test case commit 531d81d Author: William Patton <[email protected]> Date: Tue Dec 19 10:06:44 2023 -0700 update the pad tests parametrized the use of constant or reflect padding. Now avoids using the unittest framework commit 443c666 Author: Manan Lalit <[email protected]> Date: Fri Nov 3 00:09:33 2023 -0400 Replace .ndim by len() commit a7503d7 Author: lmanan <[email protected]> Date: Thu Nov 2 11:52:27 2023 -0400 Update pad.py to include reflective padding

we were using `graph.data.items()` to iterate over nodes instead of `graph.nodes` Squashed commit of the following: commit d027f5a260a1e2a9cf851efca85b7318434675d6 Author: William Patton <[email protected]> Date: Tue Jan 2 09:44:21 2024 -0800 refactor rasterize_points test to use pytest commit eadb0476d8475b55120486df6cf30f95b6df86f4 Author: William Patton <[email protected]> Date: Tue Jan 2 09:25:11 2024 -0800 remove extra roi handling The node only needs to request the data it needs for its own operations. If you request a mask for a set of points that extend outside the bounds of your mask you will get an error commit 29507f1f21d69cf76e34e7b0f05cd780100fd68b Author: William Patton <[email protected]> Date: Tue Jan 2 09:22:21 2024 -0800 remove type cast we do a bitwise during the `__rasterize` call which results fails if you change the dtype commit 96e93e53ce0bc8240357259dab92f1ca64a08199 Author: William Patton <[email protected]> Date: Tue Jan 2 09:21:16 2024 -0800 remove matplotlib commit eb2977a187a1cad95da54a515c84ce44d73b8315 Author: Samia Mohinta <[email protected]> Date: Thu Dec 14 15:27:41 2023 +0000 fix mask intersection with request outputs must match request rois when a mask is provided commit 682189dac2ef6b94876bd30df813717da6530060 Author: Samia Mohinta <[email protected]> Date: Thu Dec 14 15:25:12 2023 +0000 Update rasterize_graph.py commit e36dcf179ccd1aec6a5cafd31e7a9a858352faa1 Author: Mohinta2892 <[email protected]> Date: Thu Nov 2 19:18:00 2023 +0000 reformat rasterize_graph and rasterize_points commit 42da2702e746f702d3d07144ab9fc1d4352b0c0d Author: Samia Mohinta <[email protected]> Date: Thu Nov 2 14:23:01 2023 +0000 Test for issue #193 Test added to pass mask to `RasterizeGraph()` via `RasterizationSettings`. commit b17cfad413f5ad7f48045a2167ec20d89674d939 Author: Samia Mohinta <[email protected]> Date: Thu Nov 2 14:19:42 2023 +0000 fix for issue #193 lines 224-226: replace graph.data.items() with graph.nodes lines 255-257: explicitly cast the boolean mask data to the original dtype of mask_array

We can filter out some more of the excess error traceback that isn't helpful to the readers.

…ailable if torch isn't installed

codecov-commenter · 2024-01-03T04:45:50Z

Codecov Report

Attention: 17 lines in your changes are missing coverage. Please review.

Comparison is base (a33e5b4) 67.60% compared to head (cceeb41) 67.51%.
Report is 1 commits behind head on main.

Files	Patch %	Lines
gunpowder/torch/nodes/predict.py	16.66%	5 Missing ⚠️
gunpowder/torch/nodes/train.py	33.33%	4 Missing ⚠️
gunpowder/ext/__init__.py	81.25%	3 Missing ⚠️
gunpowder/contrib/nodes/add_blobs_from_points.py	0.00%	1 Missing ⚠️
gunpowder/jax/nodes/train.py	50.00%	1 Missing ⚠️
gunpowder/nodes/exclude_labels.py	0.00%	1 Missing ⚠️
gunpowder/nodes/generic_predict.py	50.00%	1 Missing ⚠️
gunpowder/nodes/generic_train.py	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #198      +/-   ##
==========================================
- Coverage   67.60%   67.51%   -0.10%     
==========================================
  Files          98       98              
  Lines        6829     6810      -19     
==========================================
- Hits         4617     4598      -19     
  Misses       2212     2212

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pattonw and others added 30 commits October 6, 2023 13:37

remove duplicated for loop

ec88b0c

increment patch number

abcecf2

ArraySpec docs

b557548

fix documentation to be more accurate around nonspatial arrays

ArraySpec bug fix:

80033bd

allow None roi/voxel size for spatial arrays

fix the deform augment test

7f8b877

no longer assumes a deformed label will still exist in an array

better bounds on required packages

f1fd63a

ignore missing imports from packages that don't provide type hints

f96aa78

fix typehint mistakes

98bbe71

format pyproject.toml

3ba99da

black format

7a10397

move register hooks to the start method

1615a7d

This is to get around local functions (i.e. the hooks) not being pickle-able which we need for the "spawn" start function (spawn is the default on windows and recent macs)

fix typo

4a8ccf4

support non-spatial arrays in ArraySource

a001d7d

overhaul torch tests

98a2e34

remove multiprocess set start method monkey patch

96ff2f1

We want to test with both fork and spawn start methods, but this seems to interfere with the torch tests

only deploy docs on tagged commits to main

75ff6ff

minor black formatting and configuration changes

865fe53

properly skip torch tests if torch not installed

2078036

black formatting

46676dd

avoid testing on python 3.7, instead use 3.11

84593e8

numpy is no longer releasing updates for python 3.7, they are on 1.24 but the last release for 3.7 was 1.21. I don't think we need to support it either, but we should test on 3.11

add typed libraries to dev dependencies

ed3c7ce

pass torch train test

d49db1f

if using start method = "spawn" and the "start_subprocess" flag for the predict node, we now pass our test.

pass torch train test

797994c

if using the start method "spawn", and the "spawn_subprocess" flag for the train node, we now pass our test

remove extra error printing

b2f8c2d

switch error printing order

625cb03

Now prints the errors in reverse order of execution so the initial pipeline error is printed first

black format docs and examples

35fcd43

parameterize tests for cuda devices

b6c425f

currently failing a few of them, some are expected failures.

pattonw added 8 commits January 2, 2024 09:54

ruff: remove unused imports and fix small typos.

d55c976

Custom BatchRequestError handling in pipeline.request_batch

00543bf

We can filter out some more of the excess error traceback that isn't helpful to the readers.

black formatting

431f106

mypy workflow use dev dependencies

c069592

avoid testing on python 3.8, it doesn't support typing very well

ecb55e7

fix type hint for logdir in torch train

d7b5673

switch order of decorators to avoid trying to determine if cuda is av…

bcfb7d7

…ailable if torch isn't installed

check if torch is installed before checking if cuda is available

4be523e

update funlib.geometry version for mypy typing

cceeb41

pattonw merged commit 52248e1 into main Jan 3, 2024
7 checks passed

pattonw deleted the patch-v1.3.2 branch January 3, 2024 05:30

pattonw restored the patch-v1.3.2 branch January 3, 2024 05:30

pattonw deleted the patch-v1.3.2 branch January 3, 2024 05:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Patch v1.3.2 #198

Patch v1.3.2 #198

pattonw commented Jan 3, 2024

codecov-commenter commented Jan 3, 2024 •

edited

Loading

Patch v1.3.2 #198

Patch v1.3.2 #198

Conversation

pattonw commented Jan 3, 2024

codecov-commenter commented Jan 3, 2024 • edited Loading

Codecov Report

codecov-commenter commented Jan 3, 2024 •

edited

Loading