Gunpowder AssertionError: Cannot pick a random location, intersection of upstream ROIs is unbounded.

[Mohinta2892]

Error in RandomLocation:
AssertionError: Can not pick a random location, intersection of upstream ROIs is unbounded.

This happens if the input shape is larger than the shape of the volumes.
Example:
You have bunch of volumes of which the smallest has the following shapes, it will clearly cause a problem as the raw is never padded.

• Shape of your labels array is 150 x 150 in yx.
• Shape of input to model is 268 x 268 in zyx.
• Shape of output from model is 168 x 168 in zyx.

[Mohinta2892]

For Zebrafinch ground truth data, the labels exist in 128 x 256 x 256 and 150 x 150 x 150 px.
Labels padding when calculated as below does not work:

    p = int(round(np.sqrt(np.sum([i * i for i in output_shape])) / 2))

    # Ensure that our padding is the closest multiple of our resolution
    labels_padding = Coordinate([j * round(i / j) for i, j in zip([p, p, p], list(voxel_size))])
    print('Labels padding:', labels_padding)
    

We do two things to circumvent this problem (may not be the best solution, works for now):

• Calculate labels padding in nm as: output_size / 2
• Only train on volumes with labels shape 150 x 150 x 150

[Mohinta2892]

Why this happens?

• Your input files have varied array shapes, some much smaller than the input_shape in config.py file.
  Solution: Adjust the input_shape value in the config.py. Ensure that even after padding the input_shape remains within the input roi shape.

• This can happen also when labels shape is smaller than raw shape. As for example, when the roi was never densely painted as in CREMI.
  Solution:

  • Fix 1: Crop the raw to match the labels. OR Pad the labels to match the raw.
  • Fix 2: Turn off labels_padding in line 182 for labels here.
    Note: you cannot do Pad(labels_mask, None), it will throw ERRORS as below. There is potentially a bug somewhere in gunpowder that leads to this. So, keep Pad(labels_mask, labels_padding). It should not disrupt the model training, as far as what we have tested.

ERROR:

Traceback (most recent call last):                  
  File "/net/ark/scratch/smohinta/connexion/codebases/restructured_packages/local_shape_descriptors/conda_env/src/gunpowder/gunpowder/nodes/batch_provider.py", line 182, in request_batch
    self.check_request_consistency(request)                                                                                                                                                                                                                                        
  File "/net/ark/scratch/smohinta/connexion/codebases/restructured_packages/local_shape_descriptors/conda_env/src/gunpowder/gunpowder/nodes/batch_provider.py", line 249, in check_request_consistency
    request_roi.shape[d] % provided_spec.voxel_size[d] == 0                                                                                                                                                                                                                        
TypeError: unsupported operand type(s) for %: 'NoneType' and 'int'