Where to start for segmentation and assisted annotation of tiff files

[natbutter]

I have a series of microscopy images (of human liver parenchymal mitochondria) as a stack of tiff files, that look like this.

I want to segment the different mitochondria, lipid droplets, nuclei, etc (basically all the blobs in the image) in a more automated fashion to what was done manually using imod in the paper this is from (Shami et al 2021).

I am bit overwhelmed with which "app" and "model" to use to begin with. I have tried various apps (pathology, radiology), file formats (tiff, png, jpg, nii.gz), models (segmentation, nuclick), and viewers (Slicer, QuPath) but manage just to crash MONAI or do a very poor segmentation but then can't add more labels without crashing MONAI again.

Any suggestions for a reasonable approach?

Cheers

Shami, Gerald J., et al. "Three-dimensional ultrastructure of giant mitochondria in human non-alcoholic fatty liver disease." Scientific reports 11.1 (2021): 1-14

[SachidanandAlle]

@tangy5 can you help @natbutter and guide him with DSA/QuPath options..
I am ok with current pretrained models (segmentation/nuclick poorly segmenting as they are trained on very tiny public dataset)
In your case, the images might be having different scale/resolution/color/jitter etc..

However it shall be interesting for us to know a bit more on Crash situations.. Server is crashing? Client is crashing?
What kind of errors at the server if it's crashing.. possibly you can share server logs.. and also explain what operations is causing this.. is it repeatable everytime u run that action..?

also helpful if you can share your system configs (cpu/memory/gpu etc..)

[natbutter]

Thanks for responding!
I think "crash" was a bit broad. Generally, errors are explanatory and reproducible, but I have tried many combinations so forget which flow results in which crash. I should note all the tutorials work smoothly. But when using my own data and straying from the comfort of the examples I get errors.... for example trying to run the pathology app with QuPath on a tiff file. Seems to load okay, then when I execute "Monai Label > Annotations > Okay" I get the error in the QuPath terminal:

...
java.io.IOException: (POST http://localhost:8000/infer/wsi/segmentation_nuclei?output=asap&image=HM25_HighRes_Aligned0020) 500
...

but more helpfully, concurrently the MONAI server error gives:

openslide.lowlevel.OpenSlideUnsupportedFormatError: Unsupported or missing image file

Presumably because this tiff file is unsupported.

I have run a few different cloud and local systems, with comparable behavior.
Currently running on 2xV100 each with 16GB vram each, 16cpu and 32gb RAM. I run the viewer on my local computer and connect via with an ssh tunnel to the server.

I will respond to tangy5 below regarding the model choice and workflow, but if anything is "unexpected" I will be sure to post! Thanks!

[tangy5]

Hi @natbutter , thanks for posting the message. Regarding your case, I feel you will need to define and train your own model. Currently, we don't have an existing model for segmenting the unstained pathology image. All nuclei segmentation models are trained using H&E stained images. In addition, you have some new cell classes. These probably the reason why we can't get good segmentation results.

1. Firstly, though the tasks are different, you can still use the pathology app as the sample app for adding a new model. You could refer to this doc.
   https://github.com/Project-MONAI/MONAILabel/tree/main/sample-apps/pathology
2. If you already have some ground truth labels, you can train your own model using the pathology app, reference to the segmentation_nuclei model, you can train from scratch using your data and give a different model name.
3. We have a tutorial on how to start monailabel with QuPath here: https://github.com/Project-MONAI/tutorials/blob/main/monailabel/monailabel_pathology_nuclei_segmentation_QuPath.ipynb. You could follow the steps to setup QuPath, you can start with training first if you don't have a good model for inference currently.

[natbutter]

Thank you, really appreciate you taking the time.

This is good insight. QuPath demo is running well.
I don't have labelled data, nor a pre-trained model.
I have briefly tried using the segmentation_nuclei model (setting pretrained false) but after labeling a few classes then submitting labels, and then training I get these one of these two errors.

Traceback (most recent call last):
  File "/build/miniconda3/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
    fn(i, *args)
  File "/build/miniconda3/lib/python3.8/site-packages/monailabel/tasks/train/basic_train.py", line 728, in main_worker
    task.train(rank, world_size, request, datalist)
  File "/build/miniconda3/lib/python3.8/site-packages/monailabel/tasks/train/basic_train.py", line 541, in train
    context.trainer = self._create_trainer(context)
  File "/build/miniconda3/lib/python3.8/site-packages/monailabel/tasks/train/basic_train.py", line 693, in _create_trainer
    train_data_loader=self.train_data_loader(context),
  File "/build/miniconda3/lib/python3.8/site-packages/monailabel/tasks/train/basic_train.py", line 267, in train_data_loader
    return self._dataloader(context, dataset, context.train_batch_size, num_workers, shuffle)
  File "/build/miniconda3/lib/python3.8/site-packages/monailabel/tasks/train/basic_train.py", line 255, in _dataloader
    return DataLoader(
  File "/build/miniconda3/lib/python3.8/site-packages/monai/data/dataloader.py", line 89, in __init__
    super().__init__(dataset=dataset, num_workers=num_workers, **kwargs)
  File "/build/miniconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 277, in __init__
    sampler = RandomSampler(dataset, generator=generator)  # type: ignore[arg-type]
  File "/build/miniconda3/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 97, in __init__
    raise ValueError("num_samples should be a positive integer "
ValueError: num_samples should be a positive integer value, but got num_samples=0
[2023-05-29 05:23:54,703] [447046] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:83) - Return code: 1

Traceback (most recent call last):
 File "/build/miniconda3/lib/python3.8/runpy.py", line 194, in _run_module_as_main
   return _run_code(code, main_globals, None,
 File "/build/miniconda3/lib/python3.8/runpy.py", line 87, in _run_code
   exec(code, run_globals)
 File "/build/miniconda3/lib/python3.8/site-packages/monailabel/interfaces/utils/app.py", line 132, in <module>
   run_main()
 File "/build/miniconda3/lib/python3.8/site-packages/monailabel/interfaces/utils/app.py", line 117, in run_main
   result = a.train(request)
 File "/build/miniconda3/lib/python3.8/site-packages/monailabel/interfaces/app.py", line 429, in train
   result = task(request, self.datastore())
 File "/build/miniconda3/lib/python3.8/site-packages/monailabel/tasks/train/bundle.py", line 227, in __call__
   self._load_checkpoint(os.path.join(self.bundle_path, "models"), pretrained, train_handlers)
 File "/build/miniconda3/lib/python3.8/site-packages/monailabel/tasks/train/bundle.py", line 180, in _load_checkpoint
   if os.path.exists(load_path):
 File "/build/miniconda3/lib/python3.8/genericpath.py", line 19, in exists
   os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
[2023-05-29 05:27:44,883] [447046] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:83) - Return code: 1

I will explore a bit more and upload full error logs soon.

[GabLop12]

I had the same problem. I have used SAM model. It works great!