monailabel/monailabel_monaibundle_3dslicer_lung_nodule_detection.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) MONAI Consortium  \n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");  \n",
    "you may not use this file except in compliance with the License.  \n",
    "You may obtain a copy of the License at  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;http://www.apache.org/licenses/LICENSE-2.0  \n",
    "Unless required by applicable law or agreed to in writing, software  \n",
    "distributed under the License is distributed on an \"AS IS\" BASIS,  \n",
    "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  \n",
    "See the License for the specific language governing permissions and  \n",
    "limitations under the License. \n",
    "\n",
    "# MONAI Label - Lung Nodule Detection - 3D Slicer Integration\n",
    "\n",
    "***The Detection Task Workflow in MONAI Label***\n",
    "\n",
    "In this notebook, we provide a monaibundle app usecase example with lung nodule detection. The first detection task that can be used within MONAI Label.\n",
    "\n",
    "3D Slicer as the client viewer, we show how MONAI Label workflow serves as an interactive AI-Assisted tool for box-based ROI detection labeling.\n",
    "\n",
    "**Lung Nodule Detection Bundle:**\n",
    "\n",
    "![workflow](./figures/monailabel_detection_3dslicer/3dslicer_demo.png)\n",
    "\n",
    "\n",
    "**Method:** Lung Nodule Detection Networks\n",
    "\n",
    "The workflow of lung nodule detection with CT scans is from RetinaNet (Lin, Tsung-Yi, et al. \"Focal loss for dense object detection.\" ICCV 2017. https://arxiv.org/abs/1708.02002).\n",
    "\n",
    "![workflow](./figures/monailabel_detection_3dslicer/detection_networks.png)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup environment\n",
    "\n",
    "MONAI Label is pre-built within the MONAI Toolkit. Download monaibundle app\n",
    "\n",
    "### Prerequisites\n",
    "- **Install MONAI Label**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install monailabel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Get latest MONAI Label Slicer plugin** (Optional)\n",
    "\n",
    "If you are using older release of 3D Slicer MONAI Label plugin, you can update to the latest plugin which supports ***drawing ROI nodes from MONAI Label server prediction***.\n",
    "\n",
    "Or to use the developer mode of MONAI Label plugin by:\n",
    "\n",
    " - Pull latest MONAI Label repo:\n",
    " ```bash\n",
    " git clone https://github.com/Project-MONAI/MONAILabel.git\n",
    " ```\n",
    " - Go to `MONAILabel/plugins/slicer/`\n",
    " - Follow the Readme page to use developer mode of Slicer plugin.\n",
    " https://github.com/Project-MONAI/MONAILabel/tree/main/plugins/slicer#install-plugin-in-developer-mode\n",
    "    - Install Plugin in Developer Mode\n",
    "        - Open 3D Slicer: Go to Edit -> Application Settings -> Modules -> Additional Module Paths\n",
    "        - Add New Module Path: <FULL_PATH>/plugins/slicer/MONAILabel\n",
    "        - Restart 3D Slicer\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Download monaibundle app\n",
    "Sample-app with built-in versions are in the Toolkit, pull the monaibundle app using monailabel API "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using PYTHONPATH=/home/yucheng:\n",
      "\n",
      "monaibundle is copied at: /home/yucheng/yucheng/2023/detection_totalsegment_tutorial/tutorials/monailabel/apps/monaibundle\n"
     ]
    }
   ],
   "source": [
    "!monailabel apps --download --name monaibundle --output apps"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Download sample data\n",
    "The lung nodule detection model is trained and evaluated using [LUNA16](https://luna16.grand-challenge.org/Home/) dataset.\n",
    "\n",
    "LUNA16 is a public dataset of CT lung nodule detection. Using raw CT scans, the goal is to identify locations of possible nodules and to assign a probability for being a nodule to each location.\n",
    "\n",
    "Disclaimer: We are not the host of the data. Please make sure to read the requirements and usage policies of the data and give credit to the authors of the dataset! We acknowledge the National Cancer Institute and the Foundation for the National Institutes of Health, and their critical role in the creation of the free publicly available LIDC/IDRI Database used in this study.\n",
    "\n",
    " - If you are going to use full dataset of LUNA16, please refer to the dataset link, download the data, create and preprocess the images following [this page](        https://github.com/Project-MONAI/model-zoo/tree/dev/models/lung_nodule_ct_detection#data \n",
    ").\n",
    " \n",
    " - In this tutorial, we prepared a sample subset, resampled and ready to use. The subset is only for demonstration. Download [here](https://github.com/Project-MONAI/MONAILabel/releases/download/data/detection_dataset.zip).\n",
    " \n",
    "   Or use these commands to download and extract directly.\n",
    "\n",
    "```bash\n",
    "mkdir datasets\n",
    "wget \"https://github.com/Project-MONAI/MONAILabel/releases/download/data/detection_dataset.zip\" -O datasets/detection_dataset.zip\n",
    "unzip datasets/detection_dataset.zip -d datasets/detection_dataset\n",
    "```\n",
    " \n",
    " - The detection model also supports MSD Task06_Lung dataset for inference. Note this dataset is created for nodule segmentation task. We do not use the segmentation mask, only the images for inference. Use following monailabel API to automatically download the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!monailabel datasets --download --name Task06_Lung --output datasets"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Starting MONAI Label Server\n",
    "\n",
    "Specify the bundle name in `--conf models <BUNDLENAME>` argument. For example: lung_nodule_ct_detection\n",
    "\n",
    "**Note:** If you are running a MONAI Label server in this notebook and plan to close the notebook, ensure that you terminate the kernel first. Failing to do so might result in an issue where the port remains occupied from the previous session when you try to restart the server later. To prevent this, always terminate the kernel before closing the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!monailabel start_server --app apps/monaibundle --studies datasets/Task06_Lung/imagesTr --conf models lung_nodule_ct_detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 (Optional) Modify Detection Parameters in the Bundle\n",
    "\n",
    "The lung_nodule_ct_detection bundle can set box prediction probability threshold. In `apps/monaibundle/model/lung_nodule_ct_detection_v0.5.0/configs/inference.json`, it's define here `score_thresh=0.02`: \n",
    "\n",
    "```json\n",
    "    \"detector_ops\": [\n",
    "        \"$@detector.set_target_keys(box_key='box', label_key='label')\",\n",
    "        \"$@detector.set_box_selector_parameters(score_thresh=0.02,topk_candidates_per_level=1000,nms_thresh=0.22,detections_per_img=300)\",\n",
    "        \"$@detector.set_sliding_window_inferer(roi_size=@infer_patch_size,overlap=0.25,sw_batch_size=1,mode='constant',device='cpu')\"\n",
    "    ],\n",
    "\n",
    "```\n",
    "\n",
    "This parameter will impact the number of boxes in the final prediction output. Then show all boxes ROIs in 3D Slicer.\n",
    "\n",
    "If users want to limit the number of predicted boxes showing on 3D Slicer, users can set higher `score_thresh`, e.g., `score_thresh=0.5`, `score_thresh=0.6`, or even higher. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Auto Box Detection with 3D Slicer\n",
    "\n",
    "Below is a step-by-step tutorial on using 3D Slicer for auto lung nodule detection. Pre-trained model is provided by the bundle and loaded by MONAI Label server.\n",
    "\n",
    "\n",
    "**Start 3D Slicer and MONAI Label Plugin**: the tutorial starts with monaibundle app in the 3D Slicer, please refer to the [\"Hello World\" tutorial](monailabel_HelloWorld_radiology_3dslicer.ipynb) for detailed instructions on installing 3D Slicer and MONAI Label plugin.\n",
    "\n",
    "**Important**: If your MONAI Label plugin is from older release, please refer to [Prerequisites](#Prerequisites) in this notebook to update and use latest MONAI Label plugin. The detection task needs to draw ROI nodes to 3D Slicer markups.\n",
    "\n",
    "### 2.1 Select Model and Fetch Data\n",
    "\n",
    "- Select the bundle model loaded with MONAI Label server, **lung_nodule_ct_detection** in this tutorial. Start MONAI Label server will automatically download the bundle.\n",
    "\n",
    "\n",
    "click **next sample** to load image data from datastore.\n",
    "\n",
    "![slicer1](./figures/monailabel_detection_3dslicer/detection1.png)\n",
    "\n",
    "\n",
    "### 2.2 Run Auto Inference\n",
    "\n",
    "- MONAI Label has loaded the pre-trained weights within the bundle for multi-organ segmentation task, click **run** to do inference on the currently loaded data. \n",
    "\n",
    "- Note: the pre-trained models are in the bundle's \"**models**\" folder. For instance, in this usecase, a downloaded pre-trained model is saved at \"**apps/monaibundle/model/lung_nodule_ct_detection/models/model.pt**\".\n",
    "\n",
    "- Note: the detection model uses RetinaNet, please reserve at least ~12G memory for inference and training, modify the bundle config file if out of memory.\n",
    "- Users can move the axis and slices to view the correct location of segmentation.\n",
    "\n",
    "Users can monitor the logs in the MONAI Label server terminal\n",
    "\n",
    "![slicer1](./figures/monailabel_detection_3dslicer/detection2.png)\n",
    "\n",
    "\n",
    "### 2.3 Edit Annotations and Submit Label\n",
    "\n",
    "- Users can edit the inference labels manually using 3D Slicer ROI box Editor, click the **Markups** tab on the top, the tools for adjusting ROIs. Once done annotation, users can go back to MONAI Label plugin and click **Submit Label** to save the ground truth label to the file system.\n",
    "\n",
    "- Note: the final annotations will be saved to \"**labels/final**\" folder in the study dataset, for example, in this usecase, the ground truth label will be saved to \"**datasets/Task06_Lung/imagesTr/labels/final**\"\n",
    "\n",
    "- The saved annotation will be in JSON file with the 3D Slicer ROINode format. It can be loaded with 3D Slicer.\n",
    "\n",
    "![slicer1](./figures/monailabel_detection_3dslicer/detection3.png)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Active Learning with 3D Slicer\n",
    "\n",
    "Active learning and interactive fine-tuning models have highlighted features in MONAI Label, users can train their models anytime when new annotated labels are saved. Click **Train** button, MONAI Label server will fetch the saved final ground truth label and fine-tune the prior model. \n",
    "\n",
    "- Sample of annotated box coordinate in JSON file. \n",
    "\n",
    "![slicer1](./figures/monailabel_detection_3dslicer/detection4.png)\n",
    "\n",
    "\n",
    "Users can monitor the logs in the MONAI Label server terminal\n",
    "\n",
    "- The ROINode box annotation will be converted to the model-compatible ground truth label automatically.\n",
    "\n",
    "![slicer1](./figures/monailabel_detection_3dslicer/detection5.png)\n",
    "\n",
    "Same to the Radiology app, users can do auto segmentation using the latest fine-tuned model. \n",
    "The active learning process will select unlabeled batch data.\n",
    "\n",
    "The next image is selected, already trained images will be marked as labeled data, then won't be selected for the next labeling batch. Active learning strategies such as \"first/random\" will be used for selecting which unlabeled data to fetch. \n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Repeat Until All Data are Annotated and Trained \n",
    "\n",
    "Repeated fetch data and active learning process to section 2.2, until all unlabeled data are annotated and trained. \n",
    "\n",
    "For each training loop, the new best metric model will be saved in \"**model/model.pt**\", in this usecase, \"**apps/monaibundle/lung_nodule_ct_detection/model/model.pt**\" is saved."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Conclusion\n",
    "\n",
    "This tutorial demonstrates MONAI Label usage with monaibundle app, the lung nodule ct detection task is introduced. Active learning process, auto inference, detection editor, submit labels, and save models are included with the bundle. The monaibundle app highlights the robust model deployment for MONAI Label."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "vscode": {
   "interpreter": {
    "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}