Migrating 3D image classification from CT scans example to Keras 3 #1725

sitamgithub-MSIT · 2024-01-11T18:04:27Z

This PR changes the 3D image classification from CT scans example to keras 3.0 [TF-Only Example] as requested in keras-team/keras-cv#2211.

For example, here is the notebook link provided:
https://colab.research.google.com/drive/1CLhjLZiidFB7qPIDH5a7JGls-Wfqe7v5?usp=sharing

cc: @divyashreepathihalli @fchollet

The following describes the Git difference for the changed files:

Changes:

index d69680df..2987dc96 100644
--- a/examples/vision/3D_image_classification.py
+++ b/examples/vision/3D_image_classification.py
@@ -2,10 +2,9 @@
 Title: 3D image classification from CT scans
 Author: [Hasib Zunair](https://twitter.com/hasibzunair)
 Date created: 2020/09/23
-Last modified: 2024/01/11
+Last modified: 2020/09/23
 Description: Train a 3D convolutional neural network to predict presence of pneumonia.
 Accelerator: GPU
-Converted to Keras 3 by: [Sitam Meur](https://github.com/sitamgithub-MSIT)
 """
 """
 ## Introduction
@@ -28,15 +27,13 @@ equivalent: it takes as input a 3D volume or a sequence of 2D frames (e.g. slice
 """
 
 import os
-os.environ["KERAS_BACKEND"] = "tensorflow"
 import zipfile
 import numpy as np
-import keras
-from keras import ops
-from keras import layers
-
 import tensorflow as tf
 
+from tensorflow import keras
+from tensorflow.keras import layers
+
 """
 ## Downloading the MosMedData: Chest CT Scans with COVID-19 Related Findings
 
@@ -213,28 +210,37 @@ import random
 
 from scipy import ndimage
 
-random_rotation = layers.RandomRotation(factor=(-0.06, 0.06))
-
 
+@tf.function
 def rotate(volume):
     """Rotate the volume by a few degrees"""
-    # rotate volume
-    volume = random_rotation(volume)
-    volume = ops.clip(volume, 0, 1)
-    return volume
+
+    def scipy_rotate(volume):
+        # define some rotation angles
+        angles = [-20, -10, -5, 5, 10, 20]
+        # pick angles at random
+        angle = random.choice(angles)
+        # rotate volume
+        volume = ndimage.rotate(volume, angle, reshape=False)
+        volume[volume < 0] = 0
+        volume[volume > 1] = 1
+        return volume
+
+    augmented_volume = tf.numpy_function(scipy_rotate, [volume], tf.float32)
+    return augmented_volume
 
 
 def train_preprocessing(volume, label):
     """Process training data by rotating and adding a channel."""
     # Rotate volume
     volume = rotate(volume)
-    volume = ops.expand_dims(volume, axis=3)
+    volume = tf.expand_dims(volume, axis=3)
     return volume, label
 
 
 def validation_preprocessing(volume, label):
     """Process validation data by only adding a channel."""
-    volume = ops.expand_dims(volume, axis=3)
+    volume = tf.expand_dims(volume, axis=3)
     return volume, label
 
 
@@ -373,7 +379,7 @@ model.compile(
 
 # Define callbacks.
 checkpoint_cb = keras.callbacks.ModelCheckpoint(
-    "3d_image_classification.keras", save_best_only=True
+    "3d_image_classification.h5", save_best_only=True
 )
 early_stopping_cb = keras.callbacks.EarlyStopping(monitor="val_acc", patience=15)
 
@@ -420,7 +426,7 @@ for i, metric in enumerate(["acc", "loss"]):
 """
 
 # Load best weights.
-model.load_weights("3d_image_classification.keras")
+model.load_weights("3d_image_classification.h5")
 prediction = model.predict(np.expand_dims(x_val[0], axis=0))[0]
 scores = [1 - prediction[0], prediction[0]]
 
diff --git a/examples/vision/ipynb/3D_image_classification.ipynb b/examples/vision/ipynb/3D_image_classification.ipynb
index 625bddb9..a2ecf827 100644
--- a/examples/vision/ipynb/3D_image_classification.ipynb
+++ b/examples/vision/ipynb/3D_image_classification.ipynb
@@ -10,7 +10,7 @@
     "\n",
     "**Author:** [Hasib Zunair](https://twitter.com/hasibzunair)<br>\n",
     "**Date created:** 2020/09/23<br>\n",
-    "**Last modified:** 2024/01/11<br>\n",
+    "**Last modified:** 2020/09/23<br>\n",
     "**Description:** Train a 3D convolutional neural network to predict presence of pneumonia."
    ]
   },
@@ -47,21 +47,19 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 0,
    "metadata": {
     "colab_type": "code"
    },
    "outputs": [],
    "source": [
     "import os\n",
-    "os.environ[\"KERAS_BACKEND\"] = \"tensorflow\"\n",
     "import zipfile\n",
     "import numpy as np\n",
-    "import keras\n",
-    "from keras import ops\n",
-    "from keras import layers\n",
+    "import tensorflow as tf\n",
     "\n",
-    "import tensorflow as tf"
+    "from tensorflow import keras\n",
+    "from tensorflow.keras import layers"
    ]
   },
   {
@@ -83,7 +81,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 0,
    "metadata": {
     "colab_type": "code"
    },
@@ -137,7 +135,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 0,
    "metadata": {
     "colab_type": "code"
    },
@@ -201,7 +199,8 @@
     "    volume = normalize(volume)\n",
     "    # Resize width, height and depth\n",
     "    volume = resize_volume(volume)\n",
-    "    return volume\n"
+    "    return volume\n",
+    ""
    ]
   },
   {
@@ -215,7 +214,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 0,
    "metadata": {
     "colab_type": "code"
    },
@@ -235,7 +234,8 @@
     "]\n",
     "\n",
     "print(\"CT scans with normal lung tissue: \" + str(len(normal_scan_paths)))\n",
-    "print(\"CT scans with abnormal lung tissue: \" + str(len(abnormal_scan_paths)))\n"
+    "print(\"CT scans with abnormal lung tissue: \" + str(len(abnormal_scan_paths)))\n",
+    ""
    ]
   },
   {
@@ -252,7 +252,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 0,
    "metadata": {
     "colab_type": "code"
    },
@@ -297,7 +297,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 0,
    "metadata": {
     "colab_type": "code"
    },
@@ -307,29 +307,39 @@
     "\n",
     "from scipy import ndimage\n",
     "\n",
-    "random_rotation = layers.RandomRotation(factor=(-0.06, 0.06))\n",
-    "\n",
     "\n",
+    "@tf.function\n",
     "def rotate(volume):\n",
     "    \"\"\"Rotate the volume by a few degrees\"\"\"\n",
-    "    # rotate volume\n",
-    "    volume = random_rotation(volume)\n",
-    "    volume = ops.clip(volume, 0, 1)\n",
-    "    return volume\n",
+    "\n",
+    "    def scipy_rotate(volume):\n",
+    "        # define some rotation angles\n",
+    "        angles = [-20, -10, -5, 5, 10, 20]\n",
+    "        # pick angles at random\n",
+    "        angle = random.choice(angles)\n",
+    "        # rotate volume\n",
+    "        volume = ndimage.rotate(volume, angle, reshape=False)\n",
+    "        volume[volume < 0] = 0\n",
+    "        volume[volume > 1] = 1\n",
+    "        return volume\n",
+    "\n",
+    "    augmented_volume = tf.numpy_function(scipy_rotate, [volume], tf.float32)\n",
+    "    return augmented_volume\n",
     "\n",
     "\n",
     "def train_preprocessing(volume, label):\n",
     "    \"\"\"Process training data by rotating and adding a channel.\"\"\"\n",
     "    # Rotate volume\n",
     "    volume = rotate(volume)\n",
-    "    volume = ops.expand_dims(volume, axis=3)\n",
+    "    volume = tf.expand_dims(volume, axis=3)\n",
     "    return volume, label\n",
     "\n",
     "\n",
     "def validation_preprocessing(volume, label):\n",
     "    \"\"\"Process validation data by only adding a channel.\"\"\"\n",
-    "    volume = ops.expand_dims(volume, axis=3)\n",
-    "    return volume, label\n"
+    "    volume = tf.expand_dims(volume, axis=3)\n",
+    "    return volume, label\n",
+    ""
    ]
   },
   {
@@ -345,7 +355,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 0,
    "metadata": {
     "colab_type": "code"
    },
@@ -383,7 +393,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 0,
    "metadata": {
     "colab_type": "code"
    },
@@ -396,7 +406,8 @@
     "images = images.numpy()\n",
     "image = images[0]\n",
     "print(\"Dimension of the CT scan is:\", image.shape)\n",
-    "plt.imshow(np.squeeze(image[:, :, 30]), cmap=\"gray\")\n"
+    "plt.imshow(np.squeeze(image[:, :, 30]), cmap=\"gray\")\n",
+    ""
    ]
   },
   {
@@ -410,7 +421,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 0,
    "metadata": {
     "colab_type": "code"
    },
@@ -461,7 +472,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 0,
    "metadata": {
     "colab_type": "code"
    },
@@ -516,7 +527,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 0,
    "metadata": {
     "colab_type": "code"
    },
@@ -535,7 +546,7 @@
     "\n",
     "# Define callbacks.\n",
     "checkpoint_cb = keras.callbacks.ModelCheckpoint(\n",
-    "    \"3d_image_classification.keras\", save_best_only=True\n",
+    "    \"3d_image_classification.h5\", save_best_only=True\n",
     ")\n",
     "early_stopping_cb = keras.callbacks.EarlyStopping(monitor=\"val_acc\", patience=15)\n",
     "\n",
@@ -579,7 +590,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 0,
    "metadata": {
     "colab_type": "code"
    },
@@ -608,14 +619,14 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 0,
    "metadata": {
     "colab_type": "code"
    },
    "outputs": [],
    "source": [
     "# Load best weights.\n",
-    "model.load_weights(\"3d_image_classification.keras\")\n",
+    "model.load_weights(\"3d_image_classification.h5\")\n",
     "prediction = model.predict(np.expand_dims(x_val[0], axis=0))[0]\n",
     "scores = [1 - prediction[0], prediction[0]]\n",
     "\n",
@@ -657,4 +668,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 0
-}
+}
\ No newline at end of file
diff --git a/examples/vision/md/3D_image_classification.md b/examples/vision/md/3D_image_classification.md
index 454b4367..c2cc2a7d 100644
--- a/examples/vision/md/3D_image_classification.md
+++ b/examples/vision/md/3D_image_classification.md
@@ -2,7 +2,7 @@
 
 **Author:** [Hasib Zunair](https://twitter.com/hasibzunair)<br>
 **Date created:** 2020/09/23<br>
-**Last modified:** 2024/01/11<br>
+**Last modified:** 2020/09/23<br>
 **Description:** Train a 3D convolutional neural network to predict presence of pneumonia.
 
 
@@ -33,14 +33,12 @@ equivalent: it takes as input a 3D volume or a sequence of 2D frames (e.g. slice
 
 python
 import os
-os.environ["KERAS_BACKEND"] = "tensorflow"
 import zipfile
 import numpy as np
-import keras
-from keras import ops
-from keras import layers
-
 import tensorflow as tf
+
+from tensorflow import keras
+from tensorflow.keras import layers
 
 ---
@@ -249,28 +247,37 @@ import random
 
 from scipy import ndimage
 
-random_rotation = layers.RandomRotation(factor=(-0.06, 0.06))
-
 
+@tf.function
 def rotate(volume):
     """Rotate the volume by a few degrees"""
-    # rotate volume
-    volume = random_rotation(volume)
-    volume = ops.clip(volume, 0, 1)
-    return volume
+
+    def scipy_rotate(volume):
+        # define some rotation angles
+        angles = [-20, -10, -5, 5, 10, 20]
+        # pick angles at random
+        angle = random.choice(angles)
+        # rotate volume
+        volume = ndimage.rotate(volume, angle, reshape=False)
+        volume[volume < 0] = 0
+        volume[volume > 1] = 1
+        return volume
+
+    augmented_volume = tf.numpy_function(scipy_rotate, [volume], tf.float32)
+    return augmented_volume
 
 
 def train_preprocessing(volume, label):
     """Process training data by rotating and adding a channel."""
     # Rotate volume
     volume = rotate(volume)
-    volume = ops.expand_dims(volume, axis=3)
+    volume = tf.expand_dims(volume, axis=3)
     return volume, label
 
 
 def validation_preprocessing(volume, label):
     """Process validation data by only adding a channel."""
-    volume = ops.expand_dims(volume, axis=3)
+    volume = tf.expand_dims(volume, axis=3)
     return volume, label
 
@@ -478,7 +485,7 @@ model.compile(
 
 # Define callbacks.
 checkpoint_cb = keras.callbacks.ModelCheckpoint(
-    "3d_image_classification.keras", save_best_only=True
+    "3d_image_classification.h5", save_best_only=True
 )
 early_stopping_cb = keras.callbacks.EarlyStopping(monitor="val_acc", patience=15)
 
@@ -600,7 +607,7 @@ for i, metric in enumerate(["acc", "loss"]):
 
 python
 # Load best weights.
-model.load_weights("3d_image_classification.keras")
+model.load_weights("3d_image_classification.h5")
 prediction = model.predict(np.expand_dims(x_val[0], axis=0))[0]
 scores = [1 - prediction[0], prediction[0]]
 
(END)```

</details>

sitamgithub-MSIT · 2024-01-11T20:26:09Z

I tried changing the backend to "jax" but got an error as NotImplementedError: Cannot convert a symbolic tf.Tensor (Squeeze_1:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported. The error occurred at .map(train_preprocessing) part of the code

train_dataset = (
    train_loader.shuffle(len(x_train))
    .map(train_preprocessing)
    .batch(batch_size)
    .prefetch(AUTO)
)

So, I mentioned this as a TF-only example in the PR description.

sitamgithub-MSIT · 2024-01-11T20:33:14Z

Talking about tensorflow dependency, tf.data is only used in the updated example. The rest of the TF operations are handled with Keras.ops and worked successfully with the TensorFlow backend.

sitamgithub-MSIT · 2024-01-11T20:53:26Z

@tf.function
def rotate(volume):
    """Rotate the volume by a few degrees."""

    def scipy_rotate(volume):
        # define some rotation angles
        angles = [-20, -10, -5, 5, 10, 20]
        # pick angles at random
        angle = random.choice(angles)
        # rotate volume
        volume = ndimage.rotate(volume, angle, reshape=False)
        volume[volume < 0] = 0
        volume[volume > 1] = 1
        return volume

    augmented_volume = tf.numpy_function(scipy_rotate, [volume], tf.float32)
    return augmented_volume

This method is substituted in the updated code. Without removing this, this will create an error when running the example with the TF backend. Also, I read the numpy_function documentation here and decided to implement the rotate method without using numpy_function.

Now, coming to the implementation of the rotate method, the keras RandomRotation layer is used. The author proposed to take angles randomly from the set of angles angles = [-20, -10, -5, 5, 10, 20] and then perform rotation. With the random rotation layer, a factor of 0.06 is taken (precisely in the 21.6 degree range), and then the rotate method is performed with added clipping of 0 and 1. This implementation worked. Also, model performance, which was mostly similar, took 3 epochs less than the actual example. The accuracy is also the same, a little higher than the example. This is the only change that was written other than that Keras.ops handled the rest when needed.

sachinprasadhs · 2024-01-11T21:08:10Z

@sitamgithub-MSIT , Same PR is already created here #1723, could you please check it if there many changes to the actual PR.

sitamgithub-MSIT · 2024-01-11T22:22:46Z

@sitamgithub-MSIT , Same PR is already created here #1723, could you please check it if there many changes to the actual PR.

Yeah, I checked the mentioned PR, and it worked with Torch also, but the exact same code is giving an error with the TensorFlow backend. The error it is given is ValueError: Cannot iterate over a shape with unknown rank. Also, PR does not include Keras.ops.

divyashreepathihalli · 2024-01-11T22:43:03Z

@sitamgithub-MSIT, in addition to replacing tf ops, you might also want to replace the numpy ops with the corresponding keras ops.

sitamgithub-MSIT · 2024-01-11T22:51:32Z

@sitamgithub-MSIT, in addition to replacing tf ops, you might also want to replace the numpy ops with the corresponding keras ops.

Ok. I will do that in next commit then.

divyashreepathihalli · 2024-01-12T23:17:38Z

LGTM! Thank you!
Just an FYI, the inputs can still continue to be np.array, that would work fine with keras 3.

fchollet

LGTM, thanks!

fchollet · 2024-01-21T21:18:33Z

This branch has conflicts that must be resolved

Was the example already ported before?

sachinprasadhs · 2024-01-22T16:59:41Z

@sitamgithub-MSIT Looks like the other PR got merged here #1723.
If there is no significant changes from the above linked PR, you can close this.

sitamgithub-MSIT · 2024-01-22T18:07:35Z

@sitamgithub-MSIT , Same PR is already created here #1723, could you please check it if there many changes to the actual PR.

Yeah, I checked the mentioned PR, and it worked with Torch also, but the exact same code is giving an error with the TensorFlow backend. The error it is given is ValueError: Cannot iterate over a shape with unknown rank. Also, PR does not include Keras.ops.

@sachinprasadhs I mentioned the issue with other PR over here. One thing I can do is close this PR as the other PR gets merged so I can update my master branch of the forked repo and make another PR.

3d image classification migrated to keras3

6b589a1

github-actions bot assigned sachinprasadhs Jan 11, 2024

fixed some formatting related problems

b2e61ab

numpy ops replaced with keras ops

e275b3f

divyashreepathihalli approved these changes Jan 12, 2024

View reviewed changes

divyashreepathihalli requested a review from fchollet January 17, 2024 08:08

fchollet approved these changes Jan 21, 2024

View reviewed changes

fchollet closed this Jan 22, 2024

sitamgithub-MSIT deleted the 3d-classification branch November 21, 2024 18:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrating 3D image classification from CT scans example to Keras 3 #1725

Migrating 3D image classification from CT scans example to Keras 3 #1725

sitamgithub-MSIT commented Jan 11, 2024 •

edited by sachinprasadhs

Loading

sitamgithub-MSIT commented Jan 11, 2024 •

edited

Loading

sitamgithub-MSIT commented Jan 11, 2024

sitamgithub-MSIT commented Jan 11, 2024

sachinprasadhs commented Jan 11, 2024

sitamgithub-MSIT commented Jan 11, 2024

divyashreepathihalli commented Jan 11, 2024

sitamgithub-MSIT commented Jan 11, 2024

divyashreepathihalli commented Jan 12, 2024

fchollet left a comment

fchollet commented Jan 21, 2024

sachinprasadhs commented Jan 22, 2024

sitamgithub-MSIT commented Jan 22, 2024

Migrating 3D image classification from CT scans example to Keras 3 #1725

Migrating 3D image classification from CT scans example to Keras 3 #1725

Conversation

sitamgithub-MSIT commented Jan 11, 2024 • edited by sachinprasadhs Loading

sitamgithub-MSIT commented Jan 11, 2024 • edited Loading

sitamgithub-MSIT commented Jan 11, 2024

sitamgithub-MSIT commented Jan 11, 2024

sachinprasadhs commented Jan 11, 2024

sitamgithub-MSIT commented Jan 11, 2024

divyashreepathihalli commented Jan 11, 2024

sitamgithub-MSIT commented Jan 11, 2024

divyashreepathihalli commented Jan 12, 2024

fchollet left a comment

Choose a reason for hiding this comment

fchollet commented Jan 21, 2024

sachinprasadhs commented Jan 22, 2024

sitamgithub-MSIT commented Jan 22, 2024

sitamgithub-MSIT commented Jan 11, 2024 •

edited by sachinprasadhs

Loading

sitamgithub-MSIT commented Jan 11, 2024 •

edited

Loading