Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tiling notebook #712

Merged
merged 2 commits into from
Nov 18, 2022
Merged

Add tiling notebook #712

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4b75f19
Add tiling notebook
samet-akcay Nov 14, 2022
272daf8
Address the PR comments
samet-akcay Nov 18, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/source/how_to_guides/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ How to Guides
notebooks/100_datamodules/101_btech.ipynb
notebooks/100_datamodules/102_mvtec.ipynb
notebooks/100_datamodules/103_folder.ipynb
notebooks/100_datamodules/104_tiling.ipynb
notebooks/200_models/201_fastflow.ipynb
notebooks/300_benchmarking/301_benchmarking.ipynb
notebooks/300_benchmarking/302_hpo_wandb.ipynb
249 changes: 249 additions & 0 deletions notebooks/100_datamodules/104_tiling.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,249 @@
{
samet-akcay marked this conversation as resolved.
Show resolved Hide resolved
samet-akcay marked this conversation as resolved.
Show resolved Hide resolved
samet-akcay marked this conversation as resolved.
Show resolved Hide resolved
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Tiling Large Images\n",
"In the literature on deep learning, input image sizes typically range from 224 to 768 pixels. In the majority of industrial applications, however, input image sizes are significantly larger. Before the forward pass, these images are resized to a smaller scale before being fed into the models. However, this is problematic in the context of anomaly detection tasks, where the anomaly is typically quite small. The detection of abnormalities becomes extremely difficult when the image is shrunk in size. A common method for addressing this issue is to tile the input images so that no information is lost during the resizing operation.\n",
"\n",
"This notebook demonstrates how tiling works in anomalib, and how tiling can be enabled when training a model using anomalib."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import torch\n",
"from torchvision.io import read_image\n",
"from torchvision.transforms import Resize, ToPILImage\n",
"from torchvision.utils import draw_segmentation_masks, make_grid\n",
"\n",
"from anomalib.pre_processing.tiler import Tiler"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Effect of Resizing the Input Image "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Assuming that datasets directory is located in the root directory.\n",
"# image = read_image(path=\"../../datasets/MVTec/bottle/test/good/000.png\")\n",
"image = read_image(path=\"../../datasets/MVTec/transistor/test/cut_lead/000.png\")\n",
"mask = read_image(path=\"../../datasets/MVTec/transistor/ground_truth/cut_lead/000_mask.png\").bool()\n",
"overlayed_image = draw_segmentation_masks(image, mask, alpha=0.2, colors=[\"red\"])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"ToPILImage()(overlayed_image)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As can be seen above, the original MVTec image is 1024x1024. Most anomaly detection algorithms resize this input size into a smaller value such as 256x256. As mentioned above, this poses a problem for small anomalous regions as they become even smaller. Let's visualize this by resizing the input image into 256x256 image."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"resized_image = Resize((256, 256))(image)\n",
"resized_mask = Resize((256, 256))(mask)\n",
"resized_overlayed_image = draw_segmentation_masks(resized_image, resized_mask, alpha=0.5, colors=[\"red\"])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"ToPILImage()(resized_overlayed_image)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The figures above demonstrates how resizing affect the input image size. The original image is 1024x1024. The resized image is 256x256. The anomaly is now much smaller and is difficult to detect.\n",
"\n",
"### Effect of Tiling \n",
"To avoid losing information with resizing we could instead tile the image into multiple patches. We could utilize anomalib's `Tiler` object to do this. The `Tiler` object takes in the input image and the desired output size. It then tiles the image into patches of the desired size. The `Tiler` object also returns the coordinates of the patches in the original image. This is useful for reconstructing the tiled image back into the original image. Let's create a tiler object, tile the input image and visualize the tiled image."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"tiler = Tiler(tile_size=(256, 256), stride=256)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"where stride is the parameter defining the spatial distance in pixels between adjacent tiles, and tile size is the size of each tile (patches). Tile size and stride are here defined as (256, 256) and 256, respectively. As a result, we move 256 pixels to the right to take 256x256 patches. This will result in \"non-overlapping tiles\"."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"tiled_image = tiler.tile(image).type(torch.uint8)\n",
"tiled_mask = tiler.tile(mask).type(torch.bool)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(tiled_image.shape, tiled_mask.shape)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"ToPILImage()(make_grid(tiled_image, nrow=4, pad=5))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The image is tiled into non-overlapping patches as seen in the figure above. We have 16 patches in total because the original input size is 1024x1024 and the tile and stride sizes are (256, 256) and 25, respectively. As we can see, the fourteenth tile is the one that contains the anomalous region. We can visualize it by running the following:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"overlayed_tile = draw_segmentation_masks(tiled_image[13], tiled_mask[13], alpha=0.2, colors=[\"red\"])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"ToPILImage()(overlayed_tile)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We could also visualize the resized image and the tiled region by stacking them together. This would show how tiling could preserve the data that resizing operations would otherwise lose."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"ToPILImage()(torch.cat([resized_overlayed_image, overlayed_tile], dim=2))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Overall, tiling could be a useful feature when the input image size is large and the anomalous region size is small. Tiling the input could avoid information loss that would otherwise occur with resizing."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Tiling in Anomalib Training\n",
"This section demonstrates how tiling could be enabled in anomalib training. The dataset section in the configuration files needs to be changed in order to enable tiling in anomalib. The dataset contains a tiling section where tiling parameters are set. Below is an illustration of a tiling configuration;\n",
"\n",
"```yaml\n",
" tiling:\n",
" apply: false\n",
" tile_size: null\n",
" stride: null\n",
" remove_border_count: 0\n",
" use_random_tiling: False\n",
" random_tile_count: 16\n",
"```\n",
"For example, to train a dataset with tiling based on 256x256 non-overlapping patches, the following data configuration could be used:\n",
"\n",
"```yaml\n",
" tiling:\n",
" apply: true\n",
" tile_size: 256\n",
" stride: 256\n",
" remove_border_count: 0\n",
" use_random_tiling: False\n",
" random_tile_count: 16\n",
"```\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"depending on the use-case ,these tiling configurations could potentially improve the performance of the anomaly detection models."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.8.10",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
},
"orig_nbformat": 4,
"vscode": {
"interpreter": {
"hash": "ec7b890a7ad0a53be38594f25f679d3f514fc4eb4271d1db0284d66c33facb3f"
}
}
},
"nbformat": 4,
"nbformat_minor": 2
}