Jpcbertoldo/mvtec ad loco 2

.ci/README.md

@@ -0,0 +1,61 @@
+# Guide to Setting up the CI using the Docker images
+
+## Steps
+
+1. Build the docker image using the Dockerfile in the .ci directory.
+   Make sure you are in the root directory of `anomalib`.
+
+   ```bash
+   sudo docker build --build-arg HTTP_PROXY="$http_proxy" --build-arg \
+   HTTPS_PROXY="$https_proxy" --build-arg NO_PROXY="$no_proxy" \
+   . -t anomalib-ci -f .ci/cuda11.4.Dockerfile
+   ```
+
+   Here, `anomalib-ci` is the name of the image.
+
+1. Create and start a container
+
+   ```bash
+   sudo docker run --gpus all \
+   --shm-size=256M\
+    -i -t --mount type=bind,source=<path-to-datasets>,target=/home/user/datasets,readonly\
+    -d --name anomalib-ci-container anomalib-ci
+   ```
+
+   Note: `--gpus all` is required for the container to have access to the GPUs.
+   `-d` flag ensure that the container is detached when it is created.
+   `mount` is required to ensure that tests have access to the dataset.
+
+1. Enter the container by
+
+   ```bash
+   sudo docker exec -it  anomalib-ci-container /bin/bash
+   ```
+
+1. Install github actions runner in the container by navigating to [https://github.com/openvinotoolkit/anomalib/settings/actions/runners/new](https://github.com/openvinotoolkit/anomalib/settings/actions/runners/new)
+
+   For example:
+
+   ```bash
+   mkdir actions-runner && cd actions-runner
+
+   curl -o actions-runner-linux-x64-2.296.1.tar.gz -L https://github.com/actions/runner/releases/download/v2.296.1/actions-runner-linux-x64-2.296.1.tar.gz
+
+   tar xzf ./actions-runner-linux-x64-2.296.1.tar.gz
+
+   rm actions-runner-linux-x64-2.296.1.tar.gz
+
+   ./config.sh --url https://github.com/openvinotoolkit/anomalib --token <enter-your-token-here>
+   ```
+
+   Follow the instructions on the screen to complete the installation.
+
+1. Now the container is ready. Type `exit` to leave the container.
+
+1. Start github actions runner in detached mode in the container and set the
+   codacy token and the anomalib dataset environment variables.
+
+   ```bash
+   sudo docker exec -d anomalib-ci-container /bin/bash -c \
+   "export ANOMALIB_DATASET_PATH=/home/user/datasets;export CODACY_PROJECT_TOKEN=<codacy-project-token> && /home/user/actions-runner/run.sh"
+   ```

.ci/cuda10.2.Dockerfile

@@ -0,0 +1,75 @@
+#########################################################
+## Python Environment with CUDA
+#########################################################
+ARG HTTP_PROXY
+ARG HTTPS_PROXY
+ARG NO_PROXY
+
+FROM nvidia/cuda:10.2-devel-ubuntu18.04 AS python_base_cuda10.2
+LABEL maintainer="Anomalib Development Team"
+
+# Setup proxies
+
+ENV http_proxy=$HTTP_PROXY
+ENV https_proxy=$HTTPS_PROXY
+ENV no_proxy=$NO_PROXY
+ENV DEBIAN_FRONTEND="noninteractive"
+
+# Update system and install wget
+RUN apt-get update && \
+    apt-get install --no-install-recommends -y \
+        curl=7.58.0-2ubuntu3.20 \
+        wget=1.19.4-1ubuntu2 \
+        ffmpeg=7:3.4.2-2 \
+        libpython3.8=3.8.0-3ubuntu1~18.04.2 \
+        npm=3.5.2-0ubuntu4 \
+        pandoc=1.19.2.4~dfsg-1build4 \
+        ruby=1:2.5.1 \
+        software-properties-common=0.96.24.32.18 && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
+# Install latest git for github actions
+RUN add-apt-repository ppa:git-core/ppa &&\
+    apt-get update && \
+    apt-get install --no-install-recommends -y git=1:2.38.1-0ppa1~ubuntu18.04.1 &&\
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
+# Prettier requires atleast nodejs 10
+RUN curl -sL https://deb.nodesource.com/setup_14.x > nodesetup.sh && \
+    bash - nodesetup.sh && \
+    apt-get install --no-install-recommends -y nodejs=14.20.1-1nodesource1 && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
+# Create a non-root user
+RUN useradd -m user
+USER user
+
+# Install Conda
+RUN curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh > ~/miniconda.sh -s && \
+    bash ~/miniconda.sh -b -p /home/user/conda && \
+    rm ~/miniconda.sh
+ENV PATH "/home/user/conda/bin:${PATH}"
+RUN conda install python=3.8
+
+
+#########################################################
+## Anomalib Development Env
+#########################################################
+
+FROM python_base_cuda10.2 as anomalib_development_env
+
+# Install all anomalib requirements
+COPY ./requirements/base.txt /tmp/anomalib/requirements/base.txt
+RUN pip install --no-cache-dir -r /tmp/anomalib/requirements/base.txt
+
+COPY ./requirements/openvino.txt /tmp/anomalib/requirements/openvino.txt
+RUN pip install --no-cache-dir -r /tmp/anomalib/requirements/openvino.txt
+
+# Install other requirements related to development
+COPY ./requirements/dev.txt /tmp/anomalib/requirements/dev.txt
+RUN pip install --no-cache-dir -r /tmp/anomalib/requirements/dev.txt
+
+WORKDIR /home/user

Dockerfile → .ci/cuda11.4.Dockerfile

@@ -1,34 +1,59 @@
 #########################################################
 ## Python Environment with CUDA
 #########################################################
+ARG HTTP_PROXY
+ARG HTTPS_PROXY
+ARG NO_PROXY
 
-FROM nvidia/cuda:11.4.0-devel-ubuntu20.04 AS python_base_cuda
-LABEL MAINTAINER="Anomalib Development Team"
+FROM nvidia/cuda:11.4.0-devel-ubuntu20.04 AS python_base_cuda11.4
+LABEL maintainer="Anomalib Development Team"
+
+# Setup proxies
+
+ENV http_proxy=$HTTP_PROXY
+ENV https_proxy=$HTTPS_PROXY
+ENV no_proxy=$NO_PROXY
+ENV DEBIAN_FRONTEND="noninteractive"
 
 # Update system and install wget
 RUN apt-get update && \
-    DEBIAN_FRONTEND="noninteractive" apt-get install --no-install-recommends -y \
+    apt-get install --no-install-recommends -y \
+        curl=7.68.0-1ubuntu2.13 \
         wget=1.20.3-1ubuntu2 \
         ffmpeg=7:4.2.7-0ubuntu0.1 \
         libpython3.8=3.8.10-0ubuntu1~20.04.5 \
-        git=1:2.25.1-1ubuntu3.5 \
-        sudo=1.8.31-1ubuntu1.2 && \
+        nodejs=10.19.0~dfsg-3ubuntu1 \
+        npm=6.14.4+ds-1ubuntu2 \
+        pandoc=2.5-3build2 \
+        ruby=1:2.7+1 \
+        software-properties-common=0.99.9.8 && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
+# Install latest git for github actions
+RUN add-apt-repository ppa:git-core/ppa &&\
+    apt-get update && \
+    apt-get install --no-install-recommends -y git=1:2.38.1-0ppa1~ubuntu20.04.1 &&\
     apt-get clean && \
     rm -rf /var/lib/apt/lists/*
 
+# Create a non-root user
+RUN useradd -m user
+USER user
+
 # Install Conda
-RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh --quiet && \
-    bash ~/miniconda.sh -b -p /opt/conda && \
+RUN curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh > ~/miniconda.sh -s && \
+    bash ~/miniconda.sh -b -p /home/user/conda && \
     rm ~/miniconda.sh
-ENV PATH "/opt/conda/bin:${PATH}"
+ENV PATH "/home/user/conda/bin:${PATH}"
 RUN conda install python=3.8
 
 
 #########################################################
 ## Anomalib Development Env
 #########################################################
 
-FROM python_base_cuda as anomalib_development_env
+FROM python_base_cuda11.4 as anomalib_development_env
 
 # Install all anomalib requirements
 COPY ./requirements/base.txt /tmp/anomalib/requirements/base.txt
@@ -38,17 +63,7 @@ COPY ./requirements/openvino.txt /tmp/anomalib/requirements/openvino.txt
 RUN pip install --no-cache-dir -r /tmp/anomalib/requirements/openvino.txt
 
 # Install other requirements related to development
-RUN apt-get update && \
-    DEBIAN_FRONTEND="noninteractive" apt-get install --no-install-recommends -y \
-        nodejs=10.19.0~dfsg-3ubuntu1 \
-        npm=6.14.4+ds-1ubuntu2 \
-        ruby=1:2.7+1 && \
-    apt-get clean && \
-    rm -rf /var/lib/apt/lists/*
 COPY ./requirements/dev.txt /tmp/anomalib/requirements/dev.txt
 RUN pip install --no-cache-dir -r /tmp/anomalib/requirements/dev.txt
 
-# Install anomalib
-COPY . /anomalib
-WORKDIR /anomalib
-RUN pip install --no-cache-dir -e .
+WORKDIR /home/user

.devcontainer/devcontainer.json

@@ -5,10 +5,10 @@
   // Sets the run context to one level up instead of the .devcontainer folder.
   "context": "..",
   // Update the 'dockerFile' property if you aren't using the standard 'Dockerfile' filename.
-  "dockerFile": "../Dockerfile",
+  "dockerFile": "../.ci/cuda11.4.Dockerfile",
   // Set *default* container specific settings.json values on container create.
   "settings": {
-    "python.pythonPath": "/opt/conda/bin/python",
+    "python.pythonPath": "/home/user/conda/bin/python",
     "python.formatting.provider": "black",
     "editor.formatOnSave": true,
     "python.formatting.blackArgs": ["--line-length", "120"],

.dockerignore

@@ -1,4 +1,5 @@
 *.dot
+.git
 .idea
 .pytest_cache
 .tox

.github/workflows/nightly.yml

@@ -12,16 +12,12 @@ jobs:
       max-parallel: 1
     if: github.ref == 'refs/heads/main'
     steps:
-      - name: Print GPU status
-        run: nvidia-smi
       - name: CHECKOUT REPOSITORY
         uses: actions/checkout@v2
       - name: Install Tox
         run: pip install tox
       - name: Coverage
-        run: |
-          export ANOMALIB_DATASET_PATH=/media/data1/datasets/
-          tox -e nightly
+        run: tox -e nightly
       - name: Upload coverage result
         uses: actions/upload-artifact@v2
         with:

.github/workflows/pre_merge.yml

@@ -7,24 +7,27 @@ on:
   workflow_dispatch: # run on request (no need for PR)
 
 jobs:
+  Code-Quality-Checks:
+    runs-on: [self-hosted, linux, x64]
+    steps:
+      - name: CHECKOUT REPOSITORY
+        uses: actions/checkout@v2
+      - name: Install Tox
+        run: pip install tox
+      - name: Code quality checks
+        run: tox -e pre-commit
   Tox:
     runs-on: [self-hosted, linux, x64]
+    needs: Code-Quality-Checks
     strategy:
       max-parallel: 1
     steps:
-      - name: Print GPU status
-        run: nvidia-smi
       - name: CHECKOUT REPOSITORY
         uses: actions/checkout@v2
       - name: Install Tox
         run: pip install tox
-      - name: Code quality checks
-        run: tox -e pre-commit
       - name: Coverage
-        run: |
-          export ANOMALIB_DATASET_PATH=/media/data1/datasets/
-          export CUDA_VISIBLE_DEVICES=3
-          tox -e pre_merge
+        run: tox -e pre_merge
       - name: Upload coverage report
         run: |
           # If the workflow is triggered from PR then it gets the commit id from the PR.

.pre-commit-config.yaml

@@ -82,3 +82,10 @@ repos:
     rev: 2.1.6
     hooks:
       - id: markdownlint
+
+  - repo: https://github.com/AleksaC/hadolint-py
+    rev: v2.10.0
+    hooks:
+      - id: hadolint
+        name: Lint Dockerfiles
+        description: Runs hadolint to lint Dockerfiles

README.md

@@ -78,15 +78,15 @@ pip install -e .
 
 ## ⚠️ Anomalib < v.0.4.0
 
-By default [`python tools/train.py`](https://gitlab-icv.inn.intel.com/algo_rnd_team/anomaly/-/blob/main/train.py)
+By default [`python tools/train.py`](https://github.com/openvinotoolkit/anomalib/blob/main/tools/train.py)
 runs [PADIM](https://arxiv.org/abs/2011.08785) model on `leather` category from the [MVTec AD](https://www.mvtec.com/company/research/datasets/mvtec-ad) [(CC BY-NC-SA 4.0)](https://creativecommons.org/licenses/by-nc-sa/4.0/) dataset.
 
 ```bash
 python tools/train.py    # Train PADIM on MVTec AD leather
 ```
 
 Training a model on a specific dataset and category requires further configuration. Each model has its own configuration
-file, [`config.yaml`](https://gitlab-icv.inn.intel.com/algo_rnd_team/anomaly/-/blob/main/padim/anomalib/models/padim/config.yaml)
+file, [`config.yaml`](https://github.com/openvinotoolkit/anomalib/blob/main/configs/model/padim.yaml)
 , which contains data, model and training configurable parameters. To train a specific model on a specific dataset and
 category, the config file is to be provided:
 
@@ -117,15 +117,38 @@ where the currently available models are:
 - [STFPM](anomalib/models/stfpm)
 - [GANomaly](anomalib/models/ganomaly)
 
-## Exporting Model to ONNX or OpenVINO IR
+## Feature extraction & (pre-trained) backbones
 
-It is possible to export your model to ONNX or OpenVINO IR
+The pre-trained backbones come from [PyTorch Image Models (timm)](https://github.com/rwightman/pytorch-image-models), which are wrapped by `FeatureExtractor`.
 
-If you want to export your PyTorch model to an OpenVINO model, ensure that `export_mode` is set to `"openvino"` in the respective model `config.yaml`.
+For more information, please check our documentation or the [section about feature extraction in "Getting Started with PyTorch Image Models (timm): A Practitioner’s Guide"](https://towardsdatascience.com/getting-started-with-pytorch-image-models-timm-a-practitioners-guide-4e77b4bf9055#b83b:~:text=ready%20to%20train!-,Feature%20Extraction,-timm%20models%20also>).
+
+Tips:
+
+- Papers With Code has an interface to easily browse models available in timm: [https://paperswithcode.com/lib/timm](https://paperswithcode.com/lib/timm)
+
+- You can also find them with the function `timm.list_models("resnet*", pretrained=True)`
+
+The backbone can be set in the config file, two examples below.
+
+Anomalib < v.0.4.0
 
 ```yaml
-optimization:
-  export_mode: "openvino" # options: openvino, onnx
+model:
+  name: cflow
+  backbone: wide_resnet50_2
+  pre_trained: true
+Anomalib > v.0.4.0 Beta - Subject to Change
+```
+
+Anomalib >= v.0.4.0
+
+```yaml
+model:
+  class_path: anomalib.models.Cflow
+  init_args:
+    backbone: wide_resnet50_2
+    pre_trained: true
 ```
 
 ## Custom Dataset
@@ -222,6 +245,17 @@ python tools/inference/gradio_inference.py \
         --weights ./results/padim/mvtec/bottle/weights/model.ckpt
 ```
 
+## Exporting Model to ONNX or OpenVINO IR
+
+It is possible to export your model to ONNX or OpenVINO IR
+
+If you want to export your PyTorch model to an OpenVINO model, ensure that `export_mode` is set to `"openvino"` in the respective model `config.yaml`.
+
+```yaml
+optimization:
+  export_mode: "openvino" # options: openvino, onnx
+```
+
 # Hyperparameter Optimization
 
 To run hyperparameter optimization, use the following command: