Lab 7 - Feature Descriptor.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "solved-effects",
   "metadata": {},
   "source": [
    "# 2110443 - Computer Vision (2020/2)\n",
    "## Lab 7 - Feature Descriptor\n",
    "In this lab, we will learn to use useful handcrafted features to detect objects and merge images in the provided images. This notebook includes both coding and written questions. Please hand in this notebook file with all outputs and your answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ordered-marks",
   "metadata": {},
   "outputs": [],
   "source": [
    "import cv2\n",
    "import math\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "modular-stress",
   "metadata": {},
   "source": [
    "## 1. Hough Transform\n",
    "Hough transform is a feature extraction technique used in computer vision, and digital image processing. The purpose of the technique is to find imperfect instances of objects within a certain class of shapes by a voting procedure. The classical Hough transform was concerned with the identification of lines in the image, but later the Hough transform has been extended to identifying positions of arbitrary shapes, most commonly circles or ellipses. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "consistent-projection",
   "metadata": {},
   "source": [
    "### 1.1 Hough Line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "postal-lebanon",
   "metadata": {},
   "outputs": [],
   "source": [
    "documentImage = cv2.imread('assets/Lab7-sudoku.jpg')\n",
    "documentImageGray = cv2.cvtColor(documentImage, cv2.COLOR_BGR2GRAY)\n",
    "documentImageGray = cv2.blur(documentImageGray, (3,3))\n",
    "plt.imshow(documentImage, cmap='gray')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "funny-sacramento",
   "metadata": {},
   "source": [
    "OpenCV already provide <a href=\"https://docs.opencv.org/4.5.1/dd/d1a/group__imgproc__feature.html#ga46b4e588934f6c8dfd509cc6e0e4545a\">cv2.HoughLine</a> (Standard Hough Transform) and <a href=\"https://docs.opencv.org/4.5.1/dd/d1a/group__imgproc__feature.html#ga8618180a5948286384e3b7ca02f6feeb\">cv2.HoughLinesP</a> (Probabilistic Hough Transform) is an optimization version of Hough Transform."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bridal-contributor",
   "metadata": {},
   "outputs": [],
   "source": [
    "edges = cv2.Canny(documentImageGray,50,100)\n",
    "plt.figure(figsize=(10,10))\n",
    "plt.imshow(edges, cmap='gray')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "greek-building",
   "metadata": {},
   "outputs": [],
   "source": [
    "linesP  = cv2.HoughLinesP(edges, 1, np.pi / 180, 50, None, 50, 10)\n",
    "\n",
    "# Draw the lines\n",
    "if linesP is not None:\n",
    "    for i in range(0, len(linesP)):\n",
    "        l = linesP[i][0]\n",
    "        cv2.line(documentImage, (l[0], l[1]), (l[2], l[3]), (0,0,255), 1, cv2.LINE_AA)\n",
    "        \n",
    "        \n",
    "plt.figure(figsize=(10,10))\n",
    "plt.imshow(documentImage, cmap='gray')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "loved-relationship",
   "metadata": {},
   "source": [
    "### 1.2 Hough Circle\n",
    "OpenCV provided a circle detection method slightly trickier than the standard Hough Transform. The Hough gradient method, which is made up of two main stages. The first stage involves edge detection and finding the possible circle centers and the second stage finds the best radius for each candidate center. You can read more about cv2.HoughCircles by clicking <a href=\"https://docs.opencv.org/4.5.1/dd/d1a/group__imgproc__feature.html#ga47849c3be0d0406ad3ca45db65a25d2d\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "italic-settle",
   "metadata": {},
   "source": [
    "Read sample image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "retired-hours",
   "metadata": {},
   "outputs": [],
   "source": [
    "eyeImage = cv2.imread('assets/Lab7-HumanEye.bmp')\n",
    "plt.imshow(eyeImage)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "collected-highland",
   "metadata": {},
   "source": [
    "Convert image to grayscale, apply medianBlur and hough circle on converted image to find pupil and iris circles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "entertaining-blocking",
   "metadata": {},
   "outputs": [],
   "source": [
    "eyeImageGray = cv2.cvtColor(eyeImage, cv2.COLOR_BGR2GRAY) \n",
    "eyeImageGray = cv2.medianBlur(eyeImageGray, 7)\n",
    "\n",
    "pupilCircles = cv2.HoughCircles(eyeImageGray, method=cv2.HOUGH_GRADIENT, dp=1, minDist=100, param1=100, param2=70, minRadius=10, maxRadius=60)\n",
    "irisCircles = cv2.HoughCircles(eyeImageGray, method=cv2.HOUGH_GRADIENT, dp=1, minDist=100, param1=50, param2=70, minRadius=30, maxRadius=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "daily-feeding",
   "metadata": {},
   "source": [
    "Visualize the circle output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "pediatric-entity",
   "metadata": {},
   "outputs": [],
   "source": [
    "eyeImageOutput = eyeImage.copy()\n",
    "\n",
    "if pupilCircles is not None:\n",
    "\tpupilCircles = np.round(pupilCircles[0, :]).astype(\"int\")\n",
    "\tfor (x, y, r) in pupilCircles:\n",
    "\t\tcv2.circle(eyeImageOutput, (x, y), r, (0, 255, 0), 4)\n",
    "\n",
    "if irisCircles is not None:\n",
    "\tirisCircles = np.round(irisCircles[0, :]).astype(\"int\")\n",
    "\tfor (x, y, r) in irisCircles:\n",
    "\t\tcv2.circle(eyeImageOutput, (x, y), r, (255, 255, 0), 4)\n",
    "\n",
    "plt.imshow(eyeImageOutput)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "lucky-setup",
   "metadata": {},
   "source": [
    "## 2. Feature Descriptor\n",
    "OpenCV already include a large set of feature desciptor which can be call via cv2.xfeatures2d classes. More detail can be read from <a href=\"https://docs.opencv.org/4.5.1/d5/d51/group__features2d__main.html\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "reported-spirit",
   "metadata": {},
   "source": [
    "Read sample image from file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "acquired-devices",
   "metadata": {},
   "outputs": [],
   "source": [
    "watPhraKaew1 = cv2.imread('assets/Lab7-w1.jpg')\n",
    "watPhraKaew2 = cv2.imread('assets/Lab7-w2.jpg')\n",
    "\n",
    "watPhraKaew1RGB = cv2.cvtColor(watPhraKaew1, cv2.COLOR_BGR2RGB)\n",
    "watPhraKaew2RGB = cv2.cvtColor(watPhraKaew2, cv2.COLOR_BGR2RGB)\n",
    "\n",
    "watPhraKaew1Gray = cv2.cvtColor(watPhraKaew1, cv2.COLOR_BGR2GRAY)\n",
    "watPhraKaew2Gray = cv2.cvtColor(watPhraKaew2, cv2.COLOR_BGR2GRAY)\n",
    "\n",
    "plt.imshow(watPhraKaew1RGB)\n",
    "plt.show()\n",
    "\n",
    "plt.imshow(watPhraKaew2RGB)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "separated-seller",
   "metadata": {},
   "source": [
    "### 2.1 SIFT (Scale-invariant feature transform)\n",
    "SIFT is both rotation as well as scale invariant. SIFT provides key points and keypoint descriptors where keypoint descriptor describes the keypoint at a selected scale and rotation with image gradients. OpenCV already provided SIFT via cv2.xfeatures2d.SIFT_create()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "psychological-directory",
   "metadata": {},
   "outputs": [],
   "source": [
    "siftExtractor = cv2.xfeatures2d.SIFT_create()\n",
    "kp1, des1 = siftExtractor.detectAndCompute(watPhraKaew1Gray, None)\n",
    "kp2, des2 = siftExtractor.detectAndCompute(watPhraKaew2Gray, None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "statewide-hotel",
   "metadata": {},
   "outputs": [],
   "source": [
    "watPhraKaew1KpOutput = cv2.drawKeypoints(watPhraKaew1RGB, kp1, None, color=(0, 255, 0), flags=cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS)\n",
    "plt.figure(figsize=(10,10))\n",
    "plt.imshow(watPhraKaew1KpOutput)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "resident-murray",
   "metadata": {},
   "outputs": [],
   "source": [
    "watPhraKaew2KpOutput = cv2.drawKeypoints(watPhraKaew2RGB, kp2, None, color=(0, 255, 0), flags=cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS)\n",
    "plt.figure(figsize=(10,10))\n",
    "plt.imshow(watPhraKaew2KpOutput)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "developing-shoulder",
   "metadata": {},
   "outputs": [],
   "source": [
    "bruteForceMatcher = cv2.BFMatcher(cv2.NORM_L2, crossCheck=True)\n",
    "\n",
    "matches = bruteForceMatcher.match(des1, des2)\n",
    "matches = sorted(matches, key = lambda x:x.distance)\n",
    "\n",
    "siftMatches = cv2.drawMatches(watPhraKaew1RGB, kp1, watPhraKaew2RGB, kp2, matches[:40], watPhraKaew2RGB, flags=2)\n",
    "plt.figure(figsize=(30,30))\n",
    "plt.imshow(siftMatches)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "alone-philadelphia",
   "metadata": {},
   "source": [
    "### 2.2 SURF (Speeded-Up Robust Features)\n",
    "SURF is a patented local feature detector and descriptor. It can be used for tasks such as object recognition, image registration, classification, or 3D reconstruction. It is partly inspired by the scale-invariant feature transform (SIFT) descriptor. The standard version of SURF is several times faster than SIFT and claimed by its authors to be more robust against different image transformations than SIFT. However, SURF is not free and cannnot be used without OPENCV_ENABLE_NON_FREE flag! :("
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "supported-geography",
   "metadata": {},
   "outputs": [],
   "source": [
    "surfExtractor = cv2.xfeatures2d.SURF_create()\n",
    "'''\n",
    "OpenCV will thrown an error about non-free algorithm \n",
    "This algorithm is patented and is excluded in this configuration; Set OPENCV_ENABLE_NONFREE CMake option and rebuild the library in function 'create'\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "entire-alexander",
   "metadata": {},
   "source": [
    "### 2.3 FAST (Features from accelerated segment test)\n",
    "Features from accelerated segment test (FAST) is a corner detection method, which could be used to extract feature points and later used to track and map objects in many computer vision tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "micro-winter",
   "metadata": {},
   "outputs": [],
   "source": [
    "fastExtractor = cv2.FastFeatureDetector_create()\n",
    "fastExtractor.setNonmaxSuppression(False)\n",
    "kp1 = fastExtractor.detect(watPhraKaew1Gray, None)\n",
    "kp2 = fastExtractor.detect(watPhraKaew2Gray, None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "thirty-boards",
   "metadata": {},
   "outputs": [],
   "source": [
    "watPhraKaew1KpOutput = cv2.drawKeypoints(watPhraKaew1RGB, kp1, None, color=(0, 255, 0), flags=cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS)\n",
    "plt.figure(figsize=(10,10))\n",
    "plt.imshow(watPhraKaew1KpOutput)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "smaller-moral",
   "metadata": {},
   "outputs": [],
   "source": [
    "watPhraKaew2KpOutput = cv2.drawKeypoints(watPhraKaew2RGB, kp2, None, color=(0, 255, 0), flags=cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS)\n",
    "plt.figure(figsize=(10,10))\n",
    "plt.imshow(watPhraKaew2KpOutput)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "european-rescue",
   "metadata": {},
   "source": [
    "### 2.4 ORB (Oriented FAST and Rotated BRIEF)\n",
    "ORB is an efficient open source alternative to SIFT and SURF. It computes less keypoints when compared to SIFT and SURF yet they are effective."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "lesser-australian",
   "metadata": {},
   "outputs": [],
   "source": [
    "orbExtractor = cv2.ORB_create(nfeatures=1000)\n",
    "kp1, des1 = orbExtractor.detectAndCompute(watPhraKaew1Gray, None)\n",
    "kp2, des2 = orbExtractor.detectAndCompute(watPhraKaew2Gray, None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "verbal-olive",
   "metadata": {},
   "outputs": [],
   "source": [
    "watPhraKaew1KpOutput = cv2.drawKeypoints(watPhraKaew1RGB, kp1, None, color=(0, 255, 0), flags=cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS)\n",
    "plt.figure(figsize=(10,10))\n",
    "plt.imshow(watPhraKaew1KpOutput)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "comic-elements",
   "metadata": {},
   "outputs": [],
   "source": [
    "watPhraKaew2KpOutput = cv2.drawKeypoints(watPhraKaew2RGB, kp2, None, color=(0, 255, 0), flags=cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS)\n",
    "plt.figure(figsize=(10,10))\n",
    "plt.imshow(watPhraKaew2KpOutput)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "middle-university",
   "metadata": {},
   "outputs": [],
   "source": [
    "bruteForceMatcher = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)\n",
    "\n",
    "matches = bruteForceMatcher.match(des1, des2)\n",
    "matches = sorted(matches, key = lambda x:x.distance)\n",
    "\n",
    "siftMatches = cv2.drawMatches(watPhraKaew1RGB, kp1, watPhraKaew2RGB, kp2, matches[:40], watPhraKaew2RGB, flags=2)\n",
    "plt.figure(figsize=(30,30))\n",
    "plt.imshow(siftMatches)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ordinary-queens",
   "metadata": {},
   "source": [
    "## Assignment 1 - Image Stitching\n",
    "Image stitching or photo stitching is the process of combining multiple photographic images with overlapping fields of view to produce a segmented panorama or high-resolution image. Commonly performed through the use of computer software, most approaches to image stitching require nearly exact overlaps between images and identical exposures to produce seamless results. Luckily, OpenCV already provide <b>a super easy</b> class to merge a set of corespondings images in to panoram aimage. In this assignment, you will need to apply <a href=\"https://docs.opencv.org/4.5.1/d2/d8d/classcv_1_1Stitcher.html\">cv2.Stitcher_create</a> on the collection of images provided in assets directory to create a single panorama image.<br>\n",
    "<b> Hint: This assignment is super easy</b>\n",
    "<br>\n",
    "OpenCV image Stitching pipeline\n",
    "<img src=\"https://docs.opencv.org/master/StitchingPipeline.jpg\"></img>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "alpine-senior",
   "metadata": {},
   "outputs": [],
   "source": [
    "### FILL HERE ###"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}