05a_unsupervised_subsampling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ffa97cea-ceda-4a0d-8709-c3efc724d72a",
   "metadata": {
    "tags": []
   },
   "source": [
    "<font face='Calibri' size='2'> <i>eSBAE - Notebook Series - Part 5a, version 0.1, April 2023. Andreas Vollrath, UN-Food and Agricultural Organization, Rome</i>\n",
    "</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e19161d7-1cc6-4499-8f00-2432332fff6a",
   "metadata": {
    "tags": []
   },
   "source": [
    "![title](images/header.png)\n",
    "\n",
    "# V-a - Unsupervised Subsampling\n",
    "### Run a KMeans unsupervised clustering algorithm to get a first set of *\"statistically balanced\"* training data\n",
    "-------\n",
    "\n",
    "This notebook takes you through the process of clustering your data points in an unsupervised fashion. From this you can select a subsample, which can serve as a basis for annotation. Data is exported as a CEO compatible file. \n",
    "\n",
    "The reason why we use KMeans over a simple random selection is that we want to overproportoinally capture rare classes. For example, one of the clusters does represent change events, but the cluster itself consists of only 1% of all samples. In a random selection of 100 points, only 1 point would be selected. If we choose 10 clusters  adn sample 10 points within each cluster, we assure that 10% of the samples are actually change. This is beneficial for subsequqnt steps of classifying, as we need a sufficient amount of samples for our rare classes of change. \n",
    "\n",
    "The number of clusters is an iterative approach. The Statistic Plots will help to see, if any cluster actually might capture forest change pre-dominantly. The number of points per cluster are determined by th eoverall amount of samples one is able to later interpret. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a04ac340-acfa-4ab7-be58-e8288846b66d",
   "metadata": {},
   "source": [
    "### 1 - Import libraries (*only execute this cell*)\n",
    "\n",
    "This cell will provide us with the functionality we need for running the subsequent cells of the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ddc5cc15-3260-448a-a3da-c74d2d581fa2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sampling_handler import KMeansSubSampling"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11e4b85c-c245-4a12-b109-0b9fa5883373",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 2 - Basic Input Variables\n",
    "\n",
    "Here a so called class instance is initialized. The class instance needs some parameters to be set and is written into the *esbae* variable. See the commented lines for further explanation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b61618b3-507c-4b14-b15d-eba2a896361a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "esbae = KMeansSubSampling(\n",
    "\n",
    "    # your project name (NEEDS to be consistent with previous notebooks of your project)\n",
    "    project_name = 'my_first_esbae_project',\n",
    "    \n",
    "    # select the number of clusters (reasonable numbers range from 5 to 30)\n",
    "    clusters=10,\n",
    "    \n",
    "    # select the points per cluster (multiplied by the number fo cluster, will give you the overall number of samples ou will get)\n",
    "    points_per_cluster=10,\n",
    "    \n",
    "    # a random state for reproducability (can be any integer number)\n",
    "    random_state=25\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74d573c5-9a2b-450a-846c-3e41bbc5edf7",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 3 - Run the clustering algorithm\n",
    "\n",
    "The cluster algorithm shall be run with standardized inputs (e.g. all input variables are stadnardized to the same range). However, in certain cases, non standardized inputs may give better results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a985cf0-776f-4b25-a4de-fae5a97963f0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "esbae.cluster(standardize_inputs=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "838b9fdf-9ead-4998-bf5f-a1e1f9d863a4",
   "metadata": {},
   "source": [
    "### 4 - Plot the clusters on a map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "017a5439-5837-436a-8397-ce5c534e1ee5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import contextily as cx\n",
    "esbae.plot_clusters(markersize=2, basemap=cx.providers.Esri.WorldImagery) #check other basemaps here: https://contextily.readthedocs.io/en/latest/intro_guide.html"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a58f0e90-08af-4c81-9f7a-e6b19b3d84c7",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 5 - Plot the clusters against the input variables to get an idea of what they represent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17c2783f-53d3-4d84-89b5-9a11681fdf3b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "esbae.plot_stats(class_column='KMeans', cols_to_plot=['cusum_confidence', 'cusum_magnitude', 'esa_lc20'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "867b14ef-f943-40cc-8ff8-739403e9b72e",
   "metadata": {},
   "source": [
    "### 6 - Subsampling\n",
    "\n",
    "This step subsamples each cluster by the amount of points per cluster defined during the initialization step in cell 2. There are 2 ways of selecting the subset, one is *randomly* and the other uses a *space filling curve* based on the Hilbert distance. \n",
    "\n",
    "The output can be saved as CEO. A csv file with point coordinates is created. In adidtion, a zipped shapefile is created as well. In case in Notebook 3, the bounds_reduce option has been used, the polygons of the reducer are taken as geometry."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aec01bca-8afb-4362-ba71-7fe4628cf93f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "esbae.sampling_type = 'space_filling_curve'  # or 'random'\n",
    "esbae.select_samples(save_as_ceo=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85121763-7876-416b-be07-bda026b0a85c",
   "metadata": {},
   "source": [
    "### 7 - Plot subsample selection on a map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "125efa71-d946-4a06-a352-16920a488ea6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "esbae.plot_samples(markersize=5, basemap=cx.providers.Esri.WorldImagery)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": " (venv) Sample Based Area Estimation",
   "language": "python",
   "name": "venv-esbae_notebooks"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}