notebooks-v1/eods-api-example-generate-cloudless-mosaic.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Defra Earth Observation Data Service API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction \n",
    "This notebook has been created to introduce and test the Defra Earth Observation Data Service API.  Follow the three steps below to import libraries and configuration file, apply query parameters to filter available data, and create a parameterised URL using the api/layers endpoint with query parameters for date, geometry and satellite.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1) Import libraries and configuration file\n",
    "Here we *import* the libraries required to run this notebook.\n",
    "If there are any errors when running this cell (all cells can be run using *'Shift-Return'*) then check that the libraries were correctly installed through **Conda** or the package manager that you use.\n",
    "\n",
    "Before continuing with this notebook, ensure that the **USERNAME** and **ACCESS_TOKEN** parameters in the _config.py_ file relate to the username to be used for the following commands.\n",
    "\n",
    "Check all other _config.py_ parameters are as you would expect them, especially the URL and GEOM parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import general and geospatial data handling libraries\n",
    "import urllib3, requests, os, importlib, json\n",
    "import logging\n",
    "import pandas as pd\n",
    "from pandas.io.json import json_normalize\n",
    "from IPython.display import Image\n",
    "from datetime import datetime\n",
    "import time\n",
    "import os\n",
    "import pathlib\n",
    "from pathlib import Path\n",
    "# Import your configuration file\n",
    "import config\n",
    "# configure specific library calls\n",
    "importlib.reload(config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialise notebook\n",
    "urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)\n",
    "pd.options.display.max_colwidth = 100\n",
    "\n",
    "output_fmt='%Y%m%dT%H%M%S'\n",
    "pretty_fmt='%Y-%m-%d %H:%M:%S'\n",
    "\n",
    "# create output directory if it does not exist\n",
    "output_dir = Path(Path.cwd() / 'data')\n",
    "if not output_dir.is_dir():\n",
    "    output_dir.mkdir()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2) Configure query parameters\n",
    "The next cell sets a series of query parameters for searching through the data on the system. You can change these and re-run the cell to investigate different scenarios. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# configure variables\n",
    "DATE_RANGE_START = '2019-01-01'\n",
    "DATE_RANGE_END = '2019-06-01'\n",
    "SATELLITE_ID = 2\n",
    "TITLE = 'S2A'\n",
    "TYPE = 'raster'\n",
    "GEOM = \"POLYGON((-1.6+51.36,0.28+51.36,0.28+51.69,-1.6+51.69,-1.6+51.36))\"\n",
    "# User should select APPLY_MIN_CLOUD_FUNCTION == True with APPLY_FILTERS == True\n",
    "APPLY_FILTERS = True\n",
    "APPLY_MIN_CLOUD_FUNCTION = True\n",
    "\n",
    "# search locations\n",
    "# England-wide footprint\n",
    "# GEOM = \"POLYGON((-3.70583842728179391 55.44991031769518486, -2.13623226289823265 55.44991031769518486, -0.53426308481603968 55.43372881084587078, -0.58280760536398546 54.54374593413353978, -0.63135212591193124 53.65376305742121588, 0.84116499737574024 53.58903703002395247, 2.31368212066341172 53.65376305742121588, 2.3622266412113575 52.76378018070888487, 2.39458965490998743 51.84143429029792571, 2.39458965490998743 50.90290689303765248, 2.3622266412113575 50.07765004372258488, 0.9867985590195758 50.04528703002395673, 0.53371636723875326 49.9967425094760074, -0.82553020810371081 50.04528703002395673, -2.23332130399412243 50.06146853687327081, -3.60874938618590413 50.04528703002395673, -5.42107815330919074 49.9967425094760074, -6.82886924919960148 50.04528703002395673, -5.50198568755576645 50.17473908481847644, -5.40489664645987578 50.87054387933902433, -3.6249308930352182 50.91908839988696656, -3.70583842728179391 55.44991031769518486))\"\n",
    "# small London footprint\n",
    "# GEOM = \"POLYGON((-0.09799 51.49697,-0.08799 51.49697,-0.08799 51.48697,-0.09799 51.48697,-0.09799 51.49697))\"\n",
    "\n",
    "try:\n",
    "    assert not (APPLY_FILTERS == False and APPLY_MIN_CLOUD_FUNCTION == True)\n",
    "except AssertionError as err:\n",
    "    logging.error(\"\\n\\n # CONFIG ERROR! \\n If you want to use APPLY_MIN_CLOUD_FUNCTION, then set APPLY_FILTERS to 'True'\")\n",
    "    raise err\n",
    "\n",
    "# do config parameter validation. ## User needs to select APPLY_MIN_CLOUD_FUNCTION == True and SATELLTE_ID == 2\n",
    "try:\n",
    "    assert not (APPLY_MIN_CLOUD_FUNCTION == True and SATELLITE_ID == 1)\n",
    "except AssertionError as err:\n",
    "    logging.error('\\n # CONFIG ERROR \\n You are trying to apply the \"APPLY_MIN_CLOUD_FUNCTION\" but have not set \"SATELLTE_ID\" to \"2\"')\n",
    "    raise err\n",
    "\n",
    "    \n",
    "\n",
    "#****NOTES****\n",
    "#\n",
    "#APPLY_FILTERS = False >> You download all layers from the EO service\n",
    "#APPLY_FILTERS = True >> The above filter parameters will be applied to the api query#\n",
    "#\n",
    "#APPLY_MIN_CLOUD_FUNCTION = False >> Applies to S2 data only, will download all S2 data as per filter parameter spec\n",
    "#APPLY_MIN_CLOUD_FUNCTION = True >> Applies to S2 data only, will only download S2 granules that meet \n",
    "#    the lowest cloud cover criteria for each unique granule reference per orbit\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  3) Create parameterised URL\n",
    "The following cell creates a parameterised URL using the API 'layers' endpoint and a set of query parameters. Copy the output and paste it into a browser address bar. The resultant page provides an indication of the information returned to this notebook (based on the input parameters) from the server. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialise the required variables using the details in your config file (config.py)\n",
    "\n",
    "# Create URLs for the Geoserver endpoint (gs_end_point), OWS endpoint (serverURL)\n",
    "gsdownload_template_payload = '<?xml version=\"1.0\" encoding=\"UTF-8\"?><wps:Execute version=\"1.0.0\" service=\"WPS\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns=\"http://www.opengis.net/wps/1.0.0\" xmlns:wfs=\"http://www.opengis.net/wfs\" xmlns:wps=\"http://www.opengis.net/wps/1.0.0\" xmlns:ows=\"http://www.opengis.net/ows/1.1\" xmlns:gml=\"http://www.opengis.net/gml\" xmlns:ogc=\"http://www.opengis.net/ogc\" xmlns:wcs=\"http://www.opengis.net/wcs/1.1.1\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xsi:schemaLocation=\"http://www.opengis.net/wps/1.0.0 http://schemas.opengis.net/wps/1.0.0/wpsAll.xsd\"><ows:Identifier>gs:Download</ows:Identifier><wps:DataInputs><wps:Input><ows:Identifier>layerName</ows:Identifier><wps:Data><wps:LiteralData>@@@</wps:LiteralData></wps:Data></wps:Input><wps:Input><ows:Identifier>outputFormat</ows:Identifier><wps:Data><wps:LiteralData>image/tiff</wps:LiteralData></wps:Data></wps:Input></wps:DataInputs><wps:ResponseForm><wps:ResponseDocument storeExecuteResponse=\"true\" status=\"true\"><wps:Output asReference=\"true\" mimeType=\"application/zip\"><ows:Identifier>result</ows:Identifier></wps:Output></wps:ResponseDocument></wps:ResponseForm></wps:Execute>'\n",
    "headers = {'Content-type': 'application/xml','User-Agent': 'curl/7.65.1'}\n",
    "wps_server = config.URL + config.WPS_SUF + '?access_token=' + config.ACCESS_TOKEN\n",
    "\n",
    "    \n",
    "# If the APPLY_FILTERS config option is set to True then construct a URL\n",
    "if APPLY_FILTERS:\n",
    "    the_url = config.URL + ('/api/layers/?username=' + config.USERNAME \n",
    "                            + '&api_key=' + config.ACCESS_TOKEN\n",
    "                            + '&limit=20000&offset=0'\n",
    "                            + '&keywords__slug__in=sentinel-' + str(SATELLITE_ID)\n",
    "                            + '&geometry=' + GEOM\n",
    "                            + '&date__range=' + DATE_RANGE_START + '%2000:00,' + DATE_RANGE_END + '%2023:59'\n",
    "                            + '&title__icontains=' + TITLE\n",
    "                            + '&type__in=' + TYPE\n",
    "                           )\n",
    "    \n",
    "# or if the option is not set to True \n",
    "else:\n",
    "    the_url = config.URL + ('/api/layers/?username=' \n",
    "                            + config.USERNAME + '&api_key=' \n",
    "                            + config.ACCESS_TOKEN + '&limit=20000&offset=0')\n",
    "\n",
    "\n",
    "print(the_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  4) Make a GET request using the URL above and parse the output to a dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make the request to the API and parse the output to a pandas dataframe\n",
    "try:\n",
    "    # parse json response from the geonode api to a pandas dataframe object\n",
    "    payload = requests.get(url=the_url,headers=headers)\n",
    "\n",
    "    # create a json object of the api payload content\n",
    "    the_json = json.loads(payload.content)\n",
    "\n",
    "    # load json to pandas dataframe\n",
    "    df_full_layer_list = json_normalize(the_json, 'objects')\n",
    "except:\n",
    "    raise Exception('ERROR. The the server response is not correctly formed. Please check you authentication token and internet connection')\n",
    "\n",
    "if len(df_full_layer_list) == 0:\n",
    "    raise Exception('ERROR. The API response was empty, no data to process or download, check the filters match data in the EO Portal')\n",
    "\n",
    "# add granule reference and extract ARCSI_CLOUD_COVER from supplement information value\n",
    "if SATELLITE_ID == 2:\n",
    "    df_full_layer_list['granule-ref'] = df_full_layer_list['title'].str.split('_',n=4).str[3]\n",
    "    df_full_layer_list['orbit-ref'] = df_full_layer_list['title'].str.split('_',n=5).str[-2]\n",
    "    df_full_layer_list['ARCSI_CLOUD_COVER'] = df_full_layer_list['supplemental_information'].str.split(n=6).str[5]    \n",
    "    \n",
    "# show first five rows of the dataframe\n",
    "df_full_layer_list.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  5) Sort the dataframe by % cloud cover and group by granule name and orbit \n",
    "The following cell sorts the dataframe by % cloud cover ascending and groups the results to show the lowest cloud cover per granule, and per granule per relative orbit.  This is important because the imagery with the lowest cloud may be from an orbit where the granule lies on the edge of the swath and therefore does not provide complete coverage of the granule footprint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if APPLY_MIN_CLOUD_FUNCTION and SATELLITE_ID == 2:\n",
    "    df_min_cloud_per_granule = df_full_layer_list.sort_values(\"ARCSI_CLOUD_COVER\").groupby([\"granule-ref\"], as_index=False).first()  \n",
    "    df_min_cloud_per_granule_per_orbit = df_full_layer_list.sort_values(\"ARCSI_CLOUD_COVER\").groupby([\"granule-ref\",'orbit-ref'], as_index=False).first()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  6) View and download the results "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# list the column headers of the 'min cloud per granule' data frame\n",
    "df_min_cloud_per_granule.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# show the 'min cloud per granule' dataframe with just the most relevant columns\n",
    "df_min_cloud_per_granule[[\"granule-ref\",\"orbit-ref\",\"date\",\"ARCSI_CLOUD_COVER\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# output the minimum cloud lists as a CSV to a the 'data' directory\n",
    "df_min_cloud_per_granule.to_csv(Path(output_dir / 'min_cloud_per_granule.csv'))\n",
    "df_min_cloud_per_granule_per_orbit.to_csv(Path(output_dir / 'min-cloud-per-granule-per-orbit.csv'))\n",
    "\n",
    "# Check the 'data' directory in your Scripts folder, open the two CSV files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# show the list of filenames in the 'minimum cloud per granule' list\n",
    "list_to_download = df_min_cloud_per_granule.title.tolist()\n",
    "list_to_download"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# show the list of filenames in the 'minimum cloud per granule per orbit' list\n",
    "list_to_download_orbit = df_min_cloud_per_granule_per_orbit.title.tolist()\n",
    "list_to_download_orbit"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}