example-notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting started with GEDIPipeline\n",
    "\n",
    "This notebook demonstrates how to use GEDIPipeline - the unified framework to download, subset, and clip Global Ecosystem Dynamics Investigation (GEDI) data over a specific Region of Interest (ROI).\n",
    "\n",
    "With this framework, you can streamline the process of acquiring and preparing GEDI data for analysis, enabling more efficient workflows for remote sensing and environmental research.\n",
    "With this example notebook, you'll learn:\n",
    "\n",
    "- How to define an ROI and configure the Pipeline.\n",
    "- Steps to find, download, and process GEDI granules.\n",
    "- How to use the Pipeline to automate the entire workflow.\n",
    "\n",
    "## Requirements\n",
    "\n",
    "1. EarthData Credentials: Ensure you have an active NASA EarthData account. You can create one [here](https://urs.earthdata.nasa.gov/).\n",
    "2. Python Environment: Install the required Python packages listed in the repository's requirements.txt.\n",
    "\n",
    "Make sure you have access to the repository by cloning it to your working machine or another working environment.\n",
    "\n",
    "Repository Link:\n",
    "\n",
    "https://github.com/leonelluiscorado/GEDI-Pipeline\n",
    "\n",
    "For more details, consult the repository's [README](https://github.com/leonelluiscorado/GEDI-Pipeline/blob/main/README.md).\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# RUN THIS BEFORE GOING THROUGH NOTEBOOK\n",
    "\n",
    "import os\n",
    "\n",
    "example_path = \"./example_usage\" # Replace with desired notebook's output folder\n",
    "\n",
    "if not os.path.exists(example_path):\n",
    "    os.mkdir(example_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Defining the example ROI and acquisition dates\n",
    "\n",
    "In this example, we will use a small ROI in Portugal to demonstrate the execution of GEDIPipeline. In this framework, the ROI is defined as a bounding box with coordinates that *must be* in WG84 EPSG:4326 and organized as follows in a list: \n",
    "\n",
    "`[UpperLeft_Latitude, UpperLeft_Longitude, LowerRight_Latitude, LowerRight_Longitude]`\n",
    "\n",
    "- Example `[40.35, -6.93, , 38.19, -6.93]`\n",
    "\n",
    "For the acquisition dates, we define two variables `date_start` and `date_end` which describe the desired start and end dates from which to download GEDI data.\n",
    "Each date is a *string* and must be in this format `\"YYYY.MM.DD\"`.\n",
    "\n",
    "- Example date `\"2024.11.28\"`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define our ROI and data collection dates\n",
    "\n",
    "roi = [40.356011, -6.938200, 40.321162, -6.876083]  # Replace with your desired coordinates\n",
    "date_start = '2020.04.30' # Replace date_start and end with desired acquisition dates\n",
    "date_end = '2020.10.31'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using the Finder class\n",
    "\n",
    "The *Finder* searches NASA's data repository for all the available orbits that pass over the ROI returning a list of URLs containing the download links for the GEDI orbits.\n",
    "Before using the Finder, the user selects the desired GEDI product and version to download."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Available GEDI Products\n",
    "\n",
    "- GEDI L1B Geolocated Waveform Data Global Footprint Level - GEDI01_B\n",
    "- GEDI L2A Elevation and Height Metrics Data Global Footprint Level - GEDI02_A\n",
    "- GEDI L2B Canopy Cover and Vertical Profile Metrics Data Global Footprint Level - GEDI02_B\n",
    "- GEDI L4A Footprint Level Aboveground Biomass Density - GEDI04_A\n",
    "\n",
    "For each GEDI data product, you can specify which version you want to download: version '001' or version '002'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the Finder class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import class\n",
    "from pipeline.finder import GEDIFinder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help(GEDIFinder) # Describe arguments and example usage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create Finder instance\n",
    "\n",
    "finder = GEDIFinder(\n",
    "            product = 'GEDI02_A',\n",
    "            version = '002',\n",
    "            date_start = date_start,\n",
    "            date_end = date_end,\n",
    "            roi = roi\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Return all available orbits\n",
    "\n",
    "orbits = finder.find(save_file = True, output_filepath = example_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "orbits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each orbit consists of a tuple with (Download URL, Filesize). This filesize variable is valuable for file checking before downloading (e.g. if file does not exist entirely, download it again).\n",
    "To access the URL, we simple obtain the first variable of the tuple like so: `granule[0]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Obtain second URL\n",
    "\n",
    "orbits[1][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After obtaining the intersecting orbits on the desired ROI, we can download these GEDI files with the Downloader class.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using the Downloader class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before downloading, the framework will ask for your EarthData credentials. After logging in successfully, it will save your credentials in your user's `.netrc` file if the `persist_login` flag is True (by default is set to False)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import downloader class\n",
    "\n",
    "from pipeline.downloader import GEDIDownloader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help(GEDIDownloader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the Downloader module, we simply create an instance and call the `download_granule` for a single URL or `download_files` for a list of URLs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create downloader instance\n",
    "\n",
    "downloader = GEDIDownloader(persist_login = False, save_path = example_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Depending on the GEDI Product, each downloaded HDF5 file will occupy ~1-2GB. Be sure that you have enough disk space to download."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download all of the intersecting files\n",
    "\n",
    "downloader.download_files(orbits)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Your .H5 files are now saved in your specified output directory, we can do a quick check:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.listdir(example_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Downloader downloads each (entire) granule to a specified directory, as subsetting the granule before downloading is currently unavailable through the APIs provided. The listed .H5 files are not clipped to our ROI and use all of the available data product variables. To solve this, we use the following Subsetter class.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using the Subsetter class\n",
    "\n",
    "This class clips to the study area and selects all the available Science Dataset (SDS) data product variables to burn for each footprint, and outputs the subsetted orbit, which was previously downloaded. This class accepts any downloaded GEDI HDF5 from LPDAAC. Let's start by creating a Subsetter instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import class\n",
    "from pipeline.subsetter import GEDISubsetter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help(GEDISubsetter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Subsetter allows for a high level of customization (e.g. select BEAMS and variables from the specified data product).\n",
    "Before selecting SDS variables, check the Data Product Dictionary for the variable names. After this, if the variable is inside a group or subgroup except `BEAMXXXX/`, its parent group must be specified, for example `/geolocation/lat_lowestmode`. The SDS argument defaults to the lists in `subsetter.py`, otherwise, it appends to the default lists.\n",
    "\n",
    "The default lists are:\n",
    "\n",
    "```python\n",
    "# Default layers to be subset and exported, see README for information on how to add additional layers\n",
    "l1b_subset = ['/geolocation/latitude_bin0', '/geolocation/longitude_bin0', '/channel', '/shot_number', '/rx_sample_start_index',\n",
    "             '/rxwaveform','/rx_sample_count', '/stale_return_flag', '/tx_sample_count', '/txwaveform',\n",
    "             '/geolocation/degrade', '/geolocation/delta_time', '/geolocation/digital_elevation_model',\n",
    "              '/geolocation/solar_elevation',  '/geolocation/local_beam_elevation',  '/noise_mean_corrected',\n",
    "             '/geolocation/elevation_bin0', '/geolocation/elevation_lastbin', '/geolocation/surface_type', '/geolocation/digital_elevation_model_srtm' '/geolocation/degrade']\n",
    "\n",
    "l2a_subset = ['/lat_lowestmode', '/lon_lowestmode', '/channel', '/shot_number', '/degrade_flag', '/delta_time', \n",
    "             '/digital_elevation_model', '/elev_lowestmode', '/quality_flag', '/rh', '/sensitivity', '/rx_cumulative', '/digital_elevation_model_srtm', \n",
    "             '/elevation_bias_flag', '/surface_flag',  '/num_detectedmodes',  '/selected_algorithm',  '/solar_elevation']\n",
    "\n",
    "\n",
    "l2b_subset = ['/geolocation/lat_lowestmode', '/geolocation/lon_lowestmode', '/channel', '/geolocation/shot_number',\n",
    "             '/cover', '/cover_z', '/fhd_normal', '/pai', '/pai_z',  '/rhov',  '/rhog',\n",
    "             '/pavd_z', '/l2a_quality_flag', '/l2b_quality_flag', '/rh100', '/sensitivity',  \n",
    "             '/stale_return_flag', '/surface_flag', '/geolocation/degrade_flag',  '/geolocation/solar_elevation',\n",
    "             '/geolocation/delta_time', '/geolocation/digital_elevation_model', '/geolocation/elev_lowestmode', '/pgap_theta']\n",
    "\n",
    "l4a_subset = [] # TODO: select relevant L4A product variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The user can also select desired BEAMs in a list like so: `['BEAM0000', 'BEAM0001']`. If the user does not specify BEAMS, it defaults to all available beams:\n",
    "\n",
    "```python\n",
    "# Default BEAM Subset\n",
    "beam_subset = ['BEAM0000', 'BEAM0001', 'BEAM0010', 'BEAM0011', 'BEAM0101', 'BEAM0110', 'BEAM1000', 'BEAM1011']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### For now, we'll use the default BEAMS and SDS variables for the L2A data product\n",
    "\n",
    "Let's subset the previously downloaded files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "subsetter = GEDISubsetter(\n",
    "                roi = roi,              # Desired ROI to clip\n",
    "                product = 'GEDI02_A',   # Desired data product\n",
    "                out_dir = example_path, # Output file directory to save the .GPKG files\n",
    "                sds = None,             # SDS Variables to append to default\n",
    "                beams = None            # BEAMS to select, None selects all the available BEAMS\n",
    "            )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select paths of all the downloaded .H5 files\n",
    "\n",
    "files = [os.path.join(example_path, f) for f in os.listdir(example_path) if '.h5' in f] \n",
    "\n",
    "files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfs = [] # Dataframes List\n",
    "\n",
    "# Subset the downloaded granules\n",
    "for file in files:\n",
    "    file_df = subsetter.subset(file) # Subset file\n",
    "    dfs.append(file_df) # Save to all GeoDataFrames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After subsetting and outputting the .GPKG files, the subsetter function returns the datasets in GeoDataFrame format. We can process them in a notebook for later use. Let's check them out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "example_df = dfs[2]\n",
    "\n",
    "example_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have successfully downloaded and processed GEDI orbit(s)! The output can now be processed for the user's research purposes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using GEDIPipeline\n",
    "\n",
    "Using each module separately is useful for specific applications (e.g. finding only the available orbits on ROI), however, running this entire process by creating all of the 3 classes' instances can be time-consuming.\n",
    "The entire process described in this notebook can be automated using a single class, which is the GEDIPipeline, with a few improvements:\n",
    "\n",
    "- The Pipeline automatically deletes the original downloaded file after subsetting it, saving disk space.\n",
    "- More improvements WIP\n",
    "\n",
    "For this Pipeline, we specify all of the previously described arguments for each class, in one instance creation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import\n",
    "\n",
    "from pipeline.pipeline import GEDIPipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help(GEDIPipeline)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pipeline instance\n",
    "\n",
    "pipeline = GEDIPipeline(\n",
    "    out_directory = example_path,\n",
    "    product = 'GEDI02_A',\n",
    "    version = '002',\n",
    "    date_start = date_start,\n",
    "    date_end = date_end,\n",
    "    roi = roi,\n",
    "    beams = None,\n",
    "    sds = None,\n",
    "    persist_login = False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To run the entire pipeline, we simply call `run_pipeline()` on our pipeline instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline.run_pipeline()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook showed the basic usage of this Pipeline, however, each module can be used separately for specific purposes.\n",
    "If you have any questions, open an issue on the GitHub repository or contact us at: leonel.corado@uevora.pt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}