rcafetch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A more detailed overview of this notebook\n",
    "\n",
    "This notebook began as a comparison between profiler chlorophyll measurements near the\n",
    "surface nine times per day to surface chlorophyll observations by the \n",
    "MODIS satellite once every eight days. Its scope expanded from there, considerably. \n",
    "\n",
    "\n",
    "One such expansion is considering other sources of data. We have for example \n",
    "a snapshot of the global ocean called GLODAP. After inspecting that on a global\n",
    "scale we turn to a comparison of vertical profiles through the water column, \n",
    "specifically salinity and temperature. We want to compare GLODAP profiles as somewhat \n",
    "*static* snapshots with ongoing active profile measurements from ARGO drifters.\n",
    "\n",
    "\n",
    "The Regional Cabled Array (RCA)\n",
    "is an observatory stretching across the sea floor from the coast of Oregon 500 km out to\n",
    "Axial Seamount. This observatory includes two types of profilers that rise and fall through\n",
    "the water column: Both deep profilers that ascend from the sea floor and shallow profilers \n",
    "that rest on platforms at 200 meters depth and ascend to within a few meters of the surface.\n",
    "\n",
    "\n",
    "We begin the RCA work focused on the shallow profiler as this is where the highest\n",
    "concentration of chlorophyll is found.\n",
    "\n",
    "\n",
    "* Regional Cabled Array (RCA): A cabled observatory on the sea floor off the coast of Oregon\n",
    "* Site: A location in the RCA\n",
    "* Platform: A mechanical structure -- static or mobile -- that resides at a site.\n",
    "* Instrument: An electronic device carrying one or more sensors\n",
    "* Sensor: A device that measures some aspect of the ocean like pH or temperature\n",
    "* Stream: A stream of data produced by a sensor as part of an instrument located on a platform at a site in the RCA\n",
    "\n",
    "\n",
    "This notebook describes a Python package called **yodapy** used to obtain stream data.\n",
    "\n",
    "\n",
    "Here we use the traditional data path model\n",
    "\n",
    "\n",
    "* search for data\n",
    "* order data\n",
    "* download data\n",
    "* analyze data\n",
    "\n",
    "\n",
    "We prefer a newer approach where data are already in place on the public cloud and the model is\n",
    "\n",
    "* analyze data\n",
    "\n",
    "\n",
    "Since that is our end-goal some of the data for this project will be (not done yet 3-20) set\n",
    "in place in advance. \n",
    "\n",
    "\n",
    "Back to our process here: Once the data are in place we say that **yodapy** has finished its task.\n",
    "We then turn to analysis using Python and particularly **XArray**. \n",
    "\n",
    "## Purpose of the **yodapy** Python package\n",
    "\n",
    "`yodapy` is a contraction of **Y**our **O**cean **DA**ta **PY**thon library. It was written \n",
    "by Don Setiawan to facilitate working with **OOI** data in general (not just profiler data).\n",
    "\n",
    "\n",
    "Before `yodapy` was written the process of finding and ordering data for OOI \n",
    "was a bit *involved*.  `yodapy` was developed to make this process more \n",
    "*programmable* and to provide search capability\n",
    "without having to know precise terms like \n",
    "`RS01SBPS-SF01A-3D-SPKIRA101-streamed-spkir_data_record`.  \n",
    "\n",
    "\n",
    "\n",
    "This notebook uses `yodapy` to search for, identify, order and download data, all in Python. \n",
    "A future extension of `yodapy` will make this process even simpler, referencing data that\n",
    "are already in place on the public cloud. Rather than order and download data you simply\n",
    "start working with it using the `XArray` idiom. \n",
    "\n",
    "<BR>\n",
    "\n",
    "> **Takeaway 1: This notebook reviews `yodapy` specific to Regional Cabled Array (RCA) \n",
    "data but the pattern of use is relevant to data from other OOI segments.**\n",
    "\n",
    "<BR>\n",
    "\n",
    "> **Takeaway 2: A heads-up on authenticating: The OOI system requires you to *authenticate* your identity.\n",
    "You do this by registering your email at their website. This is unrestricted, there is\n",
    "no cost and it only takes a couple of minutes. `yodapy` helps you manage your resulting\n",
    "credentials so once this is set up you are authenticated automatically.**\n",
    "\n",
    "## Notebook features\n",
    "\n",
    "Come back and re-write this (and never index your own book)\n",
    "\n",
    "### Section on GLODAP and ARGO\n",
    "\n",
    "### Regional Cabled Array and MODIS\n",
    "\n",
    "- OOI / RCA data orders with `yodapy`\n",
    "- working with `xarray` `DataArrays` and `Datasets` \n",
    "- plotting with `matplotlib`\n",
    "  - line and scatter plots, multiple y-axes, labels, marker type and size\n",
    "  - profiler curtain plots: time - depth - chlorophyll (as color) \n",
    "  - animation of time series data\n",
    "  - interactivity\n",
    "  - color bars\n",
    "  - intrinsic plotting from DataArrays (with modifiers)\n",
    "\n",
    "### Ordering and pulling ARGO data from the Coriolis system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data management\n",
    "\n",
    "This Jupyter notebook resides in \n",
    "a sub-directory of the User's home directory `~`. It is bundled \n",
    "as an open source \n",
    "[github repository](https://github.com/robfatland/chlorophyll).\n",
    "(abbreviated 'repo') on GitHub using the\n",
    "Linux `git` utility. \n",
    "The repo is not intended for large data volumes.  \n",
    "\n",
    "\n",
    "The data must reside *elsewhere* in the \n",
    "working environment, i.e. not within the repo directory.\n",
    "I use `~/data/` with sub-directories to organize data content outside\n",
    "of the `~/chlorophyll` directory. \n",
    "Each data source (MODIS, GLODAP, ARGO, RCA, ...) gets a dedicated sub-directory in `~/data`.\n",
    "\n",
    "\n",
    "`xarray` has a wildcard multi-file open utility: `xr.open_mfdataset(\"Identifier*.nc\")`.\n",
    "This maps multiple NetCDF files to a single Dataset. \n",
    "\n",
    "\n",
    "The RCA data are ordered using a less convenient dimension, namely\n",
    "observation number `obs`. This is just an ordinal integer 1, 2, 3, ...\n",
    "The code in this notebook modifies this to use dimension `time`.\n",
    "\n",
    "## Obtain Regional Cabled Array data using `yodapy`\n",
    "\n",
    "As noted above the `yodapy` library enables Python-based access to OOI data. In this case we will focus\n",
    "on the Regional Cabled Array (RCA) and particularly on the shallow profiler found at the site \n",
    "**Oregon Slope Base**. This site is at the base of the continental shelf in about 3000 meters of water.\n",
    "The shallow profiler rises and falls nine times per day through the upper 200 meters of the water column.\n",
    "\n",
    "\n",
    "### OOI data access back-story\n",
    "\n",
    "\n",
    "To order data from **OOI** requires you to pre-register (free, using your email address). This provides you \n",
    "credentials when placing a data order. Orders typically take a few minutes for the OOI\n",
    "servers to assemble; after which you receive an email with a download link. You download the data to local storage\n",
    "and read files into memory and proceed from there, a very labor-intensive process.\n",
    "\n",
    "\n",
    "### How `yodapy` helps\n",
    "\n",
    "\n",
    "[`yodapy`](http://github.com/cormorack/yodapy) helps you automate OOI data access at each step. \n",
    "It sets up a credentials directory within your home directory;\n",
    "and in so doing helps you avoid accidentally pushing your credentials to `github` where they would be public. `yodapy` \n",
    "allows you to create a Python object called an `OOI()` that includes methods for finding sensor data of interest; \n",
    "for ordering time-bounded datasets for those sensors; for downloading this data; and for attaching it to a data \n",
    "structure (an `xarray Dataset`) for further analysis. It is at this point when you have your data present as a \n",
    "`Dataset` that `yodapy` has completed its job. \n",
    "\n",
    "\n",
    "The next cell installs `yodapy`. Run this each time you start up this notebook server unless your installation\n",
    "of the `yodapy` library persists. \n",
    "\n",
    "\n",
    "### Getting OOI credentials\n",
    "\n",
    "\n",
    "To get data from OOI you first create a User account as follows:\n",
    "\n",
    "\n",
    "- Visit the [OOI website](https://ooinet.oceanobservatories.org/#)\n",
    "- On the login menu (upper right) select **Register**\n",
    "- Fill out the New User Registration Form\n",
    "- Once you have your login credentials: Log in\n",
    "- The 'Login' menu should be replaced with your User name at the upper right: Also a dropdown menu\n",
    "  - Use this menu to select User Profile\n",
    "- At the bottom of your User Profile page you should find **API Username** and **API Token**\n",
    "  - These two strings comprise your authentication \n",
    "  - Keep them somewhere safe\n",
    "  - Notice that the **Refresh API Token** button permits you to regenerate them whenever you like\n",
    "\n",
    "\n",
    "Use your OOI API Token with `yodapy` as described further down to automate your authentication process.\n",
    "If this works as intended you can safely use OOI and not have to worry about cutting and pasting these\n",
    "token strings every time you want to get data access.\n",
    "\n",
    "## install yodapy if needed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Jupyter Notebook running Python 3\n",
      "the data directory is /data/ \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# mini-source control: Last copied 29-SEP-2020: to tilt*, chlorophyll*, rca*, argo*\n",
    "#                      last revised 09-OCT-2020\n",
    "import os, sys, time, glob\n",
    "\n",
    "from IPython.display import clear_output             # use inside loop with clear_output(wait = True) followed by print(i)\n",
    "import warnings                                      # use with warnings.filterwarnings('ignore') or 'once'\n",
    "\n",
    "home_dir = os.getenv(\"HOME\")\n",
    "this_dir = home_dir + '/chlorophyll/'\n",
    "data_dir = '/data/'\n",
    "data1_dir = '/data1'\n",
    "\n",
    "from matplotlib import pyplot as plt\n",
    "from matplotlib import colors as mplcolors\n",
    "import numpy as np, pandas as pd, xarray as xr\n",
    "from numpy import datetime64 as dt64, timedelta64 as td64\n",
    "\n",
    "def doy(theDatetime): return 1 + int((theDatetime - dt64(str(theDatetime)[0:4] + '-01-01')) / td64(1, 'D')) # 1, 2, .... , 365, [366]\n",
    "def dt64_from_doy(year, doy): return dt64(str(year) + '-01-01') + td64(doy-1, 'D')\n",
    "def day_of_month_to_string(d): return str(d) if d > 9 else '0' + str(d)\n",
    "\n",
    "print('\\nJupyter Notebook running Python {}'.format(sys.version_info[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ensure that the latest build of yodapy is installed directly from github using\n",
    "!pip install git+https://github.com/cormorack/yodapy.git -q     # -q cuts the stdout clutter\n",
    "\n",
    "# this line of code verifies yodapy is installed\n",
    "from yodapy.utils.creds import set_credentials_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## One time only: Configure OOI credentials using `yodapy`\n",
    "\n",
    "Only the first time through here: Carefully follow the instructions in the Python cell below.\n",
    "You are (temporarily) telling `yodapy` what your `OOI username` and `token` are. \n",
    "`yodapy` creates a hard-to-notice sub-directory of your home directory\n",
    "that contains these credentials in a text file. As long as you are not publishing\n",
    "your home directory someplace public your credentials will be hidden away.\n",
    "\n",
    "\n",
    "#### 'Why am I doing this *credentials* business?'\n",
    "\n",
    "\n",
    "When you use `yodapy` to order data from OOI it will use this 'hidden away' copy\n",
    "of your credentials to convince OOI your order is legitimate.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run the next line of code to create authentication credentials for the OOI data system. Do this\n",
    "# by ***carefully**** substituting your actual credentials in the username and token strings\n",
    "# in this line of code:\n",
    "\n",
    "\n",
    "if False: \n",
    "    \n",
    "    set_credentials_file(data_source='ooi', username='OOIAPI-XXXXXXXXXXXXXX', token='XXXXXXXXXXXX')\n",
    "\n",
    "\n",
    "# Un-comment the code and run the cell, just the one line above.\n",
    "# Once it runs: Comment it out again and delete your credentials. You can obscure them with XXXXX as they are seen now.\n",
    "# After you obscure your credentials: Be sure not to run this code again as it will break your authentication info.\n",
    "#\n",
    "# You can verify this worked by examining the .credentials file in ~/.yodapy. The credentials should match. Notice that \n",
    "#   this (slightly hidden) directory is directly connected to your home directory; whereas this IPython notebook \n",
    "#   is presumably in a distinct directory; so there should be no chance of a GitHub push sending your \n",
    "#   credentials to GitHub. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regional Cabled Array data for 2019\n",
    "\n",
    "* 3 sites: OSB, AXB, OOE for Oregon Slope Base, Axial Base, Oregon Offshore Endurance\n",
    "* 3 Platforms: Shallow and Deep profilers plus shallow platform (fixed at 200m depth)\n",
    "* Large collection of instruments, each with one or more sensors\n",
    "  * CTD + Dissolved Oxygen\n",
    "  * PAR, Spectral Irradiance, Spectrophotometer (attenuation / absorbance), Fluorometers \n",
    "  * Nitrate, pH, pCO2\n",
    "  * Ocean velocity measurement\n",
    "\n",
    "## Initialize the `OOI()` object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from yodapy.datasources import OOI\n",
    "\n",
    "# uncomment this...\n",
    "# ooi = OOI()\n",
    "\n",
    "# use this \"no-underscore\" version of directory to see the primary methods for the OOI() object\n",
    "# dirnou(ooi)\n",
    "\n",
    "# uncomment and run this to see all the components or segments of OOI available\n",
    "# ooi.sites"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# yodapy\n",
    "# We can explore these methods and attributes further. Note that yodapy has a series of \n",
    "# attributes that begin with 'cava_'. 'cava' is shorthand for \"cabled array value add\", \n",
    "#   a project at the University of Washington School of Oceanography supporting cabled array\n",
    "#   data validation and use in ocean research.\n",
    "# help(ooi.cava_sites)\n",
    "ooi.cava_sites"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# yodapy\n",
    "print('\\n\\n\\n')\n",
    "ooi.cava_parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `ooi.search()` first example\n",
    "\n",
    "We will begin using `yodapy` proper to narrow down a data search. \n",
    "\n",
    "\n",
    "### What resources are available?\n",
    "\n",
    "\n",
    "Specifically what are the names of sites served by the Regional Cabled Array? \n",
    "We begin with a broad search giving only the keyword `region`. \n",
    "Then we narrow the search by adding including keywords `site`, `node`, and `instrument` \n",
    "to arrive at individual *instruments* or *sensors*. These search results are used to order \n",
    "datasets with a specified time range. \n",
    "\n",
    "\n",
    "This first example is the broad search. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ooi.search(region='endurance')\n",
    "ooi.search(region='cabled')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Attribute 'sites' gives broad results as a table of arrays, sites, descriptions, lat/lon: Across all of OOI (62 rows)\n",
    "ooi.sites    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Narrow result: Within the Cabled Array region only (118 rows, 6 named columns)\n",
    "print(ooi.instruments)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using `yodapy` to refine OOI searches\n",
    "\n",
    "The `OOI()` object provided by `yodapy` starts out in a very *broad view* search state. \n",
    "It sees the entire OOI project at the level of the observatory sites, by name \n",
    "Endurance, Pioneer, Argentine Basin, Cabled Array, Irminger Sea, Station Papa and possibly\n",
    "others I'm forgetting. \n",
    "\n",
    "\n",
    "The `ooi.search()` method uses keywords (`keyword='string-value'`) to narrow this view. \n",
    "In this way when the view is narrowed to a single instrument we can use the `ooi.request_data()`\n",
    "method to order data from that instrument. \n",
    "\n",
    "## `ooi.search()` second example and notes on search configuration\n",
    "\n",
    "We narrow the search using keywords `site`, `node` and `instrument`. \n",
    "The `ooi.instruments` result from above provides the vocabulary to use for keyword arguments: \n",
    "\n",
    "- `site` keyword is taken from the `site_name` column\n",
    "  - for example `Oregon Slope Base Seafloor` suggests using `oregon slope base` as the keyword value\n",
    "- `node` keyword is taken from the `infrastructure_name` column\n",
    "  - for example 'Shallow Profiler (SF01A)` suggests keyword `shallow profiler` (notice these are not case-sensitive)\n",
    "- `instrument` keyword is taken from the `instrument_name` column\n",
    "  - for example `3-Wavelength Fluorometer` suggests keyword `fluorometer`\n",
    "  \n",
    "\n",
    "\n",
    "Once the narrow search runs we look at the `ooi.instruments` attribute to see how narrow the results are.\n",
    "This prints as a table where -- as in example one -- the results are sorted into *one instrument per row*.\n",
    "This can confirm whether the objective of narrowing the search down to a single instrument was met.\n",
    "\n",
    "\n",
    "We run the `.data_availability()` method. This gives two outputs: A **table** and below that a \n",
    "**time series graphic**.  The table lists each instrument as a separate column. These columns are \n",
    "then transposed for the time series graphic: One row of boxes for each instrument. \n",
    "\n",
    "\n",
    "***Detail: The green `.data_availability()` chart may fail to render in some cases. Re-running the cell might help.***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# region='cabled' or 'endurance'\n",
    "# site='slope' or 'slope base deep' or 'oregon offshore' or ... \n",
    "# node='platform' or 'shallow profiler' or ...\n",
    "# instrument='2-Wavelength' or 'ctd' or 'fluorometer' or ...\n",
    "# ooi.search(region='endurance', site='oregon offshore', node='shallow profiler', instrument='fluorometer')\n",
    "\n",
    "# ooi.data_availability()\n",
    "\n",
    "# Taking out the instrument keyword we have: \n",
    "# ooi.search(region='cabled', site='slope', node='shallow profiler')\n",
    "# This produces (with a simple ooi.instruments attribute call) a list of the following:\n",
    "#   - 3-Wavelength fluorometer (flort: got it for OSB SP 2019)\n",
    "#   - CTD (ctdpf: got it for OSB SP 2019)\n",
    "#   - Photosynthetically Available Radiation (parad: got it for OSB SP 2019)\n",
    "#   - pH (phsen: got it for OSB SP 2019)\n",
    "#   - Spectral Irradiance (spkir: got)\n",
    "#   - Spectrophotometer (optaa: got)\n",
    "#   - NOT YET: Single Point Velocity Meter (velpt: )\n",
    "#   - Nitrate (nutnr: Got both nutnr_a_sample and nutnr_a_dark_sample)\n",
    "#   - pCO2 water (two streams: pco2w_a_sami_data_record and pco2w_b (no data past 2018; placed 2018 data\n",
    "\n",
    "# instrument                  2014    2015    2016    2017    2018    2019    2020\n",
    "#\n",
    "# Oregon Slope Base\n",
    "# SP flort 3-wavelength                                                !\n",
    "# SP ctdpf                                                             !\n",
    "# SP parad                                                             !\n",
    "# SP phsen                                                             !\n",
    "# SP spkir                                                             !\n",
    "# SP optaa                                                             !\n",
    "# SP velpt                                                             !\n",
    "# SP nutnr_a, nutnr_a_dark                                             !\n",
    "# SP pco2w_a_sami                                                      !\n",
    "# SP pco2w_b_sami                                               !      NA\n",
    "# 200m ctdpf                                                           !\n",
    "# 200m flort                                                           !\n",
    "# 200m phsen                                                           !\n",
    "# 200m do_stable                                                       !\n",
    "# DP ctdpf wfp                                  !      NA       NA     NA\n",
    "# DP ctdpf inst                                                        !\n",
    "# DP acm (VEL3D) inst                                                  !\n",
    "# DP flcdrdt inst fluorometer                                          !\n",
    "#\n",
    "# Axial Base\n",
    "# SP flort                                                             !\n",
    "# SP ctdpf                                                             !\n",
    "# SP parad                                                             !\n",
    "# SP phsen                                                             !\n",
    "# SP spkir                          ?\n",
    "# SP optaa                                                             !\n",
    "# SP velpt                          ?\n",
    "# SP nutnr_a, nutnr_a_dark          ?\n",
    "# SP pco2w_a_sami                                                      !\n",
    "# SP pco2w_b_sami                   ?\n",
    "# 200m ctdpf                                                           !\n",
    "# 200m flort                                                           !\n",
    "# 200m phsen                                                           !\n",
    "# 200m do_stable                                                       !\n",
    "# DP ctdpf wfp                                                         !\n",
    "# DP ctdpf inst                                                        !\n",
    "# DP acm (VEL3D) inst               ?\n",
    "# DP flcdrdt inst CDOM fluorometer                                     !\n",
    "# DP fl????? inst 2-Wav fluorometer ?\n",
    "# DP dissolved oxygen                                                  !\n",
    "# \n",
    "# filename anatomy\n",
    "# deployment0005                            or 0006 etc\n",
    "#   _RS03AXPS                               site: AX is Axial, SB is slope base\n",
    "#   -SF03A                                  platform: SF is shallow profiler, DP is deep profiler, PC is 200m platform \n",
    "#   -3B                                     number + letter: unknown\n",
    "#   -OPTAAD301                              6-letter instrument + 'A'/'D' + 30X/10X\n",
    "#   -streamed                               'streamed' or 'recovered_inst' or 'recovered_wfp'\n",
    "#   -optaa_sample                           instrument designator, sometimes 'dpc_xxxxx_instrument_recovered'\n",
    "#   _20191004T073957.414490                 datetime start\n",
    "#   -20191014T220233.907019                 datetime end\n",
    "# .nc                                       NetCDF file\n",
    "#\n",
    "# run this to see fluorometers available at Oregon Offshore (without using the 'node' keyword)\n",
    "# \n",
    "#   filters endurance + oregon offshore + fluorometer turn up 7 hits...\n",
    "#     2 are Oregon Offshore Surface Mooring: 3 wavelength... of future interest in expanding the MODIS connection\n",
    "#     2 are Oregon Offshore deep profiler CDOM fluorometer\n",
    "#     2 are Oregon Offshore deep profiler 2 wavelength...    of future interest also (not sure if this is on the RCA)\n",
    "#     1 is Oregon Offshore shallow profiler 3 wavelength     *** Current interest: RCA MODIS connect ***\n",
    "#\n",
    "# ooi.search(region='endurance', site='oregon offshore', instrument='fluorometer')\n",
    "# ooi.instruments\n",
    "\n",
    "# ooi.data_availability()\n",
    "\n",
    "# This ooi.search() call: \n",
    "# \n",
    "# ooi.search(region='cabled', instrument='fluorometer') \n",
    "# \n",
    "# produces 12 hits. Here is the breakdown; where results suggest site and node search keywords. \n",
    "#  Note that Deep Profiler sites have degeneracy in 'recovered_inst' versus 'recovered_wfp' (appear twice)\n",
    "# \n",
    "#     - (4) Axial Base Deep Profiler Mooring (CDOM Fluorometer,  2-Wavelength Fluorometer)\n",
    "#     - (4) Oregon Slope Base Deep Profiler Mooring (CDOM Fluorometer, 2-Wavelength Fluorometer)\n",
    "#     - (1) Oregon Slope Base Shallow Profiler Mooring (200m Platform; 2-Wavelength Fluorometer)\n",
    "#     - (1) Oregon Slope Base Shallow Profiler Mooring (Shallow Profiler; 3-Wavelength Fluorometer)\n",
    "#     - (1) Axial Base Shallow Profiler Mooring (200m Platform; 2-Wavelength Fluorometer)\n",
    "#     - (1) Axial Base Shallow Profiler Mooring (Shallow Profiler; 3-Wavelength Fluorometer)\n",
    "\n",
    "# Resulting searches: Choose one of these...\n",
    "# ooi.search(region='cabled', site='oregon slope base', node='shallow', instrument='fluorometer')\n",
    "# ooi.search(region='cabled', site='oregon slope base', node='200m', instrument='fluorometer')\n",
    "# ooi.search(region='cabled', site='axial base', node='shallow', instrument='fluorometer')\n",
    "# ooi.search(region='cabled', site='axial base', node='200m', instrument='fluorometer')\n",
    "\n",
    "# ...and run...\n",
    "# ooi.data_availability()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Final `yodapy` section: Obtain data\n",
    "\n",
    "Three useful Python/yodapy cells follow. The first is run iteratively to refine a \n",
    "search. The second and third are run consecutively to place an order and download it\n",
    "when it is ready.\n",
    "\n",
    "\n",
    "> ***Strongly suggest: After data arrives move it to an out-of-repo data directory***\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "###################################\n",
    "# \n",
    "# Set up a data order\n",
    "#\n",
    "# Use this cell (possibly multiple times) to narrow your search. Then use the following two cells to \n",
    "#   order and retrieve this data. Remember that the last thing you used for .search() is stored inside\n",
    "#   the ooi object as its 'state'. This makes the ordering of the data in the next cell much simpler\n",
    "#   because you only need to specify the time range. \n",
    "#\n",
    "# What keywords does a Python method take?\n",
    "#\n",
    "# Below the ooi.search() method is provided with keywords; but how to discover what these are?\n",
    "# Answer: Enter ?ooi.search to read the document string.\n",
    "#\n",
    "# Instructions for this cell\n",
    "#\n",
    "# Notice below there is an ooi.search() method call. This uses a sequence of keywords with values \n",
    "#   to narrow the focus of your search. The next line of code 'ooi.instrument' will print the \n",
    "#   results of this search as a table. In a simplest case if your resulting table is exactly one\n",
    "#   row then your will be ordering data from just that instrument stream. \n",
    "# \n",
    "# A more advanced approach is to order multiple data streams at once; but this is not described here.\n",
    "#\n",
    "# First step: Run the search with only 'region', 'site', and 'node' keywords. Do not include  \n",
    "#   'instrument' or 'stream' keywords. This gives a results table with multiple instruments (rows).\n",
    "#   Example: ooi.search(region='cabled', site='slope', node='shallow profiler')\n",
    "#\n",
    "# Second step: Refine this search by including 'instrument' or 'stream' keywords. Give corresponding\n",
    "#   values from the results table to get a new results table with just one row for the instrument \n",
    "#   you are interested in. \n",
    "#   Example: ooi.search(region='cabled', site='slope', node='shallow profiler', stream='velpt')\n",
    "#\n",
    "# At this point the search parameters in the OOI() object can be used for a focused data order.\n",
    "# Place this order using the subsequent two cells.\n",
    "\n",
    "# run `?ooi.search` to see available keywords\n",
    "ooi.search(region='endurance', site='offshore', node='200', stream = 'optode, do_stable, phsen,pco2')\n",
    "ooi.instruments\n",
    "# ooi.data_availability()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "# 2019 CTD (9 months are available ) required ~4 mins. Other sensors closer to 20 minutes\n",
    "\n",
    "# Assume the above cell narrowed the search results to a single instrument. Assume also that we \n",
    "#   are interested in June 1 -- September 15 of 2019. We now use the ooi object to generate a \n",
    "#   data request.\n",
    "#\n",
    "# .request_data() generates a data request\n",
    "# .to_xarray() polls the OOI system until the order completes; this will take a couple of minutes\n",
    "#\n",
    "begin_date = '2019-01-01'\n",
    "end_date = '2020-01-01'\n",
    "ooi.request_data(begin_date=begin_date, end_date=end_date)\n",
    "ds = ooi.to_xarray()\n",
    "len(ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "# run this to download the data (possibly multiple files) from a completed data request from above\n",
    "#   one year can take between 4 and 10 minutes\n",
    "# \n",
    "filenamelist = ooi.download_netcdfs()\n",
    "len(filenamelist)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ooi.raw()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(ooi.raw())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Two problems with this data\n",
    "\n",
    "- The data order tends to yield multiple files that are contiguous in time. For example\n",
    "The first might run June 1 to June 27 and the second might run June 28 to July 10. We\n",
    "would like to consider them as a single Dataset and fortunately this is built into the\n",
    "XArray package as a method: `.open_mfdataset()`. Here the `mf` abbreviates *multiple files*. \n",
    "\n",
    "\n",
    "```\n",
    "ds=xr.open_mfdataset(...filename description string including wildcard...)\n",
    "```\n",
    "\n",
    "\n",
    "- The data are ordered by a dimension called `obs` for *observation number*. This runs\n",
    "`1, 2, 3, ...` for each data *file*. The coordinate `time` is available as a dependent\n",
    "coordinate; but to combine multiple files into a single dataset we do not want to have\n",
    "`obs = 1, 2, 3, ..., 7010, 7011, 1, 2, 3, ...` with redundant observations. We simply\n",
    "want everything related to a `time` dimension that increases monotonically as all the\n",
    "data are combined. For this we use the XArray Dataset `.swap_dims()` method which is\n",
    "passed a small dictionary that articulates how the swap will happen. \n",
    "\n",
    "\n",
    "```\n",
    "ds = ds.swap_dims({'obs':'time'})\n",
    "```\n",
    "\n",
    "These two commands are orchestrated together by means of a *preprocessor* function.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save streamlined datasets\n",
    "\n",
    "The following cell opens \"multi-file\" datasets. It uses short 'good stuff' lists to preserve\n",
    "important information and dump everything else: Across dimensions, coordinates, data variables \n",
    "and attributes. It then writes these simplified Datasets as NetCDF files.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "def load_and_save_streamlined(source, output, keep_dims, keep_coords, keep_data_vars, keep_attrs):\n",
    "\n",
    "    def lass_preprocessor(fds):                 # per-file datasets have dimension 'obs'\n",
    "        return fds.swap_dims({'obs':'time'})    #   ...so we pre-swap that for time\n",
    "\n",
    "    ds = xr.open_mfdataset(data_dir + source, preprocess = lass_preprocessor, concat_dim='time', combine='by_coords')\n",
    "    for key in ds.dims: \n",
    "        if key not in keep_dims: ds = ds.drop_dims(key)\n",
    "    for key in ds.coords: \n",
    "        if key not in keep_coords: ds = ds.drop(key)\n",
    "    for key in ds.data_vars: \n",
    "        if key not in keep_data_vars: ds = ds.drop(key)\n",
    "    attrs_dict = ds.attrs.copy()         \n",
    "    for key in attrs_dict: \n",
    "        if key not in keep_attrs: ds.attrs.pop(key)\n",
    "    ds.to_netcdf(data_dir + output)\n",
    "    return ds\n",
    "\n",
    "strRoot = 'rca/2019/depl*'\n",
    "strSite = 'SBPS*'\n",
    "strPlatform = 'SF*'\n",
    "\n",
    "# particular to phsen the pH sensor\n",
    "ds_phsen = load_and_save_streamlined(strRoot + strSite + strPlatform + 'phsen*.nc', \n",
    "                                    'rca/simpler/osb_sp_phsen_2019.nc', \n",
    "                                    ['time'],\n",
    "                                    ['time', 'int_ctd_pressure'],\n",
    "                                    ['ph_seawater'],\n",
    "                                    ['node', 'id', 'geospatial_lat_min', 'geospatial_lon_min'])\n",
    "\n",
    "ds_nutnr_a_dark = load_and_save_streamlined(strRoot + strSite + strPlatform + 'nutnr_a_dark*.nc', \n",
    "                                    'rca/simpler/osb_sp_nutnr_a_dark_2019.nc', \n",
    "                                    ['time', 'wavelength'],\n",
    "                                    ['time', 'int_ctd_pressure'],\n",
    "                                    ['nitrate_concentration'],\n",
    "                                    ['node', 'id', 'geospatial_lat_min', 'geospatial_lon_min'])\n",
    "\n",
    "ds_nutnr_a_sample = load_and_save_streamlined(strRoot + strSite + strPlatform + 'nutnr_a_sample*.nc', \n",
    "                                    'rca/simpler/osb_sp_nutnr_a_sample_2019.nc', \n",
    "                                    ['time', 'wavelength'],\n",
    "                                    ['time', 'int_ctd_pressure'],\n",
    "                                    ['nitrate_concentration'],\n",
    "                                    ['node', 'id', 'geospatial_lat_min', 'geospatial_lon_min'])\n",
    "\n",
    "\n",
    "# ds_ctdpf = load_and_save_streamlined('rca/2019/depl*ctdpf*.nc', 'rca/simpler/osb_sp_ctdpf_2019.nc', \n",
    "#                                     ['time'],\n",
    "#                                     ['time', 'seawater_pressure'],\n",
    "#                                     ['seawater_temperature', 'practical_salinity', \n",
    "#                                      'corrected_dissolved_oxygen', 'density'],\n",
    "#                                     ['node', 'id', 'geospatial_lat_min', 'geospatial_lon_min'])\n",
    "\n",
    "# ds_parad = load_and_save_streamlined('rca/2019/depl*parad*.nc', 'rca/simpler/osb_sp_parad_2019.nc', \n",
    "#                                     ['time'],\n",
    "#                                     ['time', 'int_ctd_pressure'],\n",
    "#                                     ['par_counts_output'],\n",
    "#                                     ['node', 'id', 'geospatial_lat_min', 'geospatial_lon_min'])\n",
    "\n",
    "# ds_spkir = load_and_save_streamlined('rca/2019/depl*spkir*.nc', 'rca/simpler/osb_sp_spkir_2019.nc', \n",
    "#                                     ['time', 'spectra'],\n",
    "#                                     ['time', 'int_ctd_pressure'],\n",
    "#                                     ['spkir_downwelling_vector'],\n",
    "#                                     ['node', 'id', 'geospatial_lat_min', 'geospatial_lon_min'])\n",
    "\n",
    "# ds_optaa = load_and_save_streamlined('rca/2019/depl*optaa*.nc', 'rca/simpler/osb_sp_optaa_2019.nc', \n",
    "#                                     ['time', 'wavelength'],\n",
    "#                                     ['time', 'wavelength', 'int_ctd_pressure'],\n",
    "#                                     ['beam_attenuation', 'optical_absorption'],\n",
    "#                                     ['node', 'id', 'geospatial_lat_min', 'geospatial_lon_min'])\n",
    "\n",
    "# ds_flort = load_and_save_streamlined('rca/2019/depl*flort*.nc', 'rca/simpler/osb_sp_flort_2019.nc', \n",
    "#                                      ['time'],\n",
    "#                                      ['time', 'int_ctd_pressure'],\n",
    "#                                      ['fluorometric_chlorophyll_a', 'fluorometric_cdom', \n",
    "#                                       'total_volume_scattering_coefficient', 'seawater_scattering_coefficient', \n",
    "#                                       'optical_backscatter'],\n",
    "#                                      ['node', 'id', 'geospatial_lat_min', 'geospatial_lon_min'])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}