04_more_erddap.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "159b6f7e-774d-4b68-82bf-4312285553b6",
   "metadata": {},
   "source": [
    "# Miscellaneous ERDDAP functionality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "818972bc-f6c3-4855-8cb6-23162f5077cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "from erddapy import ERDDAP\n",
    "import matplotlib.pyplot as plt\n",
    "import cartopy.crs as ccrs\n",
    "import types\n",
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81a7290b-4d44-4ab5-b2b6-5028fb5c8513",
   "metadata": {},
   "source": [
    "### Get metadata\n",
    "\n",
    "To see detailed metadata from a dataset, we request the **ncCF** format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5cf921a1-2408-42ff-a116-4270012651b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "e = ERDDAP(\n",
    "    server=\"https://erddap.observations.voiceoftheocean.org/erddap\",\n",
    "    protocol=\"tabledap\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dee1788f-24cf-4f6f-a2b9-6a7ed518e6fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_id = \"nrt_SEA068_M27\"\n",
    "e.dataset_id = dataset_id\n",
    "meta = e.to_ncCF()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae86854c-744d-4536-b338-fe544e527790",
   "metadata": {},
   "outputs": [],
   "source": [
    "attrs = {}\n",
    "for key_name in dir(meta):\n",
    "    if key_name[0] == \"_\":\n",
    "        continue\n",
    "    val =  meta.__getattribute__(key_name)\n",
    "    if type(val) is types.MethodType:\n",
    "        continue\n",
    "    attrs[key_name] = val"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa5f211e-9eb6-47ac-8aeb-96303ce6200b",
   "metadata": {},
   "source": [
    "### Display vector data\n",
    "\n",
    "Using some HF Radar from coastwatch\n",
    "\n",
    "https://coastwatch.pfeg.noaa.gov/erddap/griddap/ucsdHfrW1.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f81514e-fbe1-4b86-a5e9-929486e506fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "coastwatch_e = ERDDAP(\n",
    "    server=\"https://coastwatch.pfeg.noaa.gov/erddap\",\n",
    "    protocol=\"griddap\",\n",
    ")\n",
    "coastwatch_e.dataset_id='ucsdHfrW1'\n",
    "coastwatch_e.griddap_initialize() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3910014-4624-4aae-a2bd-fd128c4e7142",
   "metadata": {},
   "outputs": [],
   "source": [
    "coastwatch_e.constraints"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c51e47ab-d1b3-4457-a01f-37f0b9a249ec",
   "metadata": {},
   "source": [
    "Coarsen by 10'000 and make a quick plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee4d140e-de9a-4c9d-acaa-9f8e04ee288e",
   "metadata": {},
   "outputs": [],
   "source": [
    "coastwatch_e['longitude_step']: 100\n",
    "coastwatch_e['latitude_step']: 100\n",
    "ds = coastwatch_e.to_xarray()\n",
    "ds.water_u.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2517594d-f531-41c3-9419-a33c4af8a4a4",
   "metadata": {},
   "source": [
    "Constrain the data to a smaller region for plotting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e019efd4-2b1f-4f15-be2f-723c377a80c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "coastwatch_e.constraints = {'time>=': '2023-10-17T12:00:00Z',\n",
    " 'time<=': '2023-10-17T12:00:00Z',\n",
    " 'time_step': 1,\n",
    " 'latitude>=':  48.9,\n",
    " 'latitude<=':  49.35,\n",
    " 'latitude_step': 1,                            \n",
    " 'longitude>=': -123.6,\n",
    " 'longitude<=': -123,\n",
    " 'longitude_step': 1}\n",
    "ds = coastwatch_e.to_xarray()\n",
    "ds.water_u.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1f73013-ea21-4bb4-84ac-6d2b3df203b8",
   "metadata": {},
   "source": [
    "Use [matplotlib quiver](https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.quiver.html) to plot surface velocity vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1beb1b0-e356-4bdb-9fc9-68357879d3cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize=[8, 8])\n",
    "\n",
    "ax = fig.add_subplot(projection=ccrs.AzimuthalEquidistant(central_latitude=49, central_longitude=-123))\n",
    "\n",
    "q = ax.quiver(ds.longitude, ds.latitude, ds.water_u.values[0, :,:],\n",
    "          ds.water_v.values[0, :, :], scale=20, transform=ccrs.PlateCarree())\n",
    "q = ax.quiverkey(q, 0.8, 0.9, 1, r'1 m s$^{-1}$', labelpos='E',\n",
    "                    transform=ccrs.PlateCarree())\n",
    "ax.gridlines(color='k', alpha=0.2)\n",
    "\n",
    "ax.set(title=ds.attrs['title']);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4219e71-dc45-4a5e-a38f-d1a9bdf28e54",
   "metadata": {},
   "source": [
    "### Display satellite data\n",
    "\n",
    "Pathfinder SST\n",
    "\n",
    "https://coastwatch.pfeg.noaa.gov/erddap/griddap/erdPH53sstdmday.graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de608b69-e651-459b-aa9f-a1d65d69c0a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "satellite_e = ERDDAP(\n",
    "    server=\"https://coastwatch.pfeg.noaa.gov/erddap\",\n",
    "    protocol=\"griddap\",\n",
    ")\n",
    "satellite_e.dataset_id='erdPH53sstdmday'\n",
    "satellite_e.griddap_initialize() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "338e2aa2-ceec-41b5-b32e-35c493d48037",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"variables: \", satellite_e.variables)\n",
    "satellite_e.constraints"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33663303-36e1-43b8-8c86-38c421c6274b",
   "metadata": {},
   "outputs": [],
   "source": [
    "satellite_e.variables = ['sea_surface_temperature']\n",
    "satellite_e.constraints['latitude_step'] = 100\n",
    "satellite_e.constraints['longitude_step'] = 100\n",
    "sat = satellite_e.to_xarray()\n",
    "sat.sea_surface_temperature.plot(figsize=(10, 6), vmin=-5, vmax=35, cmap=\"RdBu_r\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53871ecd-4e96-4606-848c-4eed7ecbe79a",
   "metadata": {},
   "source": [
    "### Get data into matlab/R\n",
    "\n",
    "Using the xarray and pandas objects from the previous notebooks, we can write out our datasets as **.csv** or **.nc** files that can be read by a range of programs\n",
    "\n",
    "**NOTE FOR FUTURE**\n",
    "\n",
    "Funtionality to download arbitrary filetypes has been added to erddapy in PR #330 https://github.com/ioos/erddapy/pull/330\n",
    "\n",
    "This will be available in the next release\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cf44e0a-a553-4999-9704-bac1b62ea05e",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = e.to_pandas()\n",
    "df.to_csv(f\"{dataset_id}.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96dc80b1-77af-46fd-949a-de4b0f4b8364",
   "metadata": {},
   "source": [
    "There is no matlab client for ERDDAP that I am aware of, but you can use Python to download matlab data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29d9f7fb-d479-474c-a7fe-b5b05e74e705",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Do all you usual edits here for variables, limits etc.\n",
    "url = e.get_download_url()\n",
    "matlab_url = url.replace(\"html\", \"mat\")\n",
    "matlab_url"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6086847-a4e0-49f8-844e-96d9704fb145",
   "metadata": {},
   "source": [
    "We use requests to write out to **.mat**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13aa155b-209a-42fe-aef7-2f31dbcc354b",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get(matlab_url)\n",
    "with open(f\"{dataset_id}.mat\", \"wb\") as fout:\n",
    "    fout.write(response.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c5d866c-50b8-4d81-8175-54f46a4ab254",
   "metadata": {},
   "outputs": [],
   "source": [
    "import scipy.io\n",
    "mat = scipy.io.loadmat(f\"{dataset_id}.mat\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50629b82-7bda-4ce4-bff8-2f0b03298ddc",
   "metadata": {},
   "source": [
    "### Or use an ERDDAP that serves native .mat files 🤷\n",
    "http://erddap.oleander.bios.edu:8080/erddap/files/oleanderXbtMatFiles/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dd89613-23d6-4db0-aa14-a750ba8a78d5",
   "metadata": {},
   "source": [
    "See notebook 05 for using R. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc01b753-1cbd-4b53-913f-c669133414ff",
   "metadata": {},
   "source": [
    "### ERDDAP visitors analysis\n",
    "\n",
    "See notebooks at https://github.com/callumrollo/website-log-parse/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44d48329-a49c-412e-b416-f9350516ff04",
   "metadata": {},
   "source": [
    "### Computation and data handling in a cloud environment\n",
    "\n",
    "The Littlest JupyterHub https://tljh.jupyter.org/en/latest/index.html\n",
    "\n",
    "- Cohost your dataq with your processing\n",
    "- Set up accounts for users "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5371242-986a-41f3-8c95-54f60bd40fe2",
   "metadata": {},
   "source": [
    "---------------------------\n",
    "### References\n",
    "\n",
    "Quiver example modified from https://matplotlib.org/stable/gallery/images_contours_and_fields/quiver_demo.html\n",
    "\n",
    "Read matlab files into python https://stackoverflow.com/questions/874461/read-mat-files-in-python"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}