tutorial/examples/notebooks/save_to_cesnet_bucket_zarr_s3fs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "several-banking",
   "metadata": {},
   "source": [
    "# Save files to EOSC (CESNET)\n",
    "\n",
    "\n",
    "- This example show you how to save your zarr file to  object storage https://object-store.cloud.muni.cz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "another-shelf",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import s3fs\n",
    "import xarray as xr\n",
    "import zarr"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "simple-uzbekistan",
   "metadata": {},
   "source": [
    "## Get a sample file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "heard-queen",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds = xr.tutorial.open_dataset(\"air_temperature.nc\").rename({\"air\": \"Tair\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "quick-style",
   "metadata": {},
   "source": [
    "## Save your results to Remote object storage\n",
    "- If not done, create your credentials by follwoing [this link](../EOSC_to_bucket.md)\n",
    "- Verify your credentials in `$HOME/.aws/credentials` \n",
    "It should look like \n",
    "\n",
    "```\n",
    "[default]\n",
    "aws_access_key_id=xxxxx\n",
    "aws_secret_access_key=yyyy\n",
    "aws_endpoint_url=https://object-store.cloud.muni.cz\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecef7710-dd4b-4bb4-9d26-f56921645181",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-danger alert-info\">\n",
    "    <b>It is important to save your results in 'your' bucket.  [The credential created here ](../EOSC_to_bucket.md) is a common space for pangeo-eosc cloud users.  You need to not to 'over write' data on other users</b>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cdeab0c-ca53-4cd8-9dd8-bccb8d1f58c0",
   "metadata": {},
   "source": [
    "#### Define your s3 storage parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b72337a1-e0e4-476f-9561-587198cb985c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3://tmp/tinaok\n"
     ]
    }
   ],
   "source": [
    "your_name='tinaok'\n",
    "path='tmp/'+your_name\n",
    "s3_prefix =  \"s3://\"+path\n",
    "print(s3_prefix)\n",
    "access_key = !aws configure get aws_access_key_id\n",
    "access_key = access_key[0]\n",
    "secret_key = !aws configure get aws_secret_access_key\n",
    "secret_key = secret_key[0]\n",
    "client_kwargs={'endpoint_url': 'https://object-store.cloud.muni.cz'}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02834a9c-228d-4031-9410-214aa108ed8e",
   "metadata": {},
   "source": [
    "## set your zarr store (with S3Map)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "363169ee-7c47-4583-961c-9f95482d53f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "zarr_file_name= 'S3Map_zarr'\n",
    "s3 = s3fs.S3FileSystem(client_kwargs=client_kwargs,key=access_key, secret=secret_key)\n",
    "uri = f\"{s3_prefix}/{zarr_file_name}\"\n",
    "store_s3= s3fs.S3Map(root=uri,s3=s3,check=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e33cb553-df83-49d7-9d27-c7d2348e3c6e",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Write your file down to zarr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "eight-coach",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/core/dataset.py:2060: SerializationWarning: saving variable None with floating point data as an integer dtype without any _FillValue to use for NaNs\n",
      "  return to_zarr(  # type: ignore\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 865 ms, sys: 154 ms, total: 1.02 s\n",
      "Wall time: 52.4 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<xarray.backends.zarr.ZarrStore at 0x7fb264c9ca50>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%time ds.to_zarr(store=store_s3, mode='w', consolidated=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b8bbdd1-63dd-49d2-ad6e-deefa54a2660",
   "metadata": {},
   "source": [
    "#### Verify that your 'zarr' file is at your storage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "99cd5138-d8b3-408d-8bfc-61915798a25d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['tmp/tinaok/S3Map_zarr']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target = s3fs.S3FileSystem(anon=False,client_kwargs=client_kwargs)\n",
    "target.ls(path)#, detail=True, refresh=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52aa8b67-7525-498d-8db2-04233b17051b",
   "metadata": {},
   "source": [
    "#### Open your zarr storage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "096e617d-b8af-40b8-9229-0090c7342de6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><svg style=\"position: absolute; width: 0; height: 0; overflow: hidden\">\n",
       "<defs>\n",
       "<symbol id=\"icon-database\" viewBox=\"0 0 32 32\">\n",
       "<path d=\"M16 0c-8.837 0-16 2.239-16 5v4c0 2.761 7.163 5 16 5s16-2.239 16-5v-4c0-2.761-7.163-5-16-5z\"></path>\n",
       "<path d=\"M16 17c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z\"></path>\n",
       "<path d=\"M16 26c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z\"></path>\n",
       "</symbol>\n",
       "<symbol id=\"icon-file-text2\" viewBox=\"0 0 32 32\">\n",
       "<path d=\"M28.681 7.159c-0.694-0.947-1.662-2.053-2.724-3.116s-2.169-2.030-3.116-2.724c-1.612-1.182-2.393-1.319-2.841-1.319h-15.5c-1.378 0-2.5 1.121-2.5 2.5v27c0 1.378 1.122 2.5 2.5 2.5h23c1.378 0 2.5-1.122 2.5-2.5v-19.5c0-0.448-0.137-1.23-1.319-2.841zM24.543 5.457c0.959 0.959 1.712 1.825 2.268 2.543h-4.811v-4.811c0.718 0.556 1.584 1.309 2.543 2.268zM28 29.5c0 0.271-0.229 0.5-0.5 0.5h-23c-0.271 0-0.5-0.229-0.5-0.5v-27c0-0.271 0.229-0.5 0.5-0.5 0 0 15.499-0 15.5 0v7c0 0.552 0.448 1 1 1h7v19.5z\"></path>\n",
       "<path d=\"M23 26h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z\"></path>\n",
       "<path d=\"M23 22h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z\"></path>\n",
       "<path d=\"M23 18h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z\"></path>\n",
       "</symbol>\n",
       "</defs>\n",
       "</svg>\n",
       "<style>/* CSS stylesheet for displaying xarray objects in jupyterlab.\n",
       " *\n",
       " */\n",
       "\n",
       ":root {\n",
       "  --xr-font-color0: var(--jp-content-font-color0, rgba(0, 0, 0, 1));\n",
       "  --xr-font-color2: var(--jp-content-font-color2, rgba(0, 0, 0, 0.54));\n",
       "  --xr-font-color3: var(--jp-content-font-color3, rgba(0, 0, 0, 0.38));\n",
       "  --xr-border-color: var(--jp-border-color2, #e0e0e0);\n",
       "  --xr-disabled-color: var(--jp-layout-color3, #bdbdbd);\n",
       "  --xr-background-color: var(--jp-layout-color0, white);\n",
       "  --xr-background-color-row-even: var(--jp-layout-color1, white);\n",
       "  --xr-background-color-row-odd: var(--jp-layout-color2, #eeeeee);\n",
       "}\n",
       "\n",
       "html[theme=dark],\n",
       "body[data-theme=dark],\n",
       "body.vscode-dark {\n",
       "  --xr-font-color0: rgba(255, 255, 255, 1);\n",
       "  --xr-font-color2: rgba(255, 255, 255, 0.54);\n",
       "  --xr-font-color3: rgba(255, 255, 255, 0.38);\n",
       "  --xr-border-color: #1F1F1F;\n",
       "  --xr-disabled-color: #515151;\n",
       "  --xr-background-color: #111111;\n",
       "  --xr-background-color-row-even: #111111;\n",
       "  --xr-background-color-row-odd: #313131;\n",
       "}\n",
       "\n",
       ".xr-wrap {\n",
       "  display: block !important;\n",
       "  min-width: 300px;\n",
       "  max-width: 700px;\n",
       "}\n",
       "\n",
       ".xr-text-repr-fallback {\n",
       "  /* fallback to plain text repr when CSS is not injected (untrusted notebook) */\n",
       "  display: none;\n",
       "}\n",
       "\n",
       ".xr-header {\n",
       "  padding-top: 6px;\n",
       "  padding-bottom: 6px;\n",
       "  margin-bottom: 4px;\n",
       "  border-bottom: solid 1px var(--xr-border-color);\n",
       "}\n",
       "\n",
       ".xr-header > div,\n",
       ".xr-header > ul {\n",
       "  display: inline;\n",
       "  margin-top: 0;\n",
       "  margin-bottom: 0;\n",
       "}\n",
       "\n",
       ".xr-obj-type,\n",
       ".xr-array-name {\n",
       "  margin-left: 2px;\n",
       "  margin-right: 10px;\n",
       "}\n",
       "\n",
       ".xr-obj-type {\n",
       "  color: var(--xr-font-color2);\n",
       "}\n",
       "\n",
       ".xr-sections {\n",
       "  padding-left: 0 !important;\n",
       "  display: grid;\n",
       "  grid-template-columns: 150px auto auto 1fr 20px 20px;\n",
       "}\n",
       "\n",
       ".xr-section-item {\n",
       "  display: contents;\n",
       "}\n",
       "\n",
       ".xr-section-item input {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       ".xr-section-item input + label {\n",
       "  color: var(--xr-disabled-color);\n",
       "}\n",
       "\n",
       ".xr-section-item input:enabled + label {\n",
       "  cursor: pointer;\n",
       "  color: var(--xr-font-color2);\n",
       "}\n",
       "\n",
       ".xr-section-item input:enabled + label:hover {\n",
       "  color: var(--xr-font-color0);\n",
       "}\n",
       "\n",
       ".xr-section-summary {\n",
       "  grid-column: 1;\n",
       "  color: var(--xr-font-color2);\n",
       "  font-weight: 500;\n",
       "}\n",
       "\n",
       ".xr-section-summary > span {\n",
       "  display: inline-block;\n",
       "  padding-left: 0.5em;\n",
       "}\n",
       "\n",
       ".xr-section-summary-in:disabled + label {\n",
       "  color: var(--xr-font-color2);\n",
       "}\n",
       "\n",
       ".xr-section-summary-in + label:before {\n",
       "  display: inline-block;\n",
       "  content: '►';\n",
       "  font-size: 11px;\n",
       "  width: 15px;\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       ".xr-section-summary-in:disabled + label:before {\n",
       "  color: var(--xr-disabled-color);\n",
       "}\n",
       "\n",
       ".xr-section-summary-in:checked + label:before {\n",
       "  content: '▼';\n",
       "}\n",
       "\n",
       ".xr-section-summary-in:checked + label > span {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       ".xr-section-summary,\n",
       ".xr-section-inline-details {\n",
       "  padding-top: 4px;\n",
       "  padding-bottom: 4px;\n",
       "}\n",
       "\n",
       ".xr-section-inline-details {\n",
       "  grid-column: 2 / -1;\n",
       "}\n",
       "\n",
       ".xr-section-details {\n",
       "  display: none;\n",
       "  grid-column: 1 / -1;\n",
       "  margin-bottom: 5px;\n",
       "}\n",
       "\n",
       ".xr-section-summary-in:checked ~ .xr-section-details {\n",
       "  display: contents;\n",
       "}\n",
       "\n",
       ".xr-array-wrap {\n",
       "  grid-column: 1 / -1;\n",
       "  display: grid;\n",
       "  grid-template-columns: 20px auto;\n",
       "}\n",
       "\n",
       ".xr-array-wrap > label {\n",
       "  grid-column: 1;\n",
       "  vertical-align: top;\n",
       "}\n",
       "\n",
       ".xr-preview {\n",
       "  color: var(--xr-font-color3);\n",
       "}\n",
       "\n",
       ".xr-array-preview,\n",
       ".xr-array-data {\n",
       "  padding: 0 5px !important;\n",
       "  grid-column: 2;\n",
       "}\n",
       "\n",
       ".xr-array-data,\n",
       ".xr-array-in:checked ~ .xr-array-preview {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       ".xr-array-in:checked ~ .xr-array-data,\n",
       ".xr-array-preview {\n",
       "  display: inline-block;\n",
       "}\n",
       "\n",
       ".xr-dim-list {\n",
       "  display: inline-block !important;\n",
       "  list-style: none;\n",
       "  padding: 0 !important;\n",
       "  margin: 0;\n",
       "}\n",
       "\n",
       ".xr-dim-list li {\n",
       "  display: inline-block;\n",
       "  padding: 0;\n",
       "  margin: 0;\n",
       "}\n",
       "\n",
       ".xr-dim-list:before {\n",
       "  content: '(';\n",
       "}\n",
       "\n",
       ".xr-dim-list:after {\n",
       "  content: ')';\n",
       "}\n",
       "\n",
       ".xr-dim-list li:not(:last-child):after {\n",
       "  content: ',';\n",
       "  padding-right: 5px;\n",
       "}\n",
       "\n",
       ".xr-has-index {\n",
       "  font-weight: bold;\n",
       "}\n",
       "\n",
       ".xr-var-list,\n",
       ".xr-var-item {\n",
       "  display: contents;\n",
       "}\n",
       "\n",
       ".xr-var-item > div,\n",
       ".xr-var-item label,\n",
       ".xr-var-item > .xr-var-name span {\n",
       "  background-color: var(--xr-background-color-row-even);\n",
       "  margin-bottom: 0;\n",
       "}\n",
       "\n",
       ".xr-var-item > .xr-var-name:hover span {\n",
       "  padding-right: 5px;\n",
       "}\n",
       "\n",
       ".xr-var-list > li:nth-child(odd) > div,\n",
       ".xr-var-list > li:nth-child(odd) > label,\n",
       ".xr-var-list > li:nth-child(odd) > .xr-var-name span {\n",
       "  background-color: var(--xr-background-color-row-odd);\n",
       "}\n",
       "\n",
       ".xr-var-name {\n",
       "  grid-column: 1;\n",
       "}\n",
       "\n",
       ".xr-var-dims {\n",
       "  grid-column: 2;\n",
       "}\n",
       "\n",
       ".xr-var-dtype {\n",
       "  grid-column: 3;\n",
       "  text-align: right;\n",
       "  color: var(--xr-font-color2);\n",
       "}\n",
       "\n",
       ".xr-var-preview {\n",
       "  grid-column: 4;\n",
       "}\n",
       "\n",
       ".xr-var-name,\n",
       ".xr-var-dims,\n",
       ".xr-var-dtype,\n",
       ".xr-preview,\n",
       ".xr-attrs dt {\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "  padding-right: 10px;\n",
       "}\n",
       "\n",
       ".xr-var-name:hover,\n",
       ".xr-var-dims:hover,\n",
       ".xr-var-dtype:hover,\n",
       ".xr-attrs dt:hover {\n",
       "  overflow: visible;\n",
       "  width: auto;\n",
       "  z-index: 1;\n",
       "}\n",
       "\n",
       ".xr-var-attrs,\n",
       ".xr-var-data {\n",
       "  display: none;\n",
       "  background-color: var(--xr-background-color) !important;\n",
       "  padding-bottom: 5px !important;\n",
       "}\n",
       "\n",
       ".xr-var-attrs-in:checked ~ .xr-var-attrs,\n",
       ".xr-var-data-in:checked ~ .xr-var-data {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       ".xr-var-data > table {\n",
       "  float: right;\n",
       "}\n",
       "\n",
       ".xr-var-name span,\n",
       ".xr-var-data,\n",
       ".xr-attrs {\n",
       "  padding-left: 25px !important;\n",
       "}\n",
       "\n",
       ".xr-attrs,\n",
       ".xr-var-attrs,\n",
       ".xr-var-data {\n",
       "  grid-column: 1 / -1;\n",
       "}\n",
       "\n",
       "dl.xr-attrs {\n",
       "  padding: 0;\n",
       "  margin: 0;\n",
       "  display: grid;\n",
       "  grid-template-columns: 125px auto;\n",
       "}\n",
       "\n",
       ".xr-attrs dt,\n",
       ".xr-attrs dd {\n",
       "  padding: 0;\n",
       "  margin: 0;\n",
       "  float: left;\n",
       "  padding-right: 10px;\n",
       "  width: auto;\n",
       "}\n",
       "\n",
       ".xr-attrs dt {\n",
       "  font-weight: normal;\n",
       "  grid-column: 1;\n",
       "}\n",
       "\n",
       ".xr-attrs dt:hover span {\n",
       "  display: inline-block;\n",
       "  background: var(--xr-background-color);\n",
       "  padding-right: 10px;\n",
       "}\n",
       "\n",
       ".xr-attrs dd {\n",
       "  grid-column: 2;\n",
       "  white-space: pre-wrap;\n",
       "  word-break: break-all;\n",
       "}\n",
       "\n",
       ".xr-icon-database,\n",
       ".xr-icon-file-text2 {\n",
       "  display: inline-block;\n",
       "  vertical-align: middle;\n",
       "  width: 1em;\n",
       "  height: 1.5em !important;\n",
       "  stroke-width: 0;\n",
       "  stroke: currentColor;\n",
       "  fill: currentColor;\n",
       "}\n",
       "</style><pre class='xr-text-repr-fallback'>&lt;xarray.Dataset&gt;\n",
       "Dimensions:  (time: 2920, lat: 25, lon: 53)\n",
       "Coordinates:\n",
       "  * lat      (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0\n",
       "  * lon      (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0\n",
       "  * time     (time) datetime64[ns] 2013-01-01 ... 2014-12-31T18:00:00\n",
       "Data variables:\n",
       "    Tair     (time, lat, lon) float32 dask.array&lt;chunksize=(730, 13, 27), meta=np.ndarray&gt;\n",
       "Attributes:\n",
       "    Conventions:  COARDS\n",
       "    description:  Data is from NMC initialized reanalysis\\n(4x/day).  These a...\n",
       "    platform:     Model\n",
       "    references:   http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanaly...\n",
       "    title:        4x daily NMC reanalysis (1948)</pre><div class='xr-wrap' style='display:none'><div class='xr-header'><div class='xr-obj-type'>xarray.Dataset</div></div><ul class='xr-sections'><li class='xr-section-item'><input id='section-9707d41c-64ca-4e2e-ad54-36e1f0b5c223' class='xr-section-summary-in' type='checkbox' disabled ><label for='section-9707d41c-64ca-4e2e-ad54-36e1f0b5c223' class='xr-section-summary'  title='Expand/collapse section'>Dimensions:</label><div class='xr-section-inline-details'><ul class='xr-dim-list'><li><span class='xr-has-index'>time</span>: 2920</li><li><span class='xr-has-index'>lat</span>: 25</li><li><span class='xr-has-index'>lon</span>: 53</li></ul></div><div class='xr-section-details'></div></li><li class='xr-section-item'><input id='section-43e65ee0-990c-48fd-848c-b899b9c35c44' class='xr-section-summary-in' type='checkbox'  checked><label for='section-43e65ee0-990c-48fd-848c-b899b9c35c44' class='xr-section-summary' >Coordinates: <span>(3)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><ul class='xr-var-list'><li class='xr-var-item'><div class='xr-var-name'><span class='xr-has-index'>lat</span></div><div class='xr-var-dims'>(lat)</div><div class='xr-var-dtype'>float32</div><div class='xr-var-preview xr-preview'>75.0 72.5 70.0 ... 20.0 17.5 15.0</div><input id='attrs-0465c55f-0985-43d4-97b4-d27b644eefc7' class='xr-var-attrs-in' type='checkbox' ><label for='attrs-0465c55f-0985-43d4-97b4-d27b644eefc7' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-3b757392-c331-4532-a773-4d0c4a225a42' class='xr-var-data-in' type='checkbox'><label for='data-3b757392-c331-4532-a773-4d0c4a225a42' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'><dt><span>axis :</span></dt><dd>Y</dd><dt><span>long_name :</span></dt><dd>Latitude</dd><dt><span>standard_name :</span></dt><dd>latitude</dd><dt><span>units :</span></dt><dd>degrees_north</dd></dl></div><div class='xr-var-data'><pre>array([75. , 72.5, 70. , 67.5, 65. , 62.5, 60. , 57.5, 55. , 52.5, 50. , 47.5,\n",
       "       45. , 42.5, 40. , 37.5, 35. , 32.5, 30. , 27.5, 25. , 22.5, 20. , 17.5,\n",
       "       15. ], dtype=float32)</pre></div></li><li class='xr-var-item'><div class='xr-var-name'><span class='xr-has-index'>lon</span></div><div class='xr-var-dims'>(lon)</div><div class='xr-var-dtype'>float32</div><div class='xr-var-preview xr-preview'>200.0 202.5 205.0 ... 327.5 330.0</div><input id='attrs-997771e1-2ab9-4340-8568-b1a1c95a8d34' class='xr-var-attrs-in' type='checkbox' ><label for='attrs-997771e1-2ab9-4340-8568-b1a1c95a8d34' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-c2f646ac-6c8b-43e5-badb-1f13161e3edb' class='xr-var-data-in' type='checkbox'><label for='data-c2f646ac-6c8b-43e5-badb-1f13161e3edb' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'><dt><span>axis :</span></dt><dd>X</dd><dt><span>long_name :</span></dt><dd>Longitude</dd><dt><span>standard_name :</span></dt><dd>longitude</dd><dt><span>units :</span></dt><dd>degrees_east</dd></dl></div><div class='xr-var-data'><pre>array([200. , 202.5, 205. , 207.5, 210. , 212.5, 215. , 217.5, 220. , 222.5,\n",
       "       225. , 227.5, 230. , 232.5, 235. , 237.5, 240. , 242.5, 245. , 247.5,\n",
       "       250. , 252.5, 255. , 257.5, 260. , 262.5, 265. , 267.5, 270. , 272.5,\n",
       "       275. , 277.5, 280. , 282.5, 285. , 287.5, 290. , 292.5, 295. , 297.5,\n",
       "       300. , 302.5, 305. , 307.5, 310. , 312.5, 315. , 317.5, 320. , 322.5,\n",
       "       325. , 327.5, 330. ], dtype=float32)</pre></div></li><li class='xr-var-item'><div class='xr-var-name'><span class='xr-has-index'>time</span></div><div class='xr-var-dims'>(time)</div><div class='xr-var-dtype'>datetime64[ns]</div><div class='xr-var-preview xr-preview'>2013-01-01 ... 2014-12-31T18:00:00</div><input id='attrs-c59913fd-44fa-4ab3-9895-e455411d2a9c' class='xr-var-attrs-in' type='checkbox' ><label for='attrs-c59913fd-44fa-4ab3-9895-e455411d2a9c' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-a0b2aa8e-da6d-437a-ae33-34cfd24a7710' class='xr-var-data-in' type='checkbox'><label for='data-a0b2aa8e-da6d-437a-ae33-34cfd24a7710' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'><dt><span>long_name :</span></dt><dd>Time</dd><dt><span>standard_name :</span></dt><dd>time</dd></dl></div><div class='xr-var-data'><pre>array([&#x27;2013-01-01T00:00:00.000000000&#x27;, &#x27;2013-01-01T06:00:00.000000000&#x27;,\n",
       "       &#x27;2013-01-01T12:00:00.000000000&#x27;, ..., &#x27;2014-12-31T06:00:00.000000000&#x27;,\n",
       "       &#x27;2014-12-31T12:00:00.000000000&#x27;, &#x27;2014-12-31T18:00:00.000000000&#x27;],\n",
       "      dtype=&#x27;datetime64[ns]&#x27;)</pre></div></li></ul></div></li><li class='xr-section-item'><input id='section-ced6c53e-eea7-4f23-a0b7-bf1f71453357' class='xr-section-summary-in' type='checkbox'  checked><label for='section-ced6c53e-eea7-4f23-a0b7-bf1f71453357' class='xr-section-summary' >Data variables: <span>(1)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><ul class='xr-var-list'><li class='xr-var-item'><div class='xr-var-name'><span>Tair</span></div><div class='xr-var-dims'>(time, lat, lon)</div><div class='xr-var-dtype'>float32</div><div class='xr-var-preview xr-preview'>dask.array&lt;chunksize=(730, 13, 27), meta=np.ndarray&gt;</div><input id='attrs-38f8f7c1-063f-4e95-a9e7-afe5e2eddc10' class='xr-var-attrs-in' type='checkbox' ><label for='attrs-38f8f7c1-063f-4e95-a9e7-afe5e2eddc10' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-fd7e05dd-3dc3-483f-b8a9-bfd82796f2ef' class='xr-var-data-in' type='checkbox'><label for='data-fd7e05dd-3dc3-483f-b8a9-bfd82796f2ef' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'><dt><span>GRIB_id :</span></dt><dd>11</dd><dt><span>GRIB_name :</span></dt><dd>TMP</dd><dt><span>actual_range :</span></dt><dd>[185.16000366210938, 322.1000061035156]</dd><dt><span>dataset :</span></dt><dd>NMC Reanalysis</dd><dt><span>level_desc :</span></dt><dd>Surface</dd><dt><span>long_name :</span></dt><dd>4xDaily Air temperature at sigma level 995</dd><dt><span>parent_stat :</span></dt><dd>Other</dd><dt><span>precision :</span></dt><dd>2</dd><dt><span>statistic :</span></dt><dd>Individual Obs</dd><dt><span>units :</span></dt><dd>degK</dd><dt><span>var_desc :</span></dt><dd>Air temperature</dd></dl></div><div class='xr-var-data'><table>\n",
       "    <tr>\n",
       "        <td>\n",
       "            <table>\n",
       "                <thead>\n",
       "                    <tr>\n",
       "                        <td> </td>\n",
       "                        <th> Array </th>\n",
       "                        <th> Chunk </th>\n",
       "                    </tr>\n",
       "                </thead>\n",
       "                <tbody>\n",
       "                    \n",
       "                    <tr>\n",
       "                        <th> Bytes </th>\n",
       "                        <td> 14.76 MiB </td>\n",
       "                        <td> 0.98 MiB </td>\n",
       "                    </tr>\n",
       "                    \n",
       "                    <tr>\n",
       "                        <th> Shape </th>\n",
       "                        <td> (2920, 25, 53) </td>\n",
       "                        <td> (730, 13, 27) </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <th> Count </th>\n",
       "                        <td> 2 Graph Layers </td>\n",
       "                        <td> 16 Chunks </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                    <th> Type </th>\n",
       "                    <td> float32 </td>\n",
       "                    <td> numpy.ndarray </td>\n",
       "                    </tr>\n",
       "                </tbody>\n",
       "            </table>\n",
       "        </td>\n",
       "        <td>\n",
       "        <svg width=\"159\" height=\"146\" style=\"stroke:rgb(0,0,0);stroke-width:1\" >\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"80\" y2=\"70\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"10\" y1=\"13\" x2=\"80\" y2=\"83\" />\n",
       "  <line x1=\"10\" y1=\"25\" x2=\"80\" y2=\"96\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"10\" y2=\"25\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"27\" y1=\"17\" x2=\"27\" y2=\"43\" />\n",
       "  <line x1=\"45\" y1=\"35\" x2=\"45\" y2=\"60\" />\n",
       "  <line x1=\"62\" y1=\"52\" x2=\"62\" y2=\"78\" />\n",
       "  <line x1=\"80\" y1=\"70\" x2=\"80\" y2=\"96\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"10.0,0.0 80.58823529411765,70.58823529411765 80.58823529411765,96.00085180870013 10.0,25.41261651458249\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"38\" y2=\"0\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"27\" y1=\"17\" x2=\"56\" y2=\"17\" />\n",
       "  <line x1=\"45\" y1=\"35\" x2=\"73\" y2=\"35\" />\n",
       "  <line x1=\"62\" y1=\"52\" x2=\"91\" y2=\"52\" />\n",
       "  <line x1=\"80\" y1=\"70\" x2=\"109\" y2=\"70\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"80\" y2=\"70\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"24\" y1=\"0\" x2=\"95\" y2=\"70\" />\n",
       "  <line x1=\"38\" y1=\"0\" x2=\"109\" y2=\"70\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"10.0,0.0 38.48973265594604,0.0 109.0779679500637,70.58823529411765 80.58823529411765,70.58823529411765\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"80\" y1=\"70\" x2=\"109\" y2=\"70\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"80\" y1=\"83\" x2=\"109\" y2=\"83\" />\n",
       "  <line x1=\"80\" y1=\"96\" x2=\"109\" y2=\"96\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"80\" y1=\"70\" x2=\"80\" y2=\"96\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"95\" y1=\"70\" x2=\"95\" y2=\"96\" />\n",
       "  <line x1=\"109\" y1=\"70\" x2=\"109\" y2=\"96\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"80.58823529411765,70.58823529411765 109.0779679500637,70.58823529411765 109.0779679500637,96.00085180870013 80.58823529411765,96.00085180870013\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Text -->\n",
       "  <text x=\"94.833102\" y=\"116.000852\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" >53</text>\n",
       "  <text x=\"129.077968\" y=\"83.294544\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(0,129.077968,83.294544)\">25</text>\n",
       "  <text x=\"35.294118\" y=\"80.706734\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(45,35.294118,80.706734)\">2920</text>\n",
       "</svg>\n",
       "        </td>\n",
       "    </tr>\n",
       "</table></div></li></ul></div></li><li class='xr-section-item'><input id='section-1a09b05d-6ad2-4a3c-91de-8b28eab42e49' class='xr-section-summary-in' type='checkbox'  checked><label for='section-1a09b05d-6ad2-4a3c-91de-8b28eab42e49' class='xr-section-summary' >Attributes: <span>(5)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><dl class='xr-attrs'><dt><span>Conventions :</span></dt><dd>COARDS</dd><dt><span>description :</span></dt><dd>Data is from NMC initialized reanalysis\n",
       "(4x/day).  These are the 0.9950 sigma level values.</dd><dt><span>platform :</span></dt><dd>Model</dd><dt><span>references :</span></dt><dd>http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanalysis.html</dd><dt><span>title :</span></dt><dd>4x daily NMC reanalysis (1948)</dd></dl></div></li></ul></div></div>"
      ],
      "text/plain": [
       "<xarray.Dataset>\n",
       "Dimensions:  (time: 2920, lat: 25, lon: 53)\n",
       "Coordinates:\n",
       "  * lat      (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0\n",
       "  * lon      (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0\n",
       "  * time     (time) datetime64[ns] 2013-01-01 ... 2014-12-31T18:00:00\n",
       "Data variables:\n",
       "    Tair     (time, lat, lon) float32 dask.array<chunksize=(730, 13, 27), meta=np.ndarray>\n",
       "Attributes:\n",
       "    Conventions:  COARDS\n",
       "    description:  Data is from NMC initialized reanalysis\\n(4x/day).  These a...\n",
       "    platform:     Model\n",
       "    references:   http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanaly...\n",
       "    title:        4x daily NMC reanalysis (1948)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xr.open_zarr(store_s3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f369705-3d91-4cad-b430-8e797d3b7330",
   "metadata": {},
   "source": [
    "#### Delete the zarr file you created"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d116d941-8ba3-40d8-aa44-9c59444971a9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['tmp/tinaok/S3Map_zarr']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target.ls(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "4083647c-0dab-4aa9-a432-6805b2d4631d",
   "metadata": {},
   "outputs": [],
   "source": [
    "target.rm(path+'/S3Map_zarr', recursive=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ccdbc75d-ef90-4532-8142-78a231cc881e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['tmp/fepit',\n",
       " 'tmp/fepitzarr-demo',\n",
       " 'tmp/guillaumeeb',\n",
       " 'tmp/xarray-demo-dask-s3']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target.ls('tmp/')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b466043-2367-4ef9-8137-e978b1f5c668",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-danger alert-info\">\n",
    "    <b>one users can delete another user's file.... So please be carefull especially when using recursive=True </b>\n",
    "</div>\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}