Add artifical information filter

docs/ArtificialInformation_Filter.ipynb

@@ -0,0 +1,335 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "38ac6d1a",
+   "metadata": {},
+   "source": [
+    "<span style=\"color:red\">**<<<<<<< local**</span>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "895e1d5a",
+   "metadata": {},
+   "source": [
+    "# Artificial information filtering\n",
+    "\n",
+    "In simple terms the bitinformation is retrieved by checking how variable a bit pattern is. However, this approach cannot distinguish between actual information content and artifical information content. By studying the distribution of the information content the user can often identify clear cut-offs of real information content and artificial information content.\n",
+    "\n",
+    "The following example shows how such a separation of real information and artificial information can look like. To do so, artificial information is artificially added to an example dataset by applying linear quantization. Linear quantization is often applied to climate datasets (e.g. ERA5) and needs to be accounted for in order to retrieve meaningful bitinformation content. An algorithm that aims at detecting this artificial information itself is introduced."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3c37dd36",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import xarray as xr\n",
+    "import xbitinfo as xb\n",
+    "import numpy as np"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e8e1424f",
+   "metadata": {},
+   "source": [
+    "## Loading example dataset\n",
+    "We use here the openly accessible CONUS dataset. The dataset is available at full precision."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b18b9e24",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ds = xr.open_zarr(\n",
+    "    \"s3://hytest/conus404/conus404_hourly.zarr\",\n",
+    "    storage_options={\n",
+    "        \"anon\": True,\n",
+    "        \"requester_pays\": False,\n",
+    "        \"client_kwargs\": {\"endpoint_url\": \"https://usgs.osn.mghpcc.org\"},\n",
+    "    },\n",
+    ")\n",
+    "# selecting water vapor mixing ratio at 2 meters\n",
+    "data = ds[\"ACSWUPB\"]\n",
+    "# select subset of data for demonstration purposes\n",
+    "chunk = data.isel(time=slice(0, 9), y=slice(0, 525), x=slice(0, 525))\n",
+    "chunk"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "535ce421",
+   "metadata": {},
+   "source": [
+    "## Creating dataset copy with artificial information\n",
+    "### Functions to encode and decode"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "69543b4c",
+   "metadata": {},
+   "source": [
+    "<span style=\"color:red\">**=======**</span>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1842f792",
+   "metadata": {},
+   "source": [
+    "# Artificial information filtering\n",
+    "\n",
+    "In simple terms the bitinformation is retrieved by checking how variable a bit pattern is. However, this approach cannot distinguish between actual information content and artifical information content. By studying the distribution of the information content the user can often identify clear cut-offs of real information content and artificial information content.\n",
+    "\n",
+    "The following example shows how such a separation of real information and artificial information can look like. To do so, artificial information is artificially added to an example dataset by applying linear quantization. Linear quantization is often applied to climate datasets (e.g. ERA5) and needs to be accounted for in order to retrieve meaningful bitinformation content. An algorithm that aims at detecting this artificial information itself is introduced."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "bb998fbb",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import xarray as xr\n",
+    "import xbitinfo as xb\n",
+    "import numpy as np"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "32ac97e0",
+   "metadata": {},
+   "source": [
+    "## Loading example dataset\n",
+    "We use here the openly accessible CONUS dataset. The dataset is available at full precision."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9639a618",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ds = xr.open_zarr(\n",
+    "    \"s3://hytest/conus404/conus404_monthly.zarr\",\n",
+    "    storage_options={\n",
+    "        \"anon\": True,\n",
+    "        \"requester_pays\": False,\n",
+    "        \"client_kwargs\": {\"endpoint_url\": \"https://usgs.osn.mghpcc.org\"},\n",
+    "    },\n",
+    ")\n",
+    "# selecting water vapor mixing ratio at 2 meters\n",
+    "data = ds[\"ACSWDNT\"]\n",
+    "# select subset of data for demonstration purposes\n",
+    "chunk = data.isel(time=slice(0, 2), y=slice(0, 1015), x=slice(0, 1050))\n",
+    "chunk"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3d735e4b",
+   "metadata": {},
+   "source": [
+    "## Creating dataset copy with artificial information\n",
+    "### Functions to encode and decode"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0d30feaa",
+   "metadata": {},
+   "source": [
+    "<span style=\"color:red\">**>>>>>>> remote**</span>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b3a7c7ae",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Encoding function to compress data\n",
+    "def encode(chunk, scale, offset, dtype, astype):\n",
+    "    enc = (chunk - offset) * scale\n",
+    "    enc = np.around(enc)\n",
+    "    enc = enc.astype(astype, copy=False)\n",
+    "    return enc\n",
+    "\n",
+    "\n",
+    "# Decoding function to decompress data\n",
+    "def decode(enc, scale, offset, dtype, astype):\n",
+    "    dec = (enc / scale) + offset\n",
+    "    dec = dec.astype(dtype, copy=False)\n",
+    "    return dec"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fa6f26c7",
+   "metadata": {},
+   "source": [
+    "### Transform dataset to introduce artificial information"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c09e3cf3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "xmin = np.min(chunk)\n",
+    "xmax = np.max(chunk)\n",
+    "scale = (2**16 - 1) / (xmax - xmin)\n",
+    "offset = xmin\n",
+    "enc = encode(chunk, scale, offset, \"f4\", \"u2\")\n",
+    "dec = decode(enc, scale, offset, \"f4\", \"u2\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7126810d",
+   "metadata": {},
+   "source": [
+    "## Comparison of bitinformation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "05ef8a94",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# original dataset without artificial information\n",
+    "orig_info = xb.get_bitinformation(\n",
+    "    xr.Dataset({\"w/o artif. info\": chunk}),\n",
+    "    dim=\"x\",\n",
+    "    implementation=\"python\",\n",
+    ")\n",
+    "\n",
+    "# dataset with artificial information\n",
+    "arti_info = xb.get_bitinformation(\n",
+    "    xr.Dataset({\"w artif. info\": dec}),\n",
+    "    dim=\"x\",\n",
+    "    implementation=\"python\",\n",
+    ")\n",
+    "\n",
+    "# plotting distribution of bitwise information content\n",
+    "info = xr.merge([orig_info, arti_info])\n",
+    "plot = xb.plot_bitinformation(info)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "de1ecb7e",
+   "metadata": {},
+   "source": [
+    "The figure reveals that artificial information is introduced by applying linear quantization. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8600d4b8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "keepbits = xb.get_keepbits(info, inflevel=[0.99])\n",
+    "print(\n",
+    "    f\"The number of keepbits increased from {keepbits['w/o artif. info'].item(0)} bits in the original dataset to {keepbits['w artif. info'].item(0)} bits in the dataset with artificial information.\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fa80f988",
+   "metadata": {},
+   "source": [
+    "In the following, a gradient based filter is introduced to remove this artificial information again so that even in case artificial information is present in a dataset the number of keepbits remains similar."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3f7a7c2e",
+   "metadata": {},
+   "source": [
+    "## Artificial information filter\n",
+    "The filter `gradient` works as follows:\n",
+    "\n",
+    "1. It determines the Cumulative Distribution Function(CDF) of the bitwise information content\n",
+    "2. It computes the gradient of the CDF to identify points where the gradient becomes close to a given tolerance indicating a drop in information.\n",
+    "3. Simultaneously, it keeps track of the minimum cumulative sum of information content which is threshold here, which signifies at least this much fraction of total information needs to be passed.\n",
+    "4. So the bit where the intersection of the gradient reaching the tolerance and the cumulative sum exceeding the threshold is our TrueKeepbits. All bits beyond this index are assumed to contain artificial information and are set to zero in order to cut them off.\n",
+    "5. You can see the above concept implemented in the function get_cdf_without_artificial_information in xbitinfo.py\n",
+    "\n",
+    "Please note that this filter relies on a clear separation between real and artificial information content and might not work in all cases."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b0ab6633",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "xb.get_keepbits(\n",
+    "    arti_info,\n",
+    "    inflevel=[0.99],\n",
+    "    information_filter=\"Gradient\",\n",
+    "    **{\"threshold\": 0.7, \"tolerance\": 0.001}\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "21c6369d",
+   "metadata": {},
+   "source": [
+    "With the application of the filter the keepbits are closer/identical to their original value in the dataset without artificial information. The plot of the bitinformation visualizes this:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8e9183b2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "plot = xb.plot_bitinformation(arti_info, information_filter=\"Gradient\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.7"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/environment.yml

@@ -21,6 +21,9 @@ dependencies:
   - sphinx-book-theme>=0.1.7
   - myst-nb
   - numcodecs>=0.10.0
+  - intake-xarray
+  - metpy
+  - s3fs
   - pip
   - pip:
     - -e ../.

docs/index.rst

@@ -96,17 +96,16 @@ Credits
 
     quick-start.ipynb
 
-
 **User Guide**
 
-* :doc:`chunking`
+* :doc:`artificialinformation`
 
 .. toctree::
-    :maxdepth: 1
-    :hidden:
-    :caption: User Guide
+      :maxdepth: 1
+      :hidden:
+      :caption: User Guide
 
-    chunking.ipynb
+      ArtificialInformation_Filter.ipynb
 
 **Help & Reference**
 

environment.yml

@@ -28,6 +28,9 @@ dependencies:
   - sphinx-book-theme
   - myst-nb
   - numcodecs>=0.10.0
+  - pytest-lazy-fixture
+  - aiohttp
+  - s3fs
   - pip
   - pip:
     - -e .