bigquery/tutorial/08-numpy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5f0e8cc2-b968-4059-b48a-a99bad1feecb",
   "metadata": {},
   "source": [
    "# Tutorial 8: Running NumPy In Your Data Warehouse"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e30c24dd-ebbe-413e-bdd5-433d615e718a",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\"> <b>Before we get started: </b> \n",
    "    <ul style=\"list-style-type: none;margin: 0;padding: 0;\">\n",
    "        <li>✍️ To run this notebook, you need to have Ponder installed and set up on your machine. If you have not done so already, please refer to our <a href=\"https://docs.ponder.io/getting_started/quickstart.html\">Quickstart guide</a> to get started.</li>\n",
    "        <li>📖 Otherwise, if you're just interested in browsing through the tutorial, keep reading below!</li>\n",
    "    </ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28eea415-26b6-423e-b8de-2e37c01a0184",
   "metadata": {},
   "source": [
    "Pandas and NumPy are the two most popular Python data science libraries, used by more than 60% of Python developers. NumPy supports linear algebra operations in Python, and as a result, is the fundamental building block of machine learning."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5e5aa6b-259a-4845-baf8-05dea286d2c1",
   "metadata": {},
   "source": [
    "Ponder lets you run NumPy commands directly in your warehouse. This means you can work with the NumPy API to build data and ML pipelines, and let BigQuery take care of scaling and security for you."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cba60d7b-8399-4574-baf1-1b4c8def6ff4",
   "metadata": {},
   "source": [
    "Here, we'll show a few examples of Ponder in action with NumPy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "fc87e13e-b30a-4146-b82d-a49212f8085a",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-05-12 10:01:29 - Creating session xSpjy6Ay-h5m_J842U4KQtaq5XpS3szSf8m1quDMkC\n"
     ]
    }
   ],
   "source": [
    "import ponder; ponder.init()\n",
    "import modin.pandas as pd\n",
    "from google.cloud import bigquery\n",
    "from google.cloud.bigquery import dbapi\n",
    "from google.oauth2 import service_account\n",
    "import json\n",
    "bigquery_con = dbapi.Connection(bigquery.Client(credentials=service_account.Credentials.from_service_account_info(json.loads(open(\"../credential.json\").read()),scopes=[\"https://www.googleapis.com/auth/bigquery\"])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e12c33df-fce1-45b1-8b50-5db1e57b35b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_sql(\"PONDER.PONDER_CUSTOMER\", bigquery_con)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f61b127-fd79-4ac9-980a-84bfd03506ef",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\"> <b>Note: </b> <span>NumPy support is currently part of Modin's experimental API, please drop us a note at <a href\"mailto:support@ponder.io\">support@ponder.io</a> if you run into any issues. Feedback welcome!</span></div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "05a81146-43a0-4ab4-b1b6-1db3ae221cf3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import modin.config as cfg\n",
    "cfg.ExperimentalNumPyAPI.put(True)\n",
    "import modin.numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1152fa4f-dbdb-4cb1-88c1-ff3cdf1d7163",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "UserWarning: Using Modin's new NumPy API. To convert from a Modin object to a NumPy array, either turn off the ExperimentalNumPyAPI flag, or use `modin.utils.to_numpy`.\n"
     ]
    }
   ],
   "source": [
    "arr = df.select_dtypes(\"number\").to_numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16b64eda-5ad1-4398-9861-891f25b714b7",
   "metadata": {},
   "source": [
    "We can convert the numerical values of the dataframe into Modin's NumPy array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e82312b4-e2a0-4412-9add-db8591d9fd3f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "modin.numpy.arr.array"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(arr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4833cc0e-4aff-403e-a582-59f9fd8e3702",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 6.00820e+04,  0.00000e+00,  3.64547e+03],\n",
       "       [ 6.00800e+04,  0.00000e+00,  6.89240e+02],\n",
       "       [ 6.00180e+04,  0.00000e+00,  5.75983e+03],\n",
       "       [ 6.00620e+04,  0.00000e+00,  6.21099e+03],\n",
       "       [ 6.00220e+04,  0.00000e+00, -7.59740e+02],\n",
       "       [ 6.00530e+04,  0.00000e+00,  6.51551e+03],\n",
       "       [ 6.00430e+04,  0.00000e+00,  7.37030e+02],\n",
       "       [ 6.00130e+04,  0.00000e+00, -4.85690e+02],\n",
       "       [ 6.00980e+04,  0.00000e+00,  1.44968e+03],\n",
       "       [ 6.00710e+04,  1.00000e+00,  7.05068e+03],\n",
       "       [ 6.00740e+04,  1.00000e+00,  8.36434e+03],\n",
       "       [ 6.00190e+04,  1.00000e+00,  3.44477e+03],\n",
       "       [ 6.00170e+04,  1.00000e+00,  3.65317e+03],\n",
       "       [ 6.00450e+04,  1.00000e+00,  8.38811e+03],\n",
       "       [ 6.00160e+04,  2.00000e+00,  4.48097e+03],\n",
       "       [ 6.00470e+04,  2.00000e+00,  2.26476e+03],\n",
       "       [ 6.00240e+04,  2.00000e+00, -6.64000e+00],\n",
       "       [ 6.00570e+04,  2.00000e+00,  9.84870e+02],\n",
       "       [ 6.00080e+04,  2.00000e+00,  5.62144e+03],\n",
       "       [ 6.00560e+04,  3.00000e+00,  8.88028e+03],\n",
       "       [ 6.00920e+04,  3.00000e+00,  6.71167e+03],\n",
       "       [ 6.00770e+04,  4.00000e+00,  9.00061e+03],\n",
       "       [ 6.00440e+04,  4.00000e+00, -8.12910e+02],\n",
       "       [ 6.00670e+04,  4.00000e+00,  1.87419e+03],\n",
       "       [ 6.00950e+04,  4.00000e+00,  8.10000e+02],\n",
       "       [ 6.00400e+04,  4.00000e+00,  3.44678e+03],\n",
       "       [ 6.00370e+04,  4.00000e+00,  1.03242e+03],\n",
       "       [ 6.00860e+04,  4.00000e+00,  5.83595e+03],\n",
       "       [ 6.00210e+04,  5.00000e+00,  9.91951e+03],\n",
       "       [ 6.00690e+04,  5.00000e+00,  4.65611e+03],\n",
       "       [ 6.00280e+04,  6.00000e+00,  3.49499e+03],\n",
       "       [ 6.00600e+04,  6.00000e+00,  2.63607e+03],\n",
       "       [ 6.00380e+04,  7.00000e+00,  2.77118e+03],\n",
       "       [ 6.00900e+04,  7.00000e+00,  4.29350e+03],\n",
       "       [ 6.00110e+04,  8.00000e+00,  3.42248e+03],\n",
       "       [ 6.00410e+04,  8.00000e+00,  1.94382e+03],\n",
       "       [ 6.00310e+04,  8.00000e+00,  6.76574e+03],\n",
       "       [ 6.00480e+04,  8.00000e+00,  1.79813e+03],\n",
       "       [ 6.00490e+04,  9.00000e+00,  2.12883e+03],\n",
       "       [ 6.00660e+04,  9.00000e+00,  2.47224e+03],\n",
       "       [ 6.00090e+04,  9.00000e+00,  9.54801e+03],\n",
       "       [ 6.00150e+04,  9.00000e+00,  4.62239e+03],\n",
       "       [ 6.00730e+04,  9.00000e+00,  7.31030e+02],\n",
       "       [ 6.00640e+04,  9.00000e+00,  8.02604e+03],\n",
       "       [ 6.00550e+04,  1.00000e+01,  3.72535e+03],\n",
       "       [ 6.00390e+04,  1.00000e+01, -7.65420e+02],\n",
       "       [ 6.00040e+04,  1.00000e+01,  7.97522e+03],\n",
       "       [ 6.00830e+04,  1.20000e+01,  8.43043e+03],\n",
       "       [ 6.00070e+04,  1.20000e+01,  6.01717e+03],\n",
       "       [ 6.00700e+04,  1.20000e+01, -7.26590e+02],\n",
       "       [ 6.00050e+04,  1.20000e+01,  2.50474e+03],\n",
       "       [ 6.00360e+04,  1.20000e+01,  7.89790e+03],\n",
       "       [ 6.00960e+04,  1.20000e+01,  4.62025e+03],\n",
       "       [ 6.00940e+04,  1.20000e+01, -4.08230e+02],\n",
       "       [ 6.00890e+04,  1.30000e+01,  9.84912e+03],\n",
       "       [ 6.00540e+04,  1.30000e+01, -8.61620e+02],\n",
       "       [ 6.00350e+04,  1.30000e+01,  7.18950e+03],\n",
       "       [ 6.00610e+04,  1.40000e+01,  5.34963e+03],\n",
       "       [ 6.00290e+04,  1.40000e+01,  9.28115e+03],\n",
       "       [ 6.00510e+04,  1.40000e+01,  4.73758e+03],\n",
       "       [ 6.00010e+04,  1.40000e+01,  9.95756e+03],\n",
       "       [ 6.00750e+04,  1.40000e+01,  8.59308e+03],\n",
       "       [ 6.00340e+04,  1.40000e+01,  7.23069e+03],\n",
       "       [ 6.00810e+04,  1.50000e+01, -9.24450e+02],\n",
       "       [ 6.00020e+04,  1.50000e+01,  7.42460e+02],\n",
       "       [ 6.00200e+04,  1.50000e+01,  1.18730e+03],\n",
       "       [ 6.00030e+04,  1.60000e+01,  2.52692e+03],\n",
       "       [ 6.00500e+04,  1.60000e+01,  5.18199e+03],\n",
       "       [ 6.00140e+04,  1.60000e+01,  7.93215e+03],\n",
       "       [ 6.01000e+04,  1.60000e+01, -5.85000e+00],\n",
       "       [ 6.00520e+04,  1.60000e+01,  4.51517e+03],\n",
       "       [ 6.00630e+04,  1.60000e+01,  7.19147e+03],\n",
       "       [ 6.00320e+04,  1.60000e+01,  2.89686e+03],\n",
       "       [ 6.00650e+04,  1.70000e+01,  9.75739e+03],\n",
       "       [ 6.00420e+04,  1.70000e+01,  8.68196e+03],\n",
       "       [ 6.00840e+04,  1.80000e+01,  8.25173e+03],\n",
       "       [ 6.00850e+04,  1.90000e+01,  7.95774e+03],\n",
       "       [ 6.00720e+04,  2.00000e+01,  6.79179e+03],\n",
       "       [ 6.00760e+04,  2.00000e+01,  3.04010e+03],\n",
       "       [ 6.00880e+04,  2.00000e+01,  3.12900e+02],\n",
       "       [ 6.00460e+04,  2.00000e+01,  1.02257e+03],\n",
       "       [ 6.00970e+04,  2.10000e+01,  1.62661e+03],\n",
       "       [ 6.00260e+04,  2.10000e+01,  7.50813e+03],\n",
       "       [ 6.00100e+04,  2.10000e+01,  3.49791e+03],\n",
       "       [ 6.00990e+04,  2.10000e+01,  8.76765e+03],\n",
       "       [ 6.00230e+04,  2.10000e+01, -1.29700e+01],\n",
       "       [ 6.00300e+04,  2.10000e+01,  6.39750e+03],\n",
       "       [ 6.00780e+04,  2.10000e+01, -7.04560e+02],\n",
       "       [ 6.00870e+04,  2.20000e+01,  6.81075e+03],\n",
       "       [ 6.00910e+04,  2.20000e+01,  9.91995e+03],\n",
       "       [ 6.00060e+04,  2.20000e+01,  9.05140e+03],\n",
       "       [ 6.00930e+04,  2.20000e+01,  7.01033e+03],\n",
       "       [ 6.00250e+04,  2.20000e+01,  7.29615e+03],\n",
       "       [ 6.00120e+04,  2.30000e+01,  6.71600e+02],\n",
       "       [ 6.00680e+04,  2.30000e+01,  4.49281e+03],\n",
       "       [ 6.00580e+04,  2.30000e+01,  6.67212e+03],\n",
       "       [ 6.00790e+04,  2.40000e+01,  3.32955e+03],\n",
       "       [ 6.00590e+04,  2.40000e+01,  2.33746e+03],\n",
       "       [ 6.00330e+04,  2.40000e+01, -4.93590e+02],\n",
       "       [ 6.00270e+04,  2.40000e+01,  6.11522e+03]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efcbe12c-5bee-47ad-9150-66782d3a374b",
   "metadata": {},
   "source": [
    "We can perform reduce operations such as `np.sum` and `np.mean` across the entire matrix: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "acbdd7ee-5bef-48ee-84dd-b4dfea2b003a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6448157.58"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(arr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c3ec997-4af0-4743-83f9-ac9c80df41dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.mean(arr)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42433f63-cdd7-451f-93d7-10eeb70a0939",
   "metadata": {},
   "source": [
    "or we can perform the reduce operation along a specific axis: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "54490f21-89cb-4dcb-a907-44d1937589b2",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "UserWarning: Modin does not yet support broadcasting between nested 1D arrays and 2D arrays.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[21242.49      ],\n",
       "       [20256.41333333],\n",
       "       [21925.94333333],\n",
       "       [22090.99666667],\n",
       "       [19754.08666667],\n",
       "       [22189.50333333],\n",
       "       [20260.01      ],\n",
       "       [19842.43666667],\n",
       "       [20515.89333333],\n",
       "       [22374.22666667],\n",
       "       [22813.11333333],\n",
       "       [21154.92333333],\n",
       "       [21223.72333333],\n",
       "       [22811.37      ],\n",
       "       [21499.65666667],\n",
       "       [20771.25333333],\n",
       "       [20006.45333333],\n",
       "       [20347.95666667],\n",
       "       [21877.14666667],\n",
       "       [22979.76      ],\n",
       "       [22268.89      ],\n",
       "       [23027.20333333],\n",
       "       [19745.03      ],\n",
       "       [20648.39666667],\n",
       "       [20303.        ],\n",
       "       [21163.59333333],\n",
       "       [20357.80666667],\n",
       "       [21975.31666667],\n",
       "       [23315.17      ],\n",
       "       [21576.70333333],\n",
       "       [21176.33      ],\n",
       "       [20900.69      ],\n",
       "       [20938.72666667],\n",
       "       [21463.5       ],\n",
       "       [21147.16      ],\n",
       "       [20664.27333333],\n",
       "       [22268.24666667],\n",
       "       [20618.04333333],\n",
       "       [20728.94333333],\n",
       "       [20849.08      ],\n",
       "       [23188.67      ],\n",
       "       [21548.79666667],\n",
       "       [20271.01      ],\n",
       "       [22699.68      ],\n",
       "       [21263.45      ],\n",
       "       [19761.19333333],\n",
       "       [22663.07333333],\n",
       "       [22841.81      ],\n",
       "       [22012.05666667],\n",
       "       [19785.13666667],\n",
       "       [20840.58      ],\n",
       "       [22648.63333333],\n",
       "       [21576.08333333],\n",
       "       [19899.25666667],\n",
       "       [23317.04      ],\n",
       "       [19735.12666667],\n",
       "       [22412.5       ],\n",
       "       [21808.21      ],\n",
       "       [23108.05      ],\n",
       "       [21600.86      ],\n",
       "       [23324.18666667],\n",
       "       [22894.02666667],\n",
       "       [22426.23      ],\n",
       "       [19723.85      ],\n",
       "       [20253.15333333],\n",
       "       [20407.43333333],\n",
       "       [20848.64      ],\n",
       "       [21749.33      ],\n",
       "       [22654.05      ],\n",
       "       [20036.71666667],\n",
       "       [21527.72333333],\n",
       "       [22423.49      ],\n",
       "       [20981.62      ],\n",
       "       [23279.79666667],\n",
       "       [22913.65333333],\n",
       "       [22784.57666667],\n",
       "       [22687.24666667],\n",
       "       [22294.59666667],\n",
       "       [21045.36666667],\n",
       "       [20140.3       ],\n",
       "       [20362.85666667],\n",
       "       [20581.53666667],\n",
       "       [22518.37666667],\n",
       "       [21176.30333333],\n",
       "       [22962.55      ],\n",
       "       [20010.34333333],\n",
       "       [22149.5       ],\n",
       "       [19798.14666667],\n",
       "       [22306.58333333],\n",
       "       [23344.31666667],\n",
       "       [23026.46666667],\n",
       "       [22375.11      ],\n",
       "       [22447.71666667],\n",
       "       [20235.53333333],\n",
       "       [21527.93666667],\n",
       "       [22251.04      ],\n",
       "       [21144.18333333],\n",
       "       [20806.82      ],\n",
       "       [19854.47      ],\n",
       "       [22055.40666667]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# mean of every row returning object of same dimensions\n",
    "np.mean(arr, axis=-1, keepdims=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70c9d47c-c37b-4720-bce0-d345524180a7",
   "metadata": {},
   "source": [
    "We can also do element-wise matrix operations such as addition of two matrices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "674ea11f-d770-42b4-aaae-c860361e3956",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[6.372747e+04, 0.000000e+00, 6.372747e+04],\n",
       "       [6.076924e+04, 0.000000e+00, 6.076924e+04],\n",
       "       [6.577783e+04, 0.000000e+00, 6.577783e+04],\n",
       "       [6.627299e+04, 0.000000e+00, 6.627299e+04],\n",
       "       [5.926226e+04, 0.000000e+00, 5.926226e+04],\n",
       "       [6.656851e+04, 0.000000e+00, 6.656851e+04],\n",
       "       [6.078003e+04, 0.000000e+00, 6.078003e+04],\n",
       "       [5.952731e+04, 0.000000e+00, 5.952731e+04],\n",
       "       [6.154768e+04, 0.000000e+00, 6.154768e+04],\n",
       "       [6.712168e+04, 2.000000e+00, 6.712168e+04],\n",
       "       [6.843834e+04, 2.000000e+00, 6.843834e+04],\n",
       "       [6.346377e+04, 2.000000e+00, 6.346377e+04],\n",
       "       [6.367017e+04, 2.000000e+00, 6.367017e+04],\n",
       "       [6.843311e+04, 2.000000e+00, 6.843311e+04],\n",
       "       [6.449697e+04, 4.000000e+00, 6.449697e+04],\n",
       "       [6.231176e+04, 4.000000e+00, 6.231176e+04],\n",
       "       [6.001736e+04, 4.000000e+00, 6.001736e+04],\n",
       "       [6.104187e+04, 4.000000e+00, 6.104187e+04],\n",
       "       [6.562944e+04, 4.000000e+00, 6.562944e+04],\n",
       "       [6.893628e+04, 6.000000e+00, 6.893628e+04],\n",
       "       [6.680367e+04, 6.000000e+00, 6.680367e+04],\n",
       "       [6.907761e+04, 8.000000e+00, 6.907761e+04],\n",
       "       [5.923109e+04, 8.000000e+00, 5.923109e+04],\n",
       "       [6.194119e+04, 8.000000e+00, 6.194119e+04],\n",
       "       [6.090500e+04, 8.000000e+00, 6.090500e+04],\n",
       "       [6.348678e+04, 8.000000e+00, 6.348678e+04],\n",
       "       [6.106942e+04, 8.000000e+00, 6.106942e+04],\n",
       "       [6.592195e+04, 8.000000e+00, 6.592195e+04],\n",
       "       [6.994051e+04, 1.000000e+01, 6.994051e+04],\n",
       "       [6.472511e+04, 1.000000e+01, 6.472511e+04],\n",
       "       [6.352299e+04, 1.200000e+01, 6.352299e+04],\n",
       "       [6.269607e+04, 1.200000e+01, 6.269607e+04],\n",
       "       [6.280918e+04, 1.400000e+01, 6.280918e+04],\n",
       "       [6.438350e+04, 1.400000e+01, 6.438350e+04],\n",
       "       [6.343348e+04, 1.600000e+01, 6.343348e+04],\n",
       "       [6.198482e+04, 1.600000e+01, 6.198482e+04],\n",
       "       [6.679674e+04, 1.600000e+01, 6.679674e+04],\n",
       "       [6.184613e+04, 1.600000e+01, 6.184613e+04],\n",
       "       [6.217783e+04, 1.800000e+01, 6.217783e+04],\n",
       "       [6.253824e+04, 1.800000e+01, 6.253824e+04],\n",
       "       [6.955701e+04, 1.800000e+01, 6.955701e+04],\n",
       "       [6.463739e+04, 1.800000e+01, 6.463739e+04],\n",
       "       [6.080403e+04, 1.800000e+01, 6.080403e+04],\n",
       "       [6.809004e+04, 1.800000e+01, 6.809004e+04],\n",
       "       [6.378035e+04, 2.000000e+01, 6.378035e+04],\n",
       "       [5.927358e+04, 2.000000e+01, 5.927358e+04],\n",
       "       [6.797922e+04, 2.000000e+01, 6.797922e+04],\n",
       "       [6.851343e+04, 2.400000e+01, 6.851343e+04],\n",
       "       [6.602417e+04, 2.400000e+01, 6.602417e+04],\n",
       "       [5.934341e+04, 2.400000e+01, 5.934341e+04],\n",
       "       [6.250974e+04, 2.400000e+01, 6.250974e+04],\n",
       "       [6.793390e+04, 2.400000e+01, 6.793390e+04],\n",
       "       [6.471625e+04, 2.400000e+01, 6.471625e+04],\n",
       "       [5.968577e+04, 2.400000e+01, 5.968577e+04],\n",
       "       [6.993812e+04, 2.600000e+01, 6.993812e+04],\n",
       "       [5.919238e+04, 2.600000e+01, 5.919238e+04],\n",
       "       [6.722450e+04, 2.600000e+01, 6.722450e+04],\n",
       "       [6.541063e+04, 2.800000e+01, 6.541063e+04],\n",
       "       [6.931015e+04, 2.800000e+01, 6.931015e+04],\n",
       "       [6.478858e+04, 2.800000e+01, 6.478858e+04],\n",
       "       [6.995856e+04, 2.800000e+01, 6.995856e+04],\n",
       "       [6.866808e+04, 2.800000e+01, 6.866808e+04],\n",
       "       [6.726469e+04, 2.800000e+01, 6.726469e+04],\n",
       "       [5.915655e+04, 3.000000e+01, 5.915655e+04],\n",
       "       [6.074446e+04, 3.000000e+01, 6.074446e+04],\n",
       "       [6.120730e+04, 3.000000e+01, 6.120730e+04],\n",
       "       [6.252992e+04, 3.200000e+01, 6.252992e+04],\n",
       "       [6.523199e+04, 3.200000e+01, 6.523199e+04],\n",
       "       [6.794615e+04, 3.200000e+01, 6.794615e+04],\n",
       "       [6.009415e+04, 3.200000e+01, 6.009415e+04],\n",
       "       [6.456717e+04, 3.200000e+01, 6.456717e+04],\n",
       "       [6.725447e+04, 3.200000e+01, 6.725447e+04],\n",
       "       [6.292886e+04, 3.200000e+01, 6.292886e+04],\n",
       "       [6.982239e+04, 3.400000e+01, 6.982239e+04],\n",
       "       [6.872396e+04, 3.400000e+01, 6.872396e+04],\n",
       "       [6.833573e+04, 3.600000e+01, 6.833573e+04],\n",
       "       [6.804274e+04, 3.800000e+01, 6.804274e+04],\n",
       "       [6.686379e+04, 4.000000e+01, 6.686379e+04],\n",
       "       [6.311610e+04, 4.000000e+01, 6.311610e+04],\n",
       "       [6.040090e+04, 4.000000e+01, 6.040090e+04],\n",
       "       [6.106857e+04, 4.000000e+01, 6.106857e+04],\n",
       "       [6.172361e+04, 4.200000e+01, 6.172361e+04],\n",
       "       [6.753413e+04, 4.200000e+01, 6.753413e+04],\n",
       "       [6.350791e+04, 4.200000e+01, 6.350791e+04],\n",
       "       [6.886665e+04, 4.200000e+01, 6.886665e+04],\n",
       "       [6.001003e+04, 4.200000e+01, 6.001003e+04],\n",
       "       [6.642750e+04, 4.200000e+01, 6.642750e+04],\n",
       "       [5.937344e+04, 4.200000e+01, 5.937344e+04],\n",
       "       [6.689775e+04, 4.400000e+01, 6.689775e+04],\n",
       "       [7.001095e+04, 4.400000e+01, 7.001095e+04],\n",
       "       [6.905740e+04, 4.400000e+01, 6.905740e+04],\n",
       "       [6.710333e+04, 4.400000e+01, 6.710333e+04],\n",
       "       [6.732115e+04, 4.400000e+01, 6.732115e+04],\n",
       "       [6.068360e+04, 4.600000e+01, 6.068360e+04],\n",
       "       [6.456081e+04, 4.600000e+01, 6.456081e+04],\n",
       "       [6.673012e+04, 4.600000e+01, 6.673012e+04],\n",
       "       [6.340855e+04, 4.800000e+01, 6.340855e+04],\n",
       "       [6.239646e+04, 4.800000e+01, 6.239646e+04],\n",
       "       [5.953941e+04, 4.800000e+01, 5.953941e+04],\n",
       "       [6.614222e+04, 4.800000e+01, 6.614222e+04]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add an array with an array with reversed columns\n",
    "arr + arr[:,::-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9daf58e0-b354-4dcc-9e5e-9ffe083e05b9",
   "metadata": {},
   "source": [
    "Putting everything together, we can do both together: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "a5d4fde6-7147-42cc-9131-3755a353bf87",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 38839.51      , -21242.49      , -17597.02      ],\n",
       "       [ 39823.58666667, -20256.41333333, -19567.17333333],\n",
       "       [ 38092.05666667, -21925.94333333, -16166.11333333],\n",
       "       [ 37971.00333333, -22090.99666667, -15880.00666667],\n",
       "       [ 40267.91333333, -19754.08666667, -20513.82666667],\n",
       "       [ 37863.49666667, -22189.50333333, -15673.99333333],\n",
       "       [ 39782.99      , -20260.01      , -19522.98      ],\n",
       "       [ 40170.56333333, -19842.43666667, -20328.12666667],\n",
       "       [ 39582.10666667, -20515.89333333, -19066.21333333],\n",
       "       [ 37696.77333333, -22373.22666667, -15323.54666667],\n",
       "       [ 37260.88666667, -22812.11333333, -14448.77333333],\n",
       "       [ 38864.07666667, -21153.92333333, -17710.15333333],\n",
       "       [ 38793.27666667, -21222.72333333, -17570.55333333],\n",
       "       [ 37233.63      , -22810.37      , -14423.26      ],\n",
       "       [ 38516.34333333, -21497.65666667, -17018.68666667],\n",
       "       [ 39275.74666667, -20769.25333333, -18506.49333333],\n",
       "       [ 40017.54666667, -20004.45333333, -20013.09333333],\n",
       "       [ 39709.04333333, -20345.95666667, -19363.08666667],\n",
       "       [ 38130.85333333, -21875.14666667, -16255.70666667],\n",
       "       [ 37076.24      , -22976.76      , -14099.48      ],\n",
       "       [ 37823.11      , -22265.89      , -15557.22      ],\n",
       "       [ 37049.79666667, -23023.20333333, -14026.59333333],\n",
       "       [ 40298.97      , -19741.03      , -20557.94      ],\n",
       "       [ 39418.60333333, -20644.39666667, -18774.20666667],\n",
       "       [ 39792.        , -20299.        , -19493.        ],\n",
       "       [ 38876.40666667, -21159.59333333, -17716.81333333],\n",
       "       [ 39679.19333333, -20353.80666667, -19325.38666667],\n",
       "       [ 38110.68333333, -21971.31666667, -16139.36666667],\n",
       "       [ 36705.83      , -23310.17      , -13395.66      ],\n",
       "       [ 38492.29666667, -21571.70333333, -16920.59333333],\n",
       "       [ 38851.67      , -21170.33      , -17681.34      ],\n",
       "       [ 39159.31      , -20894.69      , -18264.62      ],\n",
       "       [ 39099.27333333, -20931.72666667, -18167.54666667],\n",
       "       [ 38626.5       , -21456.5       , -17170.        ],\n",
       "       [ 38863.84      , -21139.16      , -17724.68      ],\n",
       "       [ 39376.72666667, -20656.27333333, -18720.45333333],\n",
       "       [ 37762.75333333, -22260.24666667, -15502.50666667],\n",
       "       [ 39429.95666667, -20610.04333333, -18819.91333333],\n",
       "       [ 39320.05666667, -20719.94333333, -18600.11333333],\n",
       "       [ 39216.92      , -20840.08      , -18376.84      ],\n",
       "       [ 36820.33      , -23179.67      , -13640.66      ],\n",
       "       [ 38466.20333333, -21539.79666667, -16926.40666667],\n",
       "       [ 39801.99      , -20262.01      , -19539.98      ],\n",
       "       [ 37364.32      , -22690.68      , -14673.64      ],\n",
       "       [ 38791.55      , -21253.45      , -17538.1       ],\n",
       "       [ 40277.80666667, -19751.19333333, -20526.61333333],\n",
       "       [ 37340.92666667, -22653.07333333, -14687.85333333],\n",
       "       [ 37241.19      , -22829.81      , -14411.38      ],\n",
       "       [ 37994.94333333, -22000.05666667, -15994.88666667],\n",
       "       [ 40284.86333333, -19773.13666667, -20511.72666667],\n",
       "       [ 39164.42      , -20828.58      , -18335.84      ],\n",
       "       [ 37387.36666667, -22636.63333333, -14750.73333333],\n",
       "       [ 38519.91666667, -21564.08333333, -16955.83333333],\n",
       "       [ 40194.74333333, -19887.25666667, -20307.48666667],\n",
       "       [ 36771.96      , -23304.04      , -13467.92      ],\n",
       "       [ 40318.87333333, -19722.12666667, -20596.74666667],\n",
       "       [ 37622.5       , -22399.5       , -15223.        ],\n",
       "       [ 38252.79      , -21794.21      , -16458.58      ],\n",
       "       [ 36920.95      , -23094.05      , -13826.9       ],\n",
       "       [ 38450.14      , -21586.86      , -16863.28      ],\n",
       "       [ 36676.81333333, -23310.18666667, -13366.62666667],\n",
       "       [ 37180.97333333, -22880.02666667, -14300.94666667],\n",
       "       [ 37607.77      , -22412.23      , -15195.54      ],\n",
       "       [ 40357.15      , -19708.85      , -20648.3       ],\n",
       "       [ 39748.84666667, -20238.15333333, -19510.69333333],\n",
       "       [ 39612.56666667, -20392.43333333, -19220.13333333],\n",
       "       [ 39154.36      , -20832.64      , -18321.72      ],\n",
       "       [ 38300.67      , -21733.33      , -16567.34      ],\n",
       "       [ 37359.95      , -22638.05      , -14721.9       ],\n",
       "       [ 40063.28333333, -20020.71666667, -20042.56666667],\n",
       "       [ 38524.27666667, -21511.72333333, -17012.55333333],\n",
       "       [ 37639.51      , -22407.49      , -15232.02      ],\n",
       "       [ 39050.38      , -20965.62      , -18084.76      ],\n",
       "       [ 36785.20333333, -23262.79666667, -13522.40666667],\n",
       "       [ 37128.34666667, -22896.65333333, -14231.69333333],\n",
       "       [ 37299.42333333, -22766.57666667, -14532.84666667],\n",
       "       [ 37397.75333333, -22668.24666667, -14729.50666667],\n",
       "       [ 37777.40333333, -22274.59666667, -15502.80666667],\n",
       "       [ 39030.63333333, -21025.36666667, -18005.26666667],\n",
       "       [ 39947.7       , -20120.3       , -19827.4       ],\n",
       "       [ 39683.14333333, -20342.85666667, -19340.28666667],\n",
       "       [ 39515.46333333, -20560.53666667, -18954.92666667],\n",
       "       [ 37507.62333333, -22497.37666667, -15010.24666667],\n",
       "       [ 38833.69666667, -21155.30333333, -17678.39333333],\n",
       "       [ 37136.45      , -22941.55      , -14194.9       ],\n",
       "       [ 40012.65666667, -19989.34333333, -20023.31333333],\n",
       "       [ 37880.5       , -22128.5       , -15752.        ],\n",
       "       [ 40279.85333333, -19777.14666667, -20502.70666667],\n",
       "       [ 37780.41666667, -22284.58333333, -15495.83333333],\n",
       "       [ 36746.68333333, -23322.31666667, -13424.36666667],\n",
       "       [ 36979.53333333, -23004.46666667, -13975.06666667],\n",
       "       [ 37717.89      , -22353.11      , -15364.78      ],\n",
       "       [ 37577.28333333, -22425.71666667, -15151.56666667],\n",
       "       [ 39776.46666667, -20212.53333333, -19563.93333333],\n",
       "       [ 38540.06333333, -21504.93666667, -17035.12666667],\n",
       "       [ 37806.96      , -22228.04      , -15578.92      ],\n",
       "       [ 38934.81666667, -21120.18333333, -17814.63333333],\n",
       "       [ 39252.18      , -20782.82      , -18469.36      ],\n",
       "       [ 40178.53      , -19830.47      , -20348.06      ],\n",
       "       [ 37971.59333333, -22031.40666667, -15940.18666667]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# subtract each element from the average of its row\n",
    "arr - np.mean(arr, axis=-1, keepdims=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b796fd3-19f2-4bf4-b872-7d58c65dc0c7",
   "metadata": {},
   "source": [
    "Some additional NumPy operations Ponder currently supports include:\n",
    "\n",
    "- Element-wise matrix operations such as addition, subtraction, multiplication, division, power\n",
    "- Axis-collapsing or reducing operations such as min, max, sum, product, mean\n",
    "- Multi-array operations such as maximum or minimum\n",
    "- And many others, such as where, ravel, and transpose"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc4548a8-c429-4fd9-8bf1-551f740fd60a",
   "metadata": {},
   "source": [
    "# Summary\n",
    "\n",
    "In this tutorial, we demonstrated how you can Ponder to run NumPy operations natively directly on your database. Congrats! This wraps up our 8-part tutorial series on how you can use Ponder to start accelerating your data science workflow. We're so excited to see what you can do with Ponder! If you have any questions, issues, or want to show off something cool you've built with Ponder:\n",
    "- [Join us on Slack](https://modin.org/slack.html) and share your thoughts on #ponder-support, or\n",
    "- Drop us a note at support@ponder.io."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}