Spike: Investigate using native PyArrow backed data types

[gsheni]

• With the upcoming pandas 1.5.0 release, pandas will support most data types with pyarrow backed Arrays and Data Types (https://pandas.pydata.org/docs/dev/whatsnew/v1.5.0.html#native-pyarrow-backed-extensionarray)
• TLDR of using Apache Arrow:
  • faster computations, less memory usage --> Woodwork DataFrames are faster & take up less memory

Initial Investigation

Install

pip install pandas==1.5.0rc0

pip install pyarrow --upgrade

Mapping

import pandas as pd 
import pyarrow as pa
from pandas.core.arrays.arrow.dtype import ArrowDtype

PANDAS_DTYPE_TO_ARROWDTYPE = {
    "np.int64": pa.int64(),
    "int64": pa.int64(),
    "Int64": pa.int64(),
    "np.float64": pa.float64(),
    "float64": pa.float64(),
    "Float64Dtype": pa.float64(),
    "np.object": pa.string(),
    "object": pa.string(),
    "string": pa.string(),
    "StringDtype": pa.string(),
    "datetime64[s]": pa.timestamp(unit="s", tz=None),
    "datetime64[s, US/Eastern]": pa.timestamp(unit="s", tz='US/Eastern'),
    "datetime64[s, US/Central]": pa.timestamp(unit="s", tz='US/Central'),
    "datetime64[ms]": pa.timestamp(unit="ms", tz=None),
    "datetime64[ms, US/Eastern]": pa.timestamp(unit="ms", tz='US/Eastern'),
    "datetime64[ms, US/Central]": pa.timestamp(unit="ms", tz='US/Central'),
    "datetime64[us]": pa.timestamp(unit="us", tz=None),
    "datetime64[us, US/Eastern]": pa.timestamp(unit="us", tz='US/Eastern'),
    "datetime64[us, US/Central]": pa.timestamp(unit="us", tz='US/Central'),
    "datetime64[ns]": pa.timestamp(unit="ns", tz=None),
    "datetime64[ns, US/Eastern]": pa.timestamp(unit="ns", tz='US/Eastern'),
    "datetime64[ns, US/Central]": pa.timestamp(unit="ns", tz='US/Central'),
    "np.bool_": pa.bool_(),
    "boolean": pa.bool_(),
    "BooleanDtype": pa.bool_()
}
for k, v in PANDAS_DTYPE_TO_ARROWDTYPE.items():
    PANDAS_DTYPE_TO_ARROWDTYPE[k] = ArrowDtype(v)