Optimize array_equivalent for NDFrame.equals

[TomAugspurger]

Closes #35249

In [1]: import numpy as np
In [2]: import pandas as pd
In [3]: N = 10**3
In [4]: float_df = pd.DataFrame(np.random.randn(N, N))
# 1.0.5
1.52 ms ± 94 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

# master
83.6 ms ± 1.15 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

# branch
31.2 ms ± 2.06 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

And proof of this being better for short-circuiting:

In [4]: N = 10**3

In [5]: a = pd.DataFrame(np.random.randn(N, N))

In [6]: b = pd.DataFrame(np.random.randn(N, N))

In [7]: %timeit a.equals(b)
# 1.0.5
1.97 ms ± 127 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [10]: %timeit a.equals(b)
56.4 µs ± 1.67 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

Discussion: the key thing here is that BlockManager.equals requires that the dtypes of left and right match. If they don't then we've already returned False. That lets us do a few things

1. Avoid many is_foo_dtype checks.
2. Use optimized checks like np.isnan
3. Avoid the slow case of considering np.array([1, 2]) equivalent to np.array([1.0, 2.0]).

[simonjayhawkins]

restarting failed travis job

=========================== short test summary info ============================
FAILED pandas/tests/io/parser/test_common.py::test_chunks_have_consistent_numerical_type[python]
= 1 failed, 69943 passed, 4255 skipped, 1020 xfailed, 13 warnings in 808.93s (0:13:28) =

[jreback]

i am not sure i understand why you actually need this
the dtype check is cheap compared to the actual check no?

[TomAugspurger]

1. They add up, especially when doing things columnwise for small, wide dataframes
2. We get to skip a few other checks, like swapping np.isnan for pd.isna for float, and we can skip the empty check at https://github.com/pandas-dev/pandas/pull/35328/files#diff-ff8364cee9a3e1ef3a3825cb2cdd26d8L431.

[jreback]

ok that's fine

[jreback]

maybe make all of these helpers private functions

[jbrockmendel]

what about use_inf_as_na?

Can this be re-used on L426? (also the prod(shape) check on L424-425 i think is redundant with earlier shape check)

[TomAugspurger]

It seems like 1.0.5 doesn't care about that

In [15]: df1 = pd.DataFrame({"A": np.array([np.nan, 1, np.inf])})

In [16]: df2 = pd.DataFrame({"A": np.array([np.nan, 1, np.nan])})

In [17]: with pd.option_context('mode.use_inf_as_na', True):
    ...:     print(df1.equals(df2))
    ...:
    ...:
False

[jbrockmendel]

Makes sense, thanks for checking

[TomAugspurger]

Travis finished. I'm going to selfishly ignore #35328 (comment) so that I can get the RC process rolling, but happy to engage more on followups if I missed something.