Fix ORC and JSON tests failures for pandas 2.2

[mroeschke]

Description

test_order_nested_json_reader was refactored to use assert_eq instead of comparing via pyarrow. This was failing in pandas 2.2 due to pandas-dev/pandas#57429

test_orc_reader_trailing_nulls I believe was failing due to a change in how integers are compared with assert_series_equal: pandas-dev/pandas#55882. The "casting workaround" doesn't seem necessary in pandas 2.2 so just avoiding it all together

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

have a question

[galipremsagar]

Can pandas now handle comparing the equality of nested types/values? I remember the last time I checked pandas wasn't able to that properly and thus we resorted to use pyarrow for comparisons in case of nested types.

[galipremsagar]

I just was able to triage the issue here, looks like pd.read_json has a regression where the dataframe is expected to have a RangeIndex but a materialized Index of int64 dtype is being returned in pandas-2.2. We should just change this test to the following:

diff --git a/python/cudf/cudf/tests/test_json.py b/python/cudf/cudf/tests/test_json.py
index ec980adc33..5a459e98d1 100644
--- a/python/cudf/cudf/tests/test_json.py
+++ b/python/cudf/cudf/tests/test_json.py
@@ -1179,6 +1179,9 @@ class TestNestedJsonReaderCommon:
 
     def test_order_nested_json_reader(self, tag, data):
         expected = pd.read_json(StringIO(data), lines=True)
+        if PANDAS_GE_200:
+            # TODO: Remove after bug fix: <Pandas-bug-URL>
+            expected = expected.reset_index(drop=True)
         target = cudf.read_json(StringIO(data), lines=True)
         if tag == "dtype_mismatch":
             with pytest.raises(AssertionError):

[mroeschke]

Ah I see. Yeah testing of nested data isn't well tested in pandas so probably better to use pyarrow to compare here. Will incorporate your change

[galipremsagar]

I just was able to triage the issue here, looks like pd.read_json has a regression where the dataframe is expected to have a RangeIndex but a materialized Index of int64 dtype is being returned in pandas-2.2. We should just change this test to the following:

diff --git a/python/cudf/cudf/tests/test_json.py b/python/cudf/cudf/tests/test_json.py
index ec980adc33..5a459e98d1 100644
--- a/python/cudf/cudf/tests/test_json.py
+++ b/python/cudf/cudf/tests/test_json.py
@@ -1179,6 +1179,9 @@ class TestNestedJsonReaderCommon:
 
     def test_order_nested_json_reader(self, tag, data):
         expected = pd.read_json(StringIO(data), lines=True)
+        if PANDAS_GE_200:
+            # TODO: Remove after bug fix: <Pandas-bug-URL>
+            expected = expected.reset_index(drop=True)
         target = cudf.read_json(StringIO(data), lines=True)
         if tag == "dtype_mismatch":
             with pytest.raises(AssertionError):

[galipremsagar]

Thanks @mroeschke !

[galipremsagar]

/merge