Clean up special casing in as_column for non-typed input

[mroeschke]

Description

Cleans up a lot of special casing for array-like inputs in as_column that are not typed.

The logic now is essentially

if dtype is not None:
    if a pandas type or object type:
        # parse with pandas
    else
        # parse with arrow
else:
    try:
        # parse with arrow
    except:
        # parse with pandas
# create column

Allows sharing of logic in the pandas/arrow code paths for consistency for different inputs and kinda aligns with the comment in #14627 (comment) cc @wence-

This also fixes a bug where as_column(list-like, dtype=cudf.CategoricalDtype|IntervalDtype(...)) did not keep its specific metadata

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[wence-]

I think this is a good direction!

[wence-]

suggestion

Suggested change

	def can_memoryview(arbitrary: Any) -> bool:
	try:
	memoryview(arbitrary)
	return True
	except TypeError:
	return False
	def as_memoryview(arbitrary: Any) -> Optional[memoryview]:
	try:
	return memoryview(arbitrary)
	except TypeError:
	return None

[wence-]

Suggested change

	elif can_memoryview(arbitrary):
	data = as_column(
	memoryview(arbitrary), dtype=dtype, nan_as_null=nan_as_null
	)
	elif (view := as_memoryview(arbitrary)) is not None:
	data = as_column(view, dtype=dtype, nan_as_null=nan_as_null)

Then one could merge this handler into the isinstance(arbitrary, memoryview) case as well with:

if isinstance((view := as_memoryview(arbitrary)), memoryview):
    data = as_column(np.asarray(view), dtype=dtype, nan_as_null=nan_as_null)

Which brings me to a further suggestion for this function. One of the issues is that it's very difficult to read the control flow. AIUI, we basically have lots of different branches to check if we can produce a Column, assign that to data and then eventually return data. But it's unclear (for example here) when we do data = as_column(memoryview(arb), ...) whether the is any further processing to be done, or if we are now ready to return the final answer.

I think I would prefer if all of these branches return ... rather than assigning to a variable and going to fallthrough.

[mroeschke]

I think I would prefer if all of these branches return ... rather than assigning to a variable and going to fallthrough.

Good point. I'll try to have every branch do return ... for clarity. One aspect I'm not sure if when dtype= is specified that an "astype" is done if the input is typed for all branches

[wence-]

Suggested change

	from_pandas = True if nan_as_null is None else nan_as_null
	from_pandas = nan_as_null is None or nan_as_null

[wence-]

question: are we guaranteed that arbitrary is iterable at this point?

[mroeschke]

I believe so since an initial pa.array call proxy-checks that, but this would fail here for generators for example so I'll make sure this branch casts the input to list or similar

[wence-]

Do we want to go via arrow first and only then try and fall back to pandas? Or should we just go via pandas? I note this PR does not fix #14627 because I think we still go down the same (different) arrow vs pandas codepath.

[wence-]

The logic now is essentially

if dtype is not None:
    if a pandas type or object type:
        # parse with pandas
    else
        # parse with arrow
else:
    try:
        # parse with arrow
    except:
        # parse with pandas
# create column

question I still wonder if this is more complicated than it needs to be. If we're in these branches we have an "arbitrary" object that wasn't a Series/array/arrow-array.

Is there a disadvantage to (in all cases) doing: data = parse_with_pandas(arbitrary). It seems like the logic as shown above could have different behaviour passing a pandas scalar object in (that arrow recognises) depending on whether one additionally specifies dtype.

Concretely:

as_column(pd_scalar_obj) # parsed with arrow
as_column(pd_scalar_obj, dtype=pd_scalar_obj.dtype) # parsed with pandas

Are there objects that arrow can convert that pandas cannot?

[mroeschke]

Are there objects that arrow can convert that pandas cannot?

Parsing via Arrow first is beneficial for

1. Returning List or Struct typed arrays. (Though looks like we need a carve-out for a list of Series for example which we support as List type)
2. nan_as_null keyword flexibility and not having an NA like value always cast the result to float/object

In [1]: import pandas as pd

In [2]: pd.Series([1, None])
Out[2]: 
0    1.0
1    NaN
dtype: float64

In [3]: import pyarrow as pa

In [4]: pa.array([1, None])
Out[4]: 
<pyarrow.lib.Int64Array object at 0x7face4728580>
[
  1,
  null
]

We could at this point do data introspection for these two cases and parse with arrow if we see either case otherwise always parse with pandas.

[mroeschke]

Going to close in favor of #15276