[Python] accept pyarrow values / scalars in constructor functions ?

[asfimport]

Currently, functions like `pyarrow.array` don't accept pyarrow Arrays, or also not scalars of it:

In [42]: arr = pa.array([1, 2, 3])

In [43]: pa.array(arr)
...
ArrowInvalid: Could not convert 1 with type pyarrow.lib.Int64Value: did not recognize Python value type when inferring an Arrow data type

In [44]: pa.array(list(arr))
...
ArrowInvalid: Could not convert 1 with type pyarrow.lib.Int64Value: did not recognize Python value type when inferring an Arrow data type

Do we want to allow those / recognize those here? (the first case could even have a fastpath, as we don't need to do it element by element).

Also scalars are not supported:

In [46]: type(arr.sum())
Out[46]: pyarrow.lib.Int64Scalar

In [47]: pa.array([arr.sum()])
...
ArrowInvalid: Could not convert 6 with type pyarrow.lib.Int64Scalar: did not recognize Python value type when inferring an Arrow data type

And also in other functions we don't accept arrow scalars / values:

In [48]: string = pa.array(['a'])[0]

In [49]: type(string)
Out[49]: pyarrow.lib.StringValue

In [50]: pa.field(string, pa.int64())
...
TypeError: expected bytes, pyarrow.lib.StringValue found

 

 

Reporter: Joris Van den Bossche / @jorisvandenbossche

Related issues:

• [Python] pa.array() doesn't support pa.lib.TimestampScalar objects (duplicates)
• [Python] Array constructor doesn't support arrow scalars. (is duplicated by)
• [Python] Scalar constructors should be callable (relates to)
• [Python] Allow pa.array/pa.chunked_array to infer pa.NA when in a non pyarrow container (relates to)

_{Note: This issue was originally created as ARROW-5295. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
I think allowing Arrays is fine. I'm not sure we want to make Scalars first-class citizens like Numpy scalars, though. At least from an implementation POV, it may add a lot of code all over the place...

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Additional case (from ARROW-6222): pyarrow Arrays are also not recognized as list-like when converting/inferring a list array:

In [43]: pa.array([np.array([1, 1]), np.array([2, 2, 2])])                                                                                                                                                         
Out[43]: 
<pyarrow.lib.ListArray object at 0x7f258fa9d0a0>
[
  [
    1,
    1
  ],
  [
    2,
    2,
    2
  ]
]

In [44]: pa.array([pa.array([1, 1]), pa.array([2, 2, 2])])                                                                                                                                                         
---------------------------------------------------------------------------
ArrowInvalid                              Traceback (most recent call last)
<ipython-input-44-a22e0a500750> in <module>
----> 1 pa.array([pa.array([1, 1]), pa.array([2, 2, 2])])

~/scipy/repos/arrow/python/pyarrow/array.pxi in pyarrow.lib.array()

~/scipy/repos/arrow/python/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()

~/scipy/repos/arrow/python/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowInvalid: Could not convert [
  1,
  1
] with type pyarrow.lib.Int64Array: did not recognize Python value type when inferring an Arrow data type

So list (or array) of numpy arrays works, but list of pyarrow arrays not. Again, not the most typical use case of pyarrow Arrays, so not sure we should add this capability.

(although we might want to find a general solution for array-like objects (eg pytorch.Tensors, see ARROW-6222), and a solution for that (somehow trying to coerce to a numpy array?) might also solve the case of a list of arrow arrays)

[asfimport]

Wes McKinney / @wesm:
A "simple" workaround would be to invoke Scalars' as_py method if they're passed in. That would add perf overhead, though, since we'd need to do isinstance checks.

Another option is to "sanitize" inputs (using a helper function) only in the case of failure on the initial try. So the normal use case won't be affected

[jorisvandenbossche]

The part about accepting an Array object is solved now with #34275. Leaving this issue open for the Scalar case.