[Python] ArrowInvalid: Could not convert <pyarrow.Int32Scalar: 1> with type pyarrow.lib.Int32Scalar: tried to convert to int32

[randolf-scholz]

Describe the bug, including details regarding any error messages, version, and platform.

Was trying to convert some json-data to list[int]:

import pyarrow as pa
import json

print(pa.__version__)
data = pa.array(["[1,2,3]", "[4,5,6]"], type=pa.string())
dtype = pa.list_(pa.int32())


def convert_py_list(x:pa.StringScalar) -> list[int]:
    return json.loads(x.as_py())

def convert_pa_list(x:pa.StringScalar) -> list[pa.Scalar]:
    return pa.scalar(json.loads(x.as_py()), type=dtype)

pa.array([convert_py_list(x) for x in data], type=dtype)  # ✔
pa.array([convert_pa_list(x) for x in data], type=dtype)  # ✘ ArrowInvalid

pyarrow version 12.0.1

Component(s)

Python

[AlenkaF]

convert_pa_list returns a list of pyarrow scalars (ListScalars) comparing to a list of lists:

>>> [convert_py_list(x) for x in data]
[[1, 2, 3], [4, 5, 6]]
>>> [convert_pa_list(x) for x in data]
[<pyarrow.ListScalar: [1, 2, 3]>, <pyarrow.ListScalar: [4, 5, 6]>]

Currently pa.array constructor doesn't except pyarrow scalars, see: #21761
We are currently working on this feature and I hope it will get added to 13.0.0 release.

[jorisvandenbossche]

Closing as a duplicate of #21761