[Core] numpy > 2 not supported

[paaragon]

What happened + What you expected to happen

Working on the Qiskit Serverless project, we discover an issue in the client ray using numpy>2:

Failed to start supervisor actor raysubmit_er8zgdCRKuMwP2mL: 'numpy.core.multiarray failed to import'. Full traceback:
Traceback (most recent call last):
  File "/usr/local/lib64/python3.11/site-packages/ray/dashboard/modules/job/job_manager.py", line 544, in submit_job
    ).remote(submission_id, entrypoint, metadata or {}, self._gcs_address)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib64/python3.11/site-packages/ray/actor.py", line 833, in remote
    return actor_cls._remote(args=args, kwargs=kwargs, **updated_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib64/python3.11/site-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib64/python3.11/site-packages/ray/util/tracing/tracing_helper.py", line 388, in _invocation_actor_class_remote_span
    return method(self, args, kwargs, *_args, **_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib64/python3.11/site-packages/ray/actor.py", line 1161, in _remote
    actor_id = worker.core_worker.create_actor(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "python/ray/_raylet.pyx", line 4114, in ray._raylet.CoreWorker.create_actor
  File "python/ray/_raylet.pyx", line 4119, in ray._raylet.CoreWorker.create_actor
  File "python/ray/_raylet.pyx", line 835, in ray._raylet.prepare_args_and_increment_put_refs
  File "python/ray/_raylet.pyx", line 826, in ray._raylet.prepare_args_and_increment_put_refs
  File "python/ray/_raylet.pyx", line 875, in ray._raylet.prepare_args_internal
  File "/usr/local/lib64/python3.11/site-packages/ray/_private/worker.py", line 679, in get_serialization_context
    context_map[job_id] = serialization.SerializationContext(self)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib64/python3.11/site-packages/ray/_private/serialization.py", line 163, in __init__
    serialization_addons.apply(self)
  File "/usr/local/lib64/python3.11/site-packages/ray/util/serialization_addons.py", line 39, in apply
    _register_custom_datasets_serializers(serialization_context)
  File "/usr/local/lib64/python3.11/site-packages/ray/_private/arrow_serialization.py", line 46, in _register_custom_datasets_serializers
    import pyarrow as pa  # noqa: F401
    ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib64/python3.11/site-packages/pyarrow/__init__.py", line 65, in <module>
    import pyarrow.lib as _lib
  File "pyarrow/lib.pyx", line 36, in init pyarrow.lib
ImportError: numpy.core.multiarray failed to import

Versions / Dependencies

ray[default,data]>=2.30.0, <2.35.0
Python=3.11
numpy>2

Reproduction script

[wip]

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[rynewang]

Hi can you print your pyarrow version? from the stack trace it seems pyarrow imports numpy.core.multiarray which seems to be gone in numpy>2.

Also curious why you set ray < 2.35.0?

[Tansito]

Hi, sorry for the delay in the answer @rynewang

can you print your pyarrow version? from the stack trace it seems pyarrow imports numpy.core.multiarray which seems to be gone in numpy>2.

Yes, let us try to collect that info for you 👍

Also curious why you set ray < 2.35.0?

Some weeks ago we tried to update from 2.34.0 to 2.35.0 our ray dependency but something was not working from our tests. If I remember correctly the problem was that we were not being able to connect with the dashboard but we will try to collect the stacktrace for this one an open another issue in case we continue seeing the problem.

Thank you @rynewang ! 🙏

[Tansito]

The pyarrow version that we are using is the next one: pyarrow==14.0.1. So probably this version doesn't support the new numpy? Maybe if we are able to update the ray version that we are using we could solve the problem, does it make sense, @rynewang ?

[paul-twelvelabs]

We ran into the same pyarrow/numpy > 2 issue on ray==2.38.0. It'd be great if we could bump pyarrow (note that pyarrow==14.0.1 is 1 year old).

[rynewang]

Per numpy/numpy#26191 it seems you need to upgrade pyarrow to >= 16.0 if you want to use numpy 2. See apache/arrow#39532

[Tansito]

Thanks @rynewang , I'm going to give a try today and let you know if this simple change works. Thanks for your support.

[Tansito]

From Qiskit/qiskit-serverless#1538 we applied the change to update pyarrow and everything worked like a charm with the new numpy major version. Thanks for your support @rynewang 👍

EDIT: feel free to close this issue as solved!