Failure to build pyarrow wheel due to Cython incompatibility during Arrow install

[branfosj]

Building Arrow-11.0.0-gfbf-2022b.eb I am hitting the error below.

The error looks to be related to the changes in Cython 3, which was released yesterday. However, I am confused as to why we're not using the Cython (0.29.32) in EB Python.

Adding the following to the Arrow easyconfig fixes it, further pointing at Cython 3 being the issue (see https://github.com/bear-rsg/bear-eb/blob/b597bbce58316d85d0d00dc1a95047c5fe9152e3/easyconfigs/a/Arrow/Arrow-11.0.0-gfbf-2022b.eb).

local_install_pyarrow_cmds += "sed -i 's/cython >= 0.29.22/cython >= 0.29.22,<3/g' pyproject.toml && "

Error:

      [  1%] Compiling Cython CXX source for lib...

      Error compiling Cython file:
      ------------------------------------------------------------
      ...

      cdef extern from "Python.h":
          int PySlice_Check(object)


      cdef int check_status(const CStatus& status) nogil except -1
                                                                  ^
      ------------------------------------------------------------

      pyarrow/lib.pxd:67:60: The keyword 'nogil' should appear at the end of the function signature line. Placing it before 'except' or 'noexcept' will be disallowed in a future version of Cython.


      Error compiling Cython file:
      ------------------------------------------------------------
      ...
      ArrowIOError = IOError


      # This function could be written directly in C++ if we didn't
      # define Arrow-specific subclasses (ArrowInvalid etc.)
      cdef int check_status(const CStatus& status) nogil except -1:
                                                                  ^
      ------------------------------------------------------------

      pyarrow/error.pxi:82:60: The keyword 'nogil' should appear at the end of the function signature line. Placing it before 'except' or 'noexcept' will be disallowed in a future version of Cython.

      Error compiling Cython file:
      ------------------------------------------------------------
      ...
                  raise ArrowException(message)


      # This is an API function for C++ PyArrow
      cdef api int pyarrow_internal_check_status(const CStatus& status) \
              nogil except -1:
                             ^
      ------------------------------------------------------------

      pyarrow/error.pxi:143:23: The keyword 'nogil' should appear at the end of the function signature line. Placing it before 'except' or 'noexcept' will be disallowed in a future version of Cython.

      gmake[2]: *** [CMakeFiles/lib_pyx.dir/build.make:71: CMakeFiles/lib_pyx] Error 1
      gmake[1]: *** [CMakeFiles/Makefile2:146: CMakeFiles/lib_pyx.dir/all] Error 2
      gmake: *** [Makefile:136: all] Error 2
      error: command '/rds/bear-apps/2022b/EL8-ice/software/CMake/3.24.3-GCCcore-12.2.0/bin/cmake' failed with exit code 2

[Micket]

However, I am confused as to why we're not using the Cython (0.29.32) in EB Python.

I'm also confused by this sentence. Did you mean you are confused as to why we see this error *because we are using version 0.29.32 and not 3+"?

[branfosj]

However, I am confused as to why we're not using the Cython (0.29.32) in EB Python.

I'm also confused by this sentence. Did you mean you are confused as to why we see this error *because we are using version 0.29.32 and not 3+"?

The error is seen when pyarrow is built with Cython 3. It is caused by one of the breaking changing in Cython 3 and that was relased this week. So, as we see the error when building pyarrow (during a postinstall command in Arrow), we must be obtaining Cython 3 during that build process and not using the Cython provided with EB Python.

Upstream commit: apache/arrow@de8df23

[jfgrimm]

@branfosj see https://discuss.python.org/t/no-way-to-pin-build-dependencies/29833

[branfosj]

@branfosj see https://discuss.python.org/t/no-way-to-pin-build-dependencies/29833

Thanks. I think that provides the correct fix: we should be adding --no-build-isolation to the pip install.

[branfosj]

see #18348

[jfgrimm]

indeed, that's something we already automatically inject in the PythonPackage easyblock