[C++] Fatal error condition occurred in aws_thread_launch

[asfimport]

I am encountering this error using the latest version of pyarrow (9.0.0) w/ Python 3.10. I received pyarrow as an indirect dependency of google-cloud-bigquery.

It seems like this issue was never fully fixed:

#20039

Fatal error condition occurred in /opt/vcpkg/buildtrees/aws-c-io/src/9e6648842a-364b708815.clean/source/event_loop.c:72: aws_thread_launch(&cleanup_thread, s_event_loop_destroy_async_thread_fn, el_group, &thread_options) == AWS_OP_SUCCESS
Exiting Application
################################################################################
Stack trace:
################################################################################
/usr/local/lib/python3.10/dist-packages/pyarrow/libarrow.so.900(+0x200af06) [0x7f4719c39f06]
/usr/local/lib/python3.10/dist-packages/pyarrow/libarrow.so.900(+0x20028e5) [0x7f4719c318e5]
/usr/local/lib/python3.10/dist-packages/pyarrow/libarrow.so.900(+0x1f27e09) [0x7f4719b56e09]
/usr/local/lib/python3.10/dist-packages/pyarrow/libarrow.so.900(+0x200ba3d) [0x7f4719c3aa3d]
/usr/local/lib/python3.10/dist-packages/pyarrow/libarrow.so.900(+0x1f25948) [0x7f4719b54948]
/usr/local/lib/python3.10/dist-packages/pyarrow/libarrow.so.900(+0x200ba3d) [0x7f4719c3aa3d]
/usr/local/lib/python3.10/dist-packages/pyarrow/libarrow.so.900(+0x1ee0b46) [0x7f4719b0fb46]
/usr/local/lib/python3.10/dist-packages/pyarrow/libarrow.so.900(+0x194546a) [0x7f471957446a]
/lib/x86_64-linux-gnu/libc.so.6(+0x45495) [0x7f473ab81495]
/lib/x86_64-linux-gnu/libc.so.6(on_exit+0) [0x7f473ab81610]
/usr/local/bin/python(+0x2790ab) [0x55965e9a00ab]
/usr/local/bin/python(+0x265d7f) [0x55965e98cd7f]
/usr/local/bin/python(PyErr_PrintEx+0x1d) [0x55965e98ccad]
/usr/local/bin/python(_PyRun_SimpleFileObject+0x1d6) [0x55965e989576]
/usr/local/bin/python(_PyRun_AnyFileObject+0x43) [0x55965e989243]
/usr/local/bin/python(Py_RunMain+0x2be) [0x55965e979b6e]
/usr/local/bin/python(Py_BytesMain+0x2d) [0x55965e94fe6d]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f473ab65d90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f473ab65e40]
/usr/local/bin/python(_start+0x25) [0x55965e94fd65]
/usr/bin/bash: line 172:    20 Aborted                 (core dumped) 

I think this issue might only occur on certain kernel versions? It's strange, I cannot reproduce it running the image on my machine (ubuntu 4.15.0-180-generic), but some of our CI runners (those w/ ubuntu using kernel 5.4.0-90-generic) consistently have this issue. I will try to gather some more information.

Environment: Python 3.10.4 in a ubuntu:22.04 docker image w/ kernel 5.4.0-90-generic
Reporter: Matthew Schurr
Assignee: Kouhei Sutou / @kou

Related issues:

• [C++] Fatal error condition occurred in aws_thread_launch (relates to)

PRs and other links:

• GitHub Pull Request #14157

_{Note: This issue was originally created as ARROW-17501. Please see the migration documentation for further details.}

[asfimport]

Weston Pace / @westonpace:
ARROW-15141 was a bug in a downstream AWS dependency. Can you determine which version of aws-sdk-cpp you are compiling Arrow with? Or, if you are not compiling Arrow, but getting it from PyPI or Conda can you figure out exactly which binary you are downloading? For example, is it PyPI or Conda? If Conda then which channel? If PyPI then which architecture/distro? Etc?

[asfimport]

Matthew Schurr:
I'm not compiling it, it's coming from pip. 

Collecting pyarrow==9.0.0  Downloading pyarrow-9.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (35.3 MB)     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 35.3/35.3 MB 109.1 MB/s eta 0:00:00 

[asfimport]

Kouhei Sutou / @kou:
Ah, sorry.
I upgraded aws-sdk-cpp installed from vcpkg by ARROW-12626. This is the cause of this problem.

Upstream doesn't solve this problem yet. So we need to add a patch for this case to our ci/vcpkg/ports.patch.

[asfimport]

Kouhei Sutou / @kou:
Or we can revert https://github.com/apache/arrow/pull/13244/files#diff-d44d8d48c04a855baa07c9c04bcc9373c924ec74bd1d9bda62429b2339c9461aL120 to use bundled aws-sdk-cpp.

[asfimport]

Ian Cook / @ianmcook:
@kou  would you be able to open a PR for this? We would like to prioritize getting this resolved. Thanks!

[asfimport]

Kouhei Sutou / @kou:
OK. I'll do it.

[asfimport]

Kouhei Sutou / @kou:
Issue resolved by pull request 14157
#14157

[asfimport]

Ian Cook / @ianmcook:
Thank you @kou!