[BUG] Segfault reading parquet file in C++ when libcudf is built with GCC 12.3.0

[cryos]

Describe the bug
Reading a parquet file written by Apache Arrow fails when compiling libcudf with GCC 12.3.0 results in a segfault - cudf::io::read_parquet function. The docs about building from source state GCC version 11.4+.

Steps/Code to reproduce bug
Build libcudf with GCC 11.4.0 and GCC 12.3.0 on Ubuntu 22.04. Following the C++ guide it looked like GCC 12.3.0 would be OK, everything compiled without issue but I see segfaults when loading parquet files written by Apache Arrow.

From the cpp directory I configured libcudf to compile with the default compiler (GCC 12.3.0) and the GCC 11.4.0 compiler.

CC=gcc-11 CXX=g++-11 cmake -DCMAKE_CUDA_ARCHITECTURES=NATIVE -DCMAKE_INSTALL_PREFIX=/tmp/cudfgcc11 -B gcc11 -S .
cmake --build gcc11 --target install
cmake -DCMAKE_CUDA_ARCHITECTURES=NATIVE -DCMAKE_INSTALL_PREFIX=/tmp/cudfgcc12 -B gcc12 -S .
cmake --build gcc12 --target install

I used the following Python snippet to create a parquet file:

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

df = pd.DataFrame({'one': [-1, 0, 2.5],
                   'two': ['foo', 'bar', 'baz'],
                   'three': [True, False, True]},
                   index=list('abc'))
table = pa.Table.from_pandas(df)

pq.write_table(table, 'example.parquet')

Create a conda environment to write an example file:

mamba create -n pyarrow pyarrow pandas
mamba activate pyarrow

This was used to create exmaple.parquet that was placed in the directory the executable was run from (source tree in this case).

I then used the following CMakeLists.txt to build the reader:

cmake_minimum_required(VERSION 3.29 FATAL_ERROR)

find_package(cudf REQUIRED)

add_executable(cudfparquet main.cpp)
target_link_libraries(cudfparquet cudf::cudf)

The main.cpp was:

#include <cudf/io/parquet.hpp>
#include <iostream>

int main() {
  auto source = cudf::io::source_info{"example.parquet"};
  auto pq_args = cudf::io::parquet_reader_options::builder(source);
  auto result = cudf::io::read_parquet(pq_args);
  std::cout << "Success" << std::endl;
}

Building the example code against the two libraries produced:

cmake -DCMAKE_PREFIX_PATH=/tmp/cudfgcc11/ -B cudfgcc11 -S .
cmake --build cudfgcc11
cmake -DCMAKE_PREFIX_PATH=/tmp/cudfgcc12/ -B cudfgcc12 -S .
cmake --build cudfgcc12
# Run the examples
./cudfgcc11/cudfparquet 
Success
./cudfgcc12/cudfparquet 
Segmentation fault (core dumped)

When running in gdb I have a backtrace indicating that the metadata reading is failing.

Backtrace

gdb ./cudfgcc12/cudfparquet 
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04.2) 12.1
(gdb) r
Starting program: /home/marcus/src/cudfparquet/cudfgcc12/cudfparquet 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffe8dff000 (LWP 118898)]
[New Thread 0x7fffdc9de000 (LWP 118902)]
[New Thread 0x7fffdc1dd000 (LWP 118903)]
[New Thread 0x7fffda9bb000 (LWP 118904)]
[Detaching after vfork from child process 118905]
[New Thread 0x7fffd9dff000 (LWP 118908)]
[New Thread 0x7fffd95fe000 (LWP 118909)]
[New Thread 0x7fffd8dfd000 (LWP 118910)]
[New Thread 0x7fffd1fff000 (LWP 118911)]
[New Thread 0x7fffd17fe000 (LWP 118912)]

Thread 1 "cudfparquet" received signal SIGSEGV, Segmentation fault.
0x00007ffff106659c in std::_Function_handler<void (unsigned int, cudf::io::parquet::detail::CompactProtocolReader*), cudf::io::parquet::detail::parquet_field_struct_list<cudf::io::parquet::detail::SchemaElement>::parquet_field_struct_list(int, std::vector<cudf::io::parquet::detail::SchemaElement, std::allocator<cudf::io::parquet::detail::SchemaElement> >&)::{lambda(unsigned int, cudf::io::parquet::detail::CompactProtocolReader*)#1}>::_M_invoke(std::_Any_data const&, unsigned int&&, cudf::io::parquet::detail::CompactProtocolReader*&&) () from /tmp/cudfgcc12/lib/libcudf.so
(gdb) bt
#0  0x00007ffff106659c in std::_Function_handler<void (unsigned int, cudf::io::parquet::detail::CompactProtocolReader*), cudf::io::parquet::detail::parquet_field_struct_list<cudf::io::parquet::detail::SchemaElement>::parquet_field_struct_list(int, std::vector<cudf::io::parquet::detail::SchemaElement, std::allocator<cudf::io::parquet::detail::SchemaElement> >&)::{lambda(unsigned int, cudf::io::parquet::detail::CompactProtocolReader*)#1}>::_M_invoke(std::_Any_data const&, unsigned int&&, cudf::io::parquet::detail::CompactProtocolReader*&&) () from /tmp/cudfgcc12/lib/libcudf.so
#1  0x00007ffff106326d in cudf::io::parquet::detail::CompactProtocolReader::read(cudf::io::parquet::detail::FileMetaData*) () from /tmp/cudfgcc12/lib/libcudf.so
#2  0x00007ffff10cadc6 in cudf::io::parquet::detail::metadata::metadata(cudf::io::datasource*) ()
   from /tmp/cudfgcc12/lib/libcudf.so
#3  0x00007ffff10ce94c in cudf::io::parquet::detail::aggregate_reader_metadata::metadatas_from_sources(cudf::host_span<std::unique_ptr<cudf::io::datasource, std::default_delete<cudf::io::datasource> > const, 18446744073709551615ul>) () from /tmp/cudfgcc12/lib/libcudf.so
#4  0x00007ffff10d1951 in cudf::io::parquet::detail::aggregate_reader_metadata::aggregate_reader_metadata(cudf::host_span<std::unique_ptr<cudf::io::datasource, std::default_delete<cudf::io::datasource> > const, 18446744073709551615ul>, bool) () from /tmp/cudfgcc12/lib/libcudf.so
#5  0x00007ffff109ecdc in cudf::io::parquet::detail::reader::impl::impl(unsigned long, unsigned long, std::vector<std::unique_ptr<cudf::io::datasource, std::default_delete<cudf::io::datasource> >, std::allocator<std::unique_ptr<cudf::io::datasource, std::default_delete<cudf::io::datasource> > > >&&, cudf::io::parquet_reader_options const&, rmm::cuda_stream_view, cuda::__4::mr::basic_resource_ref<(cuda::__4::mr::_AllocType)1, cuda::__4::mr::device_accessible>) ()
   from /tmp/cudfgcc12/lib/libcudf.so
#6  0x00007ffff109fdee in cudf::io::parquet::detail::reader::impl::impl(std::vector<std::unique_ptr<cudf::io::datasource, std::default_delete<cudf::io::datasource> >, std::allocator<std::unique_ptr<cudf::io::datasource, std::default_delete<cudf::io::datasource> > > >&&, cudf::io::parquet_reader_options const&, rmm::cuda_stream_view, cuda::__4::mr::basic_resource_ref<(cuda::__4::mr::_AllocType)1, cuda::__4::mr::device_accessible>) () from /tmp/cudfgcc12/lib/libcudf.so
#7  0x00007ffff109223a in cudf::io::parquet::detail::reader::reader(std::vector<std::unique_ptr<cudf::io::datasource, std::default_delete<cudf::io::datasource> >, std::allocator<std::unique_ptr<cudf::io::datasource, std::default_delete<cudf::io::datasource> > > >&&, cudf::io::parquet_reader_options const&, rmm::cuda_stream_view, cuda::__4::mr::basic_resource_ref<(cuda::__4::mr::_AllocType)1, cuda::__4::mr::device_accessible>) () from /tmp/cudfgcc12/lib/libcudf.so
#8  0x00007ffff0f4b2e7 in cudf::io::read_parquet(cudf::io::parquet_reader_options const&, rmm::cuda_stream_view, cuda::__4::mr::basic_resource_ref<(cuda::__4::mr::_AllocType)1, cuda::__4::mr::device_accessible>) () from /tmp/cudfgcc12/lib/libcudf.so
#9  0x000055555557d393 in main ()

Expected behavior
I expected the parquet read call to succeed with the libcudf build with both compiler versions.

Environment overview (please complete the following information)

• Environment location: Bare-metal
• Method of cuDF install: from source

Environment details
Please run and paste the output of the cudf/print_env.sh script here, to gather any other relevant environment details

Click here to see environment details

 **git***
 commit dc829b8372487615b74494a19c63d43cdbdb0d79 (HEAD -> branch-24.06, origin/branch-24.06)
 Author: Ray Douglass <[email protected]>
 Date:   Wed Jun 5 10:10:11 2024 -0400
 
 Update Changelog [skip ci]
 **git submodules***
 
 ***OS Information***
 DISTRIB_ID=Ubuntu
 DISTRIB_RELEASE=22.04
 DISTRIB_CODENAME=jammy
 DISTRIB_DESCRIPTION="Ubuntu 22.04.4 LTS"
 PRETTY_NAME="Ubuntu 22.04.4 LTS"
 NAME="Ubuntu"
 VERSION_ID="22.04"
 VERSION="22.04.4 LTS (Jammy Jellyfish)"
 VERSION_CODENAME=jammy
 ID=ubuntu
 ID_LIKE=debian
 HOME_URL="https://www.ubuntu.com/"
 SUPPORT_URL="https://help.ubuntu.com/"
 BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
 PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
 UBUNTU_CODENAME=jammy
 Linux cryos-dell 6.5.0-45-generic #45~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Jul 15 16:40:02 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
 
 ***GPU Information***
 Fri Jul 26 11:40:39 2024
 +-----------------------------------------------------------------------------------------+
 | NVIDIA-SMI 550.90.07              Driver Version: 550.90.07      CUDA Version: 12.4     |
 |-----------------------------------------+------------------------+----------------------+
 | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
 | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
 |                                         |                        |               MIG M. |
 |=========================================+========================+======================|
 |   0  NVIDIA RTX 3500 Ada Gene...    Off |   00000000:01:00.0 Off |                  Off |
 | N/A   58C    P3             21W /   90W |      15MiB /  12282MiB |      0%      Default |
 |                                         |                        |                  N/A |
 +-----------------------------------------+------------------------+----------------------+
 
 +-----------------------------------------------------------------------------------------+
 | Processes:                                                                              |
 |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
 |        ID   ID                                                               Usage      |
 |=========================================================================================|
 |    0   N/A  N/A      2165      G   /usr/lib/xorg/Xorg                              4MiB |
 +-----------------------------------------------------------------------------------------+
 
 ***CPU***
 Architecture:                       x86_64
 CPU op-mode(s):                     32-bit, 64-bit
 Address sizes:                      46 bits physical, 48 bits virtual
 Byte Order:                         Little Endian
 CPU(s):                             32
 On-line CPU(s) list:                0-31
 Vendor ID:                          GenuineIntel
 Model name:                         13th Gen Intel(R) Core(TM) i9-13950HX
 CPU family:                         6
 Model:                              183
 Thread(s) per core:                 2
 Core(s) per socket:                 24
 Socket(s):                          1
 Stepping:                           1
 CPU max MHz:                        5500.0000
 CPU min MHz:                        800.0000
 BogoMIPS:                           4838.40
 Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq tme rdpid movdiri movdir64b fsrm md_clear serialize pconfig arch_lbr ibt flush_l1d arch_capabilities
 Virtualization:                     VT-x
 L1d cache:                          896 KiB (24 instances)
 L1i cache:                          1.3 MiB (24 instances)
 L2 cache:                           32 MiB (12 instances)
 L3 cache:                           36 MiB (1 instance)
 NUMA node(s):                       1
 NUMA node0 CPU(s):                  0-31
 Vulnerability Gather data sampling: Not affected
 Vulnerability Itlb multihit:        Not affected
 Vulnerability L1tf:                 Not affected
 Vulnerability Mds:                  Not affected
 Vulnerability Meltdown:             Not affected
 Vulnerability Mmio stale data:      Not affected
 Vulnerability Retbleed:             Not affected
 Vulnerability Spec rstack overflow: Not affected
 Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
 Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
 Vulnerability Spectre v2:           Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
 Vulnerability Srbds:                Not affected
 Vulnerability Tsx async abort:      Not affected
 
 ***CMake***
 /home/marcus/cmake/bin/cmake
 cmake version 3.29.2
 
 CMake suite maintained and supported by Kitware (kitware.com/cmake).
 
 ***g++***
 /usr/bin/g++
 g++ (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
 Copyright (C) 2022 Free Software Foundation, Inc.
 This is free software; see the source for copying conditions.  There is NO
 warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 
 
 ***nvcc***
 /usr/local/cuda-12.3/bin/nvcc
 nvcc: NVIDIA (R) Cuda compiler driver
 Copyright (c) 2005-2023 NVIDIA Corporation
 Built on Wed_Nov_22_10:17:15_PST_2023
 Cuda compilation tools, release 12.3, V12.3.107
 Build cuda_12.3.r12.3/compiler.33567101_0
 
 ***Python***
 
 ***Environment Variables***
 PATH                            : /home/marcus/cmake/bin:/usr/local/cuda-12.3/bin:/home/marcus/mambaforge/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
 LD_LIBRARY_PATH                 :
 NUMBAPRO_NVVM                   :
 NUMBAPRO_LIBDEVICE              :
 CONDA_PREFIX                    :
 PYTHON_PATH                     :

Additional context
Add any other context about the problem here.

[cryos]

Just to add that this can also by reproduced by the C++ unit tests. For GCC 11.4 they all pass for me locally, whereas for GCC 12.3.0 I get the following failures (running from the build tree):

96% tests passed, 4 tests failed out of 97

Total Test time (real) =  92.80 sec

The following tests FAILED:
         31 - PARQUET_TEST (Failed)
         46 - UTILITIES_TEST (Failed)
         63 - LARGE_STRINGS_TEST (Failed)
         89 - STREAM_PARQUETIO_TEST (Failed)

ctest -V -R PARQUET_TEST
<snip>
31: '/home/marcus/src/cudf/cpp/gcc12/gtests/PARQUET_TEST'
31: [==========] Running 339 tests from 94 test suites.
31: [----------] Global test environment set-up.
31: [----------] 12 tests from ParquetChunkedReaderTest
31: [ RUN      ] ParquetChunkedReaderTest.TestChunkedReadNoData
31: CMake Error at /home/marcus/src/cudf/cpp/gcc12/rapids-cmake/run_gpu_test.cmake:34 (execute_process):
31:   execute_process failed command indexes:
31: 
31:     1: "Abnormal exit with child return code: Segmentation fault"
31: 
31: 
31: 
1/1 Test #31: PARQUET_TEST .....................***Failed    2.52 sec

[robertmaynard]

I have been able to replicate these results when building with gcc-12

[davidwendt]

All of the failures have the same stack trace. Example:

$ gdb --args gtests/UTILITIES_TEST --gtest_filter=PinnedMemoryTest.MemoryResourceGetAndSet

Output:

Thread 1 "UTILITIES_TEST" received signal SIGSEGV, Segmentation fault.
0x00007fc3b9636f4c in std::_Function_handler<void (unsigned int, cudf::io::parquet::detail::CompactProtocolReader*), cudf::io::parquet::detail::parquet_field_struct_list<cudf::io::parquet::detail::SchemaElement>::parquet_field_struct_list(int, std::vector<cudf::io::parquet::detail::SchemaElement, std::allocator<cudf::io::parquet::detail::SchemaElement> >&)::{lambda(unsigned int, cudf::io::parquet::detail::CompactProtocolReader*)#1}>::_M_invoke(std::_Any_data const&, unsigned int&&, cudf::io::parquet::detail::CompactProtocolReader*&&) () from /cudf/cpp/build/libcudf.so
(gdb) bt
#0  0x00007fc3b9636f4c in std::_Function_handler<void (unsigned int, cudf::io::parquet::detail::CompactProtocolReader*), cudf::io::parquet::detail::parquet_field_struct_list<cudf::io::parquet::detail::SchemaElement>::parquet_field_struct_list(int, std::vector<cudf::io::parquet::detail::SchemaElement, std::allocator<cudf::io::parquet::detail::SchemaElement> >&)::{lambda(unsigned int, cudf::io::parquet::detail::CompactProtocolReader*)#1}>::_M_invoke(std::_Any_data const&, unsigned int&&, cudf::io::parquet::detail::CompactProtocolReader*&&) () from /cudf/cpp/build/libcudf.so
#1  0x00007fc3b9633b5d in cudf::io::parquet::detail::CompactProtocolReader::read(cudf::io::parquet::detail::FileMetaData*) ()
   from /cudf/cpp/build/libcudf.so
#2  0x00007fc3b96a4136 in cudf::io::parquet::detail::metadata::metadata(cudf::io::datasource*) () from /cudf/cpp/build/libcudf.so
#3  0x00007fc3b96a7f0c in cudf::io::parquet::detail::aggregate_reader_metadata::metadatas_from_sources(cudf::host_span<std::unique_ptr<cudf::io::datasource, std::default_delete<cudf::io::datasource> > const, 18446744073709551615ul>) ()
   from /cudf/cpp/build/libcudf.so
#4  0x00007fc3b96aafc7 in cudf::io::parquet::detail::aggregate_reader_metadata::aggregate_reader_metadata(cudf::host_span<std::unique_ptr<cudf::io::datasource, std::default_delete<cudf::io::datasource> > const, 18446744073709551615ul>, bool) ()
   from /cudf/cpp/build/libcudf.so
#5  0x00007fc3b966a85c in cudf::io::parquet::detail::reader::impl::impl(unsigned long, unsigned long, std::vector<std::unique_ptr<cudf::io::datasource, std::default_delete<cudf::io::datasource> >, std::allocator<std::unique_ptr<cudf::io::datasource, std::default_delete<cudf::io::datasource> > > >&&, cudf::io::parquet_reader_options const&, rmm::cuda_stream_view, cuda::mr::__4::basic_resource_ref<(cuda::mr::__4::_AllocType)1, cuda::mr::__4::device_accessible>) () from /cudf/cpp/build/libcudf.so
#6  0x00007fc3b966b7ce in cudf::io::parquet::detail::reader::impl::impl(std::vector<std::unique_ptr<cudf::io::datasource, std::default_delete<cudf::io::datasource> >, std::allocator<std::unique_ptr<cudf::io::datasource, std::default_delete<cudf::io::datasource> > > >&&, cudf::io::parquet_reader_options const&, rmm::cuda_stream_view, cuda::mr::__4::basic_resource_ref<(cuda::mr::__4::_AllocType)1, cuda::mr::__4::device_accessible>) () from /cudf/cpp/build/libcudf.so
#7  0x00007fc3b9662bea in cudf::io::parquet::detail::reader::reader(std::vector<std::unique_ptr<cudf::io::datasource, std::default_delete<cudf::io::datasource> >, std::allocator<std::unique_ptr<cudf::io::datasource, std::default_delete<cudf::io::datasource> > > >&&, cudf::io::parquet_reader_options const&, rmm::cuda_stream_view, cuda::mr::__4::basic_resource_ref<(cuda::mr::__4::_AllocType)1, cuda::mr::__4::device_accessible>) () from /cudf/cpp/build/libcudf.so
#8  0x00007fc3b94e0e17 in cudf::io::read_parquet(cudf::io::parquet_reader_options const&, rmm::cuda_stream_view, cuda::mr::__4::basic_resource_ref<(cuda::mr::__4::_AllocType)1, cuda::mr::__4::device_accessible>) () from /cudf/cpp/build/libcudf.so
#9  0x000055c287ba602f in PinnedMemoryTest_MemoryResourceGetAndSet_Test::TestBody() ()
#10 0x00007fc3ddd25521 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) () from /cudf/cpp/build/libcudftestutil.so
#11 0x00007fc3ddd138c6 in testing::Test::Run() () from /cudf/cpp/build/libcudftestutil.so
#12 0x00007fc3ddd13a85 in testing::TestInfo::Run() () from /cudf/cpp/build/libcudftestutil.so
#13 0x00007fc3ddd13ba5 in testing::TestSuite::Run() () from /cudf/cpp/build/libcudftestutil.so
#14 0x00007fc3ddd1ac74 in testing::internal::UnitTestImpl::RunAllTests() () from /cudf/cpp/build/libcudftestutil.so
#15 0x00007fc3ddd13d6f in testing::UnitTest::Run() () from /cudf/cpp/build/libcudftestutil.so
#16 0x000055c28755becc in main ()