[WIP] ARROW-15678: [C++][CI] a crossbow job with MinRelSize enabled #12422

jonkeane · 2022-02-14T18:16:03Z

Adds a crossbow build that builds libarrow with CMAKE_BUILT_TYPE=MinSizeRel and runs the R tests on that

github-actions · 2022-02-14T18:16:26Z

https://issues.apache.org/jira/browse/ARROW-15678

github-actions · 2022-02-14T18:16:27Z

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

jonkeane · 2022-02-14T18:16:59Z

r/tests/testthat.R

+  arrow::arrow_info()
+


This might be helpful in debugging. We can remove it before we merge this, though it should only impact our CI builds and maybe it's good/fine to keep it around?

jonkeane · 2022-02-14T18:17:51Z

dev/tasks/tasks.yml

@@ -1252,6 +1252,12 @@ tasks:
    ci: github
    template: r/github.linux.offline.build.yml

+  test-r-minsizerel:
+    ci: github
+    template: r/github.macos-linux.local.yml


This builds on macos + linux, which isn't strictly necessary for triggering what we're trying to trigger, but might be worth checking both linux + macos anyway

jonkeane · 2022-02-14T18:21:28Z

@github-actions crossbow submit test-r-minsizerel

github-actions · 2022-02-14T18:22:35Z

Revision: 1e0ecc4

Submitted crossbow builds: ursacomputing/crossbow @ actions-1631

Task	Status
test-r-minsizerel

jonkeane · 2022-02-14T18:56:28Z

@github-actions crossbow submit test-r-minsizerel with ARROW_R_DEV globally so that it is set when the tests run

github-actions · 2022-02-14T18:57:26Z

Unable to match any tasks for `with`
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/1842912357```

jonkeane · 2022-02-14T18:57:54Z

@github-actions crossbow submit test-r-minsizerel

with ARROW_R_DEV globally so that it is set when the tests run

github-actions · 2022-02-14T18:59:11Z

Revision: 3d90069

Submitted crossbow builds: ursacomputing/crossbow @ actions-1632

Task	Status
test-r-minsizerel

jonkeane · 2022-02-14T19:44:31Z

@github-actions crossbow submit test-r-minsizerel

github-actions · 2022-02-14T19:45:47Z

Revision: 2391edf

Submitted crossbow builds: ursacomputing/crossbow @ actions-1633

Task	Status
test-r-minsizerel

pitrou · 2022-02-14T20:22:48Z

@jonkeane There is no traceback displayed anymore?

jonkeane · 2022-02-14T20:24:43Z

It's a bit hidden by the way the tests are run, but the Dump test logs still has: https://github.com/ursacomputing/crossbow/runs/5189845009?check_suite_focus=true#step:10:8126

Which also now includes arrow_info() too: https://github.com/ursacomputing/crossbow/runs/5189845009?check_suite_focus=true#step:10:98

pitrou · 2022-02-14T20:27:55Z

Ok, I see. If this is the same crash as you got locally, then I'm inclined to think it's a compiler bug where clang emits a BMI2 instruction even though only -msse4.2 is given on the command line.

jonkeane · 2022-02-14T21:35:23Z

Yup, that's the same place I saw locally.

If it helps, here's the stack trace + disassembly from when when I triggered it locally as well:

2491 Thread_829819
+ 2491 thread_start  (in libsystem_pthread.dylib) + 15  [0x7ff801c3e00f]
+   2491 _pthread_start  (in libsystem_pthread.dylib) + 125  [0x7ff801c424f4]
+     2491 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_3> >(void*)  (in arrow.so) + 380  [0x109203749]
+       2491 arrow::internal::FnOnce<void ()>::operator()() &&  (in arrow.so) + 26  [0x109201f30]
+         2491 arrow::internal::FnOnce<void ()>::FnImpl<std::__1::__bind<arrow::detail::ContinueFuture, arrow::Future<std::__1::shared_ptr<arrow::ChunkedArray> >&, parquet::arrow::(anonymous namespace)::FileReaderImpl::DecodeRowGroups(std::__1::shared_ptr<parquet::arrow::(anonymous namespace)::FileReaderImpl>, std::__1::vector<int, std::__1::allocator<int> > const&, std::__1::vector<int, std::__1::allocator<int> > const&, arrow::internal::Executor*)::$_4&, unsigned long&, std::__1::shared_ptr<parquet::arrow::ColumnReaderImpl> > >::invoke()  (in arrow.so) + 98  [0x108f125c2]
+           2491 parquet::arrow::(anonymous namespace)::FileReaderImpl::DecodeRowGroups(std::__1::shared_ptr<parquet::arrow::(anonymous namespace)::FileReaderImpl>, std::__1::vector<int, std::__1::allocator<int> > const&, std::__1::vector<int, std::__1::allocator<int> > const&, arrow::internal::Executor*)::$_4::operator()(unsigned long, std::__1::shared_ptr<parquet::arrow::ColumnReaderImpl>) const  (in arrow.so) + 47  [0x108f11ed5]
+             2491 parquet::arrow::(anonymous namespace)::FileReaderImpl::ReadColumn(int, std::__1::vector<int, std::__1::allocator<int> > const&, parquet::arrow::ColumnReader*, std::__1::shared_ptr<arrow::ChunkedArray>*)  (in arrow.so) + 273  [0x108f0c037]
+               2491 parquet::arrow::ColumnReaderImpl::NextBatch(long long, std::__1::shared_ptr<arrow::ChunkedArray>*)  (in arrow.so) + 39  [0x108f0733b]
+                 2491 parquet::arrow::(anonymous namespace)::LeafReader::LoadBatch(long long)  (in arrow.so) + 137  [0x108f0794b]
+                   2491 parquet::internal::(anonymous namespace)::TypedRecordReader<parquet::PhysicalType<(parquet::Type::type)1> >::ReadRecords(long long)  (in arrow.so) + 442  [0x108f4f53e]
+                     2491 parquet::internal::(anonymous namespace)::TypedRecordReader<parquet::PhysicalType<(parquet::Type::type)1> >::ReadRecordData(long long)  (in arrow.so) + 471  [0x108f50503]
+                       2491 void parquet::internal::standard::DefLevelsToBitmapSimd<false>(short const*, long long, parquet::internal::LevelInfo, parquet::internal::ValidityBitmapInputOutput*)  (in arrow.so) + 250  [0x108fc2a5a]
+                         2491 long long parquet::internal::standard::DefLevelsBatchToBitmap<false>(short const*, long long, long long, parquet::internal::LevelInfo, arrow::internal::FirstTimeBitmapWriter*)  (in arrow.so) + 63  [0x108fc34da]
+                           2491 ???  (in <unknown binary>)  [0x600001354518]
+                             2491 _sigtramp  (in libsystem_platform.dylib) + 29  [0x7ff801c57e2d]
+                               2491 sigactionSegv  (in libR.dylib) + 649  [0x1042598c9]  main.c:625
+                                 2491 Rstd_ReadConsole  (in libR.dylib) + 2042  [0x10435160a]  sys-std.c:1044
+                                   2491 R_SelectEx  (in libR.dylib) + 308  [0x104350854]  sys-std.c:178
+                                     2491 __select  (in libsystem_kernel.dylib) + 10  [0x7ff801c0de4a]

Dump of assembler code from 0x13ac6db00 to 0x13ac6db99ff:
 ...
--Type <RET> for more, q to quit, c to continue without paging--
   0x000000013ac6db82:	mov    $0x8,%ecx
   0x000000013ac6db87:	sub    %rax,%rcx
   0x000000013ac6db8a:	lea    0xf1520b(%rip),%rdi        # 0x13bb82d9c
   0x000000013ac6db91:	movzbl (%rcx,%rdi,1),%edi
   0x000000013ac6db95:	mov    %esi,%ebx
   0x000000013ac6db97:	and    %edi,%ebx
=> 0x000000013ac6db99:	shlx   %rax,%rbx,%rax
   0x000000013ac6db9e:	or     0x18(%r15),%al
   0x000000013ac6dba2:	mov    %al,0x18(%r15)
   0x000000013ac6dba6:	cmp    %rdx,%rcx
   0x000000013ac6dba9:	jg     0x13ac6dbf5
   0x000000013ac6dbab:	mov    %al,(%r14)
   0x000000013ac6dbae:	inc    %r14
   0x000000013ac6dbb1:	shrx   %rcx,%rsi,%rax
   0x000000013ac6dbb6:	mov    %rax,-0x20(%rbp)
   0x000000013ac6dbba:	sub    %rcx,%rdx
   0x000000013ac6dbbd:	mov    %rdx,%rbx
   0x000000013ac6dbc0:	sar    $0x3,%rbx
   0x000000013ac6dbc4:	and    $0x7,%edx
   0x000000013ac6dbc7:	cmp    $0x1,%rdx
   0x000000013ac6dbcb:	sbb    $0xffffffffffffffff,%rbx
   0x000000013ac6dbcf:	lea    -0x20(%rbp),%rsi
   0x000000013ac6dbd3:	mov    %r14,%rdi
...

pitrou · 2022-02-14T21:36:16Z

Yes, it's based on that info that I estimate that it's a compiler bug.

pitrou · 2022-02-15T09:53:09Z

@github-actions crossbow submit test-r-minsizerel

github-actions · 2022-02-15T09:54:56Z

Revision: 1969bc7

Submitted crossbow builds: ursacomputing/crossbow @ actions-1634

Task	Status
test-r-minsizerel

pitrou · 2022-02-15T10:44:27Z

@github-actions crossbow submit test-r-minsizerel

github-actions · 2022-02-15T10:45:34Z

Revision: 7b90730

Submitted crossbow builds: ursacomputing/crossbow @ actions-1635

Task	Status
test-r-minsizerel

jonkeane · 2022-02-15T13:53:45Z

@github-actions crossbow submit test-r-minsizerel

github-actions · 2022-02-15T13:55:02Z

Revision: d1c6351

Submitted crossbow builds: ursacomputing/crossbow @ actions-1636

Task	Status
test-r-minsizerel

jonkeane · 2022-02-15T14:00:06Z

@github-actions crossbow submit test-r-minsizerel

github-actions · 2022-02-15T14:01:18Z

Revision: 5e81c08

Submitted crossbow builds: ursacomputing/crossbow @ actions-1637

Task	Status
test-r-minsizerel

jonkeane · 2022-02-24T23:44:44Z

@github-actions crossbow submit test-r-minsizerel

github-actions · 2022-02-24T23:46:05Z

Revision: 20ca138

Submitted crossbow builds: ursacomputing/crossbow @ actions-1679

Task	Status
test-r-minsizerel

jonkeane · 2022-02-25T13:59:51Z

@github-actions crossbow submit test-r-minsizerel

github-actions · 2022-02-25T14:01:11Z

Revision: ffe6725

Submitted crossbow builds: ursacomputing/crossbow @ actions-1681

Task	Status
test-r-minsizerel

jonkeane · 2022-02-25T21:18:08Z

@github-actions crossbow submit test-r-minsizerel

github-actions · 2022-02-25T21:19:18Z

Revision: 5d0f404

Submitted crossbow builds: ursacomputing/crossbow @ actions-1684

Task	Status
test-r-minsizerel

jonkeane · 2022-02-25T23:08:47Z

@github-actions crossbow submit test-r-minsizerel

github-actions · 2022-02-25T23:09:58Z

Revision: bfb7b85

Submitted crossbow builds: ursacomputing/crossbow @ actions-1685

Task	Status
test-r-minsizerel

jonkeane · 2022-02-26T15:30:58Z

@github-actions crossbow submit test-r-minsizerel

github-actions · 2022-02-26T15:32:10Z

Revision: 7900fa7

Submitted crossbow builds: ursacomputing/crossbow @ actions-1686

Task	Status
test-r-minsizerel

jonkeane · 2022-03-02T17:43:00Z

I tried also running GCC, it seems to run the test just fine but I'm still getting segfaults when the tests finish(?) I also tried isolating the tests to just the one file that has the test that segfaults with clang and that has been freezing + timing out.

I'm going to close this and move on to other tasks — if anyone wants to take it up, feel free.

github-actions bot added Component: C++ Component: R labels Feb 14, 2022

jonkeane commented Feb 14, 2022

View reviewed changes

apache deleted a comment from github-actions bot Feb 14, 2022

pitrou changed the title ~~ARROW-15678: [C++][CI] a crossbow job with MinRelSize enabled~~ [WIP] ARROW-15678: [C++][CI] a crossbow job with MinRelSize enabled Feb 15, 2022

pitrou force-pushed the ARROW-15678 branch from 1969bc7 to 7b90730 Compare February 15, 2022 10:44

jonkeane mentioned this pull request Feb 24, 2022

MINOR: [C++] Allow MinSizeRel builds #12512

Closed

jonkeane and others added 10 commits February 24, 2022 17:43

This should fail cause MinRelSize is not yet fixed

8e62c82

add MinSizeRel to setupCxxFlags.cmake

9f4c751

oops

a04812d

add ARROW_R_DEV to the global env

2939738

oops, need to print

5a77aa9

add extra flag

6c56a25

Experiment with other clang version

1910cdb

Also update where R looks for compilers

d483dcb

oops, need to create the directory first

e00bc0a

try gcc-11

20ca138

jonkeane force-pushed the ARROW-15678 branch from 0663bb9 to 20ca138 Compare February 24, 2022 23:44

bundle deps to avoid gcc-clang mismatch

ffe6725

a different way of testing

5d0f404

skip some tests that complain about somethign unrelated

bfb7b85

try only the parquet tests

7900fa7

jonkeane closed this Mar 2, 2022

wjones127 mentioned this pull request Mar 22, 2022

[C++] Test failure in arrow-compute-aggregate-test #12681

Open

asfimport mentioned this pull request Oct 11, 2022

[C++][CI] a crossbow job with MinRelSize enabled #31132

Closed

[WIP] ARROW-15678: [C++][CI] a crossbow job with MinRelSize enabled #12422

[WIP] ARROW-15678: [C++][CI] a crossbow job with MinRelSize enabled #12422

Conversation

jonkeane commented Feb 14, 2022

github-actions bot commented Feb 14, 2022

github-actions bot commented Feb 14, 2022

jonkeane Feb 14, 2022

Choose a reason for hiding this comment

jonkeane Feb 14, 2022

Choose a reason for hiding this comment

jonkeane commented Feb 14, 2022

github-actions bot commented Feb 14, 2022

jonkeane commented Feb 14, 2022

github-actions bot commented Feb 14, 2022

jonkeane commented Feb 14, 2022

github-actions bot commented Feb 14, 2022

jonkeane commented Feb 14, 2022

github-actions bot commented Feb 14, 2022

pitrou commented Feb 14, 2022

jonkeane commented Feb 14, 2022 • edited Loading

pitrou commented Feb 14, 2022

jonkeane commented Feb 14, 2022

pitrou commented Feb 14, 2022

pitrou commented Feb 15, 2022

github-actions bot commented Feb 15, 2022

pitrou commented Feb 15, 2022

github-actions bot commented Feb 15, 2022

jonkeane commented Feb 15, 2022

github-actions bot commented Feb 15, 2022

jonkeane commented Feb 15, 2022

github-actions bot commented Feb 15, 2022

jonkeane commented Feb 24, 2022

github-actions bot commented Feb 24, 2022

jonkeane commented Feb 25, 2022

github-actions bot commented Feb 25, 2022

jonkeane commented Feb 25, 2022

github-actions bot commented Feb 25, 2022

jonkeane commented Feb 25, 2022

github-actions bot commented Feb 25, 2022

jonkeane commented Feb 26, 2022

github-actions bot commented Feb 26, 2022

jonkeane commented Mar 2, 2022

jonkeane commented Feb 14, 2022 •

edited

Loading