Update to rust 1.26

[stuhood]

Update to rust 1.26, explicitly use Itertools::flatten to avoid rust-lang/rust#48919, and apply necessary rustfmt changes.

I experimented with a few different OSX versions, but there is a segfault for all versions of OSX (*that are available on travis) below xcode8 / OSX 10.11. So, landing this change would raise pants' minimum OSX requirement to 10.11.

[stuhood]

Hmm... it looks like it is an OSX age issue! Will try with 10.11 next.

[wisechengyi]

Why are some other shards removed?

[stuhood]

We're iterating on getting this shard working, because it's currently failing.

[stuhood]

I'd love to file a bug with the rustlang folks about the xcode < 8 issue, but it's hard to know where to start debugging there. Their platform support page claims support down to OSX 10.7.

[illicitonion]

After a lot of poking, I finally managed to get a stacktrace out - https://travis-ci.org/pantsbuild/pants/builds/379165187

It turns out Gitignore keeps a thread-local cache, as an optimisation, which can't be disabled. It looks like the way thread_local stores thread IDs isn't fork-safe. I don't know why this isn't surfacing on more modern versions of OSX, or indeed on older rusts.

I modified Gitignore to bypass the thread_local cache, and it turns out the standard regex crate, which it uses, also uses a thread_local cache, so also triggers the same issue: https://travis-ci.org/pantsbuild/pants/builds/379216270

This was uncomfortably difficult to diagnose. I will put together some changes to make it easier in the future. I don't know what to do about fixing it, though.

./build-support/bin/release.sh: line 14:  7634 Segmentation fault: 11  (core dumped) ${ROOT}/pants "$@"
(lldb) target create --core "/cores/core.7634"
warning: (x86_64) /cores/core.7634 load command 370 LC_SEGMENT_64 has a fileoff + filesize (0x2c170000) that extends beyond the end of the file (0x2c16f000), the segment will be truncated to match
warning: (x86_64) /cores/core.7634 load command 371 LC_SEGMENT_64 has a fileoff (0x2c170000) that extends beyond the end of the file (0x2c16f000), ignoring this section
Core file '/cores/core.7634' (x86_64) was loaded.
(lldb) bt
* thread #1: tid = 0x0000, 0x00007fff81cd88ea libsystem_kernel.dylib`__kill + 10, stop reason = signal SIGSTOP
  * frame #0: 0x00007fff81cd88ea libsystem_kernel.dylib`__kill + 10
    frame #1: 0x00007fff86b2f52a libsystem_platform.dylib`_sigtramp + 26
    frame #2: 0x0000000112d7109c native_engine.so`thread_local::thread_id::get::he68caed8d4223aa8 + 124 at ptr.rs:238
    frame #3: 0x0000000112ceb514 native_engine.so`fs::is_ignored::h48b069830adbc2e2 + 404 at lib.rs:208
    frame #4: 0x0000000112c4ed2f native_engine.so`_$LT$core..iter..Map$LT$I$C$$u20$F$GT$$u20$as$u20$core..iter..iterator..Iterator$GT$::next::h35cb349c05a754e1 + 1039 at nodes.rs:63
    frame #5: 0x0000000112c5cf82 native_engine.so`_$LT$futures..future..map..Map$LT$A$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::h9bf1e53c28b2932e + 1554 at vec.rs:1780
    frame #6: 0x0000000112c551ec native_engine.so`_$LT$futures..future..map..Map$LT$A$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::h3f54f8226910c66a + 60 at mod.rs:113
    frame #7: 0x0000000112c57d5f native_engine.so`_$LT$futures..future..map..Map$LT$A$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::h939bb70dd168f93d + 175 at mod.rs:113
    frame #8: 0x0000000112c638ce native_engine.so`_$LT$futures..future..map_err..MapErr$LT$A$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::h40c1a08989a335e8 + 62 at mod.rs:113
    frame #9: 0x0000000112c54ae2 native_engine.so`_$LT$futures..future..map..Map$LT$A$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::h2fa8f442567e7313 + 34 at mod.rs:113
    frame #10: 0x0000000112c61252 native_engine.so`_$LT$futures..future..lazy..Lazy$LT$F$C$$u20$R$GT$$u20$as$u20$futures..future..Future$GT$::poll::h61cc50e6be1f6cc9 + 7106 at mod.rs:113
    frame #11: 0x0000000112c51ca4 native_engine.so`_$LT$futures..future..shared..Shared$LT$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::hdde964ed0773800d + 1796 at mod.rs:113
    frame #12: 0x0000000112c692bb native_engine.so`_$LT$futures..future..then..Then$LT$A$C$$u20$B$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::hba4daa600d369034 + 59 at chain.rs:26
    frame #13: 0x0000000112c0b540 native_engine.so`_$LT$futures..future..and_then..AndThen$LT$A$C$$u20$B$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::h4abccc0ff43bd520 + 160 at mod.rs:113
    frame #14: 0x0000000112c56d8b native_engine.so`_$LT$futures..future..map..Map$LT$A$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::h74959abde696277e + 43 at mod.rs:113
    frame #15: 0x0000000112c1207c native_engine.so`_$LT$futures..future..and_then..AndThen$LT$A$C$$u20$B$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::h93afaa7f9e473e56 + 220 at mod.rs:113
    frame #16: 0x0000000112c533cc native_engine.so`_$LT$futures..future..join_all..JoinAll$LT$I$GT$$u20$as$u20$futures..future..Future$GT$::poll::hf854dab0fdaf8369 + 140 at mod.rs:113
    frame #17: 0x0000000112c5ed6b native_engine.so`_$LT$futures..future..map..Map$LT$A$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::hcab5e9f99833ae61 + 43 at mod.rs:113
    frame #18: 0x0000000112c62bc2 native_engine.so`_$LT$futures..future..loop_fn..LoopFn$LT$A$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::h9e9ffff8f83d1d7c + 866 at mod.rs:113
    frame #19: 0x0000000112c69dfb native_engine.so`_$LT$futures..future..then..Then$LT$A$C$$u20$B$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::hec44aa2209bb5af9 + 443 at mod.rs:113
    frame #20: 0x0000000112c5eba2 native_engine.so`_$LT$futures..future..map..Map$LT$A$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::hbdf85069e9c3f730 + 34 at mod.rs:113
    frame #21: 0x0000000112c61252 native_engine.so`_$LT$futures..future..lazy..Lazy$LT$F$C$$u20$R$GT$$u20$as$u20$futures..future..Future$GT$::poll::h61cc50e6be1f6cc9 + 7106 at mod.rs:113
    frame #22: 0x0000000112c51ca4 native_engine.so`_$LT$futures..future..shared..Shared$LT$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::hdde964ed0773800d + 1796 at mod.rs:113
    frame #23: 0x0000000112c66fa8 native_engine.so`_$LT$futures..future..then..Then$LT$A$C$$u20$B$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::h715d89f7b29f9f28 + 56 at chain.rs:26
    frame #24: 0x0000000112c1207c native_engine.so`_$LT$futures..future..and_then..AndThen$LT$A$C$$u20$B$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::h93afaa7f9e473e56 + 220 at mod.rs:113
    frame #25: 0x0000000112c69cf7 native_engine.so`_$LT$futures..future..then..Then$LT$A$C$$u20$B$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::hec44aa2209bb5af9 + 183 at mod.rs:113
    frame #26: 0x0000000112c5eba2 native_engine.so`_$LT$futures..future..map..Map$LT$A$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::hbdf85069e9c3f730 + 34 at mod.rs:113
    frame #27: 0x0000000112c61252 native_engine.so`_$LT$futures..future..lazy..Lazy$LT$F$C$$u20$R$GT$$u20$as$u20$futures..future..Future$GT$::poll::h61cc50e6be1f6cc9 + 7106 at mod.rs:113
    frame #28: 0x0000000112c51ca4 native_engine.so`_$LT$futures..future..shared..Shared$LT$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::hdde964ed0773800d + 1796 at mod.rs:113
    frame #29: 0x0000000112c66fa8 native_engine.so`_$LT$futures..future..then..Then$LT$A$C$$u20$B$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::h715d89f7b29f9f28 + 56 at chain.rs:26
    frame #30: 0x0000000112c1207c native_engine.so`_$LT$futures..future..and_then..AndThen$LT$A$C$$u20$B$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::h93afaa7f9e473e56 + 220 at mod.rs:113
    frame #31: 0x0000000112c5e9e2 native_engine.so`_$LT$futures..future..map..Map$LT$A$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::hb58299e9aa548b9c + 34 at mod.rs:113
    frame #32: 0x0000000112c61252 native_engine.so`_$LT$futures..future..lazy..Lazy$LT$F$C$$u20$R$GT$$u20$as$u20$futures..future..Future$GT$::poll::h61cc50e6be1f6cc9 + 7106 at mod.rs:113
    frame #33: 0x0000000112c51ca4 native_engine.so`_$LT$futures..future..shared..Shared$LT$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::hdde964ed0773800d + 1796 at mod.rs:113
    frame #34: 0x0000000112c67808 native_engine.so`_$LT$futures..future..then..Then$LT$A$C$$u20$B$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::h7d7049b46287f99b + 56 at chain.rs:26
    frame #35: 0x0000000112c08f63 native_engine.so`_$LT$futures..future..or_else..OrElse$LT$A$C$$u20$B$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::hdc6ca86789cd0741 + 291 at mod.rs:113
    frame #36: 0x0000000112c87c4d native_engine.so`scheduler_execute + 2861 at mod.rs:113
    frame #37: 0x0000000112c9d81e native_engine.so`_cffi_f_scheduler_execute(self=<unavailable>, args=<unavailable>) + 414 at native_engine.c:2388 [opt]
    frame #38: 0x000000010f7acfed Python`PyEval_EvalFrameEx + 27113
    frame #39: 0x000000010f7a640e Python`PyEval_EvalCodeEx + 1617
    frame #40: 0x000000010f7b0c9e Python`fast_function + 117
    frame #41: 0x000000010f7acf09 Python`PyEval_EvalFrameEx + 26885
    frame #42: 0x000000010f7a640e Python`PyEval_EvalCodeEx + 1617
    frame #43: 0x000000010f7b0c9e Python`fast_function + 117
    frame #44: 0x000000010f7acf09 Python`PyEval_EvalFrameEx + 26885
    frame #45: 0x000000010f7a640e Python`PyEval_EvalCodeEx + 1617
    frame #46: 0x000000010f7b0c9e Python`fast_function + 117
    frame #47: 0x000000010f7acf09 Python`PyEval_EvalFrameEx + 26885
    frame #48: 0x000000010f7a640e Python`PyEval_EvalCodeEx + 1617
    frame #49: 0x000000010f7b0c9e Python`fast_function + 117
    frame #50: 0x000000010f7acf09 Python`PyEval_EvalFrameEx + 26885
    frame #51: 0x000000010f7a640e Python`PyEval_EvalCodeEx + 1617
    frame #52: 0x000000010f7b0c9e Python`fast_function + 117
    frame #53: 0x000000010f7acf09 Python`PyEval_EvalFrameEx + 26885
    frame #54: 0x000000010f7430f8 Python`gen_send_ex + 191
    frame #55: 0x000000010f7ac83b Python`PyEval_EvalFrameEx + 25143
    frame #56: 0x000000010f7430f8 Python`gen_send_ex + 191
    frame #57: 0x000000010f7ac83b Python`PyEval_EvalFrameEx + 25143
    frame #58: 0x000000010f7a640e Python`PyEval_EvalCodeEx + 1617
    frame #59: 0x000000010f7b0c9e Python`fast_function + 117
    frame #60: 0x000000010f7acf09 Python`PyEval_EvalFrameEx + 26885
    frame #61: 0x000000010f7a640e Python`PyEval_EvalCodeEx + 1617
    frame #62: 0x000000010f7b0c9e Python`fast_function + 117
    frame #63: 0x000000010f7acf09 Python`PyEval_EvalFrameEx + 26885
    frame #64: 0x000000010f7a640e Python`PyEval_EvalCodeEx + 1617
    frame #65: 0x000000010f7b0c9e Python`fast_function + 117
    frame #66: 0x000000010f7acf09 Python`PyEval_EvalFrameEx + 26885
    frame #67: 0x000000010f7a640e Python`PyEval_EvalCodeEx + 1617
    frame #68: 0x000000010f7b0c9e Python`fast_function + 117
    frame #69: 0x000000010f7acf09 Python`PyEval_EvalFrameEx + 26885
    frame #70: 0x000000010f7a640e Python`PyEval_EvalCodeEx + 1617
    frame #71: 0x000000010f7b0c9e Python`fast_function + 117
    frame #72: 0x000000010f7acf09 Python`PyEval_EvalFrameEx + 26885
    frame #73: 0x000000010f7a640e Python`PyEval_EvalCodeEx + 1617
    frame #74: 0x000000010f7b0c9e Python`fast_function + 117
    frame #75: 0x000000010f7acf09 Python`PyEval_EvalFrameEx + 26885
    frame #76: 0x000000010f7a640e Python`PyEval_EvalCodeEx + 1617
    frame #77: 0x000000010f7b0c9e Python`fast_function + 117
    frame #78: 0x000000010f7acf09 Python`PyEval_EvalFrameEx + 26885
    frame #79: 0x000000010f7a640e Python`PyEval_EvalCodeEx + 1617
    frame #80: 0x000000010f7b0c9e Python`fast_function + 117
    frame #81: 0x000000010f7acf09 Python`PyEval_EvalFrameEx + 26885
    frame #82: 0x000000010f7a640e Python`PyEval_EvalCodeEx + 1617
    frame #83: 0x000000010f7b0c9e Python`fast_function + 117
    frame #84: 0x000000010f7acf09 Python`PyEval_EvalFrameEx + 26885
    frame #85: 0x000000010f7a640e Python`PyEval_EvalCodeEx + 1617
    frame #86: 0x000000010f7b0c9e Python`fast_function + 117
    frame #87: 0x000000010f7acf09 Python`PyEval_EvalFrameEx + 26885
    frame #88: 0x000000010f7a640e Python`PyEval_EvalCodeEx + 1617
    frame #89: 0x000000010f7b0c9e Python`fast_function + 117
    frame #90: 0x000000010f7acf09 Python`PyEval_EvalFrameEx + 26885
    frame #91: 0x000000010f7a640e Python`PyEval_EvalCodeEx + 1617
    frame #92: 0x000000010f7a5db7 Python`PyEval_EvalCode + 48
    frame #93: 0x000000010f7c983c Python`run_mod + 53
    frame #94: 0x000000010f7c98df Python`PyRun_FileExFlags + 133
    frame #95: 0x000000010f7c9430 Python`PyRun_SimpleFileExFlags + 702
    frame #96: 0x000000010f7dad9e Python`Py_Main + 3094
    frame #97: 0x00007fff87a715ad libdyld.dylib`start + 1
    frame #98: 0x00007fff87a715ad libdyld.dylib`start + 1

[stuhood]

Thanks a ton for investigating this, and for the debugging infrastructure improvements.

One approach that should work would be to ensure that we only apply Gitignore and Regex matches on the threads owned by the core. That way when the pool is destroyed/recreated, the thread local storage will be re-initialized. I'll see about doing that here.

[stuhood]

Hm... there is one big issue with this hypothesis though: pantsd is not in use here, and so there is no forking happening.

EDIT: And moreover, no threads except the main thread survive forking, so thread local storage should not be an issue there, afaik.

[stuhood]

Linking to rustwasm/wasm-bindgen#186, which sees a related stack.

[stuhood]

No luck with updating the ignore/regex dep in #5845.

[stuhood]

TODO: Try out MACOSX_DEPLOYMENT_TARGET=10.6 for the relevant shard.

[stuhood]

Resolved in favor of #6035.