Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build hang on Linux #76980

Open
danieldeankon opened this issue Sep 20, 2020 · 74 comments · Fixed by #78946
Open

Build hang on Linux #76980

danieldeankon opened this issue Sep 20, 2020 · 74 comments · Fixed by #78946
Labels
E-hard Call for participation: Hard difficulty. Experience needed to fix: A lot. E-needs-test Call for participation: An issue has been fixed and does not reproduce, but no test has been added. I-hang Issue: The compiler never terminates, due to infinite loops, deadlock, livelock, etc. P-medium Medium priority T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Comments

@danieldeankon
Copy link

Problem
I am seeing the build process hang at the last few steps when attempting to build this particular repository on Linux using either the nightly or the beta toolchain. I've seen this happen for the repository both when building locally (Linux Mint) and in Github's CI (Ubuntu). This does not happen with the stable toolchain, and I have not seen it happen with any toolchain on Windows.

I expect to see the build process finish in approximately 20 minutes like it does with the stable toolchain, but I observe a hanging build. If there is a problem with the build process, I expect to see an indication of what it is, especially if I pass a verbose flag, but I observe no further output once it gets into the hanging state---the only output I see is from steps that have already finished, which provides no insight into why it hangs.

Steps

Switch to nightly toolchain:

$ rustup default nightly
info: using existing install for 'nightly-x86_64-unknown-linux-gnu'
info: default toolchain set to 'nightly-x86_64-unknown-linux-gnu'

  nightly-x86_64-unknown-linux-gnu unchanged - rustc 1.48.0-nightly (bbc677480 2020-09-18)

Fetch the following repository:

$ git clone https://github.com/danieldeankon/hypermine.git
$ cd hypermine/
$ git lfs pull

Attempt to build the binaries:

$ cargo build --release -v

The last thing I see before it gets into a hanging state:

   Compiling client v0.1.0 (/home/daniel/git/testinghypermine/hypermine/client)
     Running `rustc --crate-name client --edition=2018 client/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg 'feature="default"' --cfg 'feature="use-repo-assets"' -C metadata=c38d85b37e3e2e40 -C extra-filename=-c38d85b37e3e2e40 --out-dir /home/daniel/git/testinghypermine/hypermine/target/release/deps -L dependency=/home/daniel/git/testinghypermine/hypermine/target/release/deps --extern anyhow=/home/daniel/git/testinghypermine/hypermine/target/release/deps/libanyhow-43ee26a5a4ae988b.rmeta --extern ash=/home/daniel/git/testinghypermine/hypermine/target/release/deps/libash-1d91e46c2845798f.rmeta --extern ash_window=/home/daniel/git/testinghypermine/hypermine/target/release/deps/libash_window-e581873b9287fab4.rmeta --extern common=/home/daniel/git/testinghypermine/hypermine/target/release/deps/libcommon-bdb19128efefc960.rmeta --extern directories=/home/daniel/git/testinghypermine/hypermine/target/release/deps/libdirectories-4a64043fdaf679b9.rmeta --extern downcast_rs=/home/daniel/git/testinghypermine/hypermine/target/release/deps/libdowncast_rs-be493f8657cf1f42.rmeta --extern futures_util=/home/daniel/git/testinghypermine/hypermine/target/release/deps/libfutures_util-79782c1686738829.rmeta --extern fxhash=/home/daniel/git/testinghypermine/hypermine/target/release/deps/libfxhash-aa9a2aa3814ab312.rmeta --extern gltf=/home/daniel/git/testinghypermine/hypermine/target/release/deps/libgltf-ae78e76833b667c0.rmeta --extern hdrhistogram=/home/daniel/git/testinghypermine/hypermine/target/release/deps/libhdrhistogram-1c5607d77dc61a11.rmeta --extern hecs=/home/daniel/git/testinghypermine/hypermine/target/release/deps/libhecs-7ceed138b109be7b.rmeta --extern lahar=/home/daniel/git/testinghypermine/hypermine/target/release/deps/liblahar-c3a4855d049ab353.rmeta --extern memoffset=/home/daniel/git/testinghypermine/hypermine/target/release/deps/libmemoffset-33ada65033678cfe.rmeta --extern metrics=/home/daniel/git/testinghypermine/hypermine/target/release/deps/libmetrics-79a79d3342c8701e.rmeta --extern metrics_core=/home/daniel/git/testinghypermine/hypermine/target/release/deps/libmetrics_core-7e657b2c60f0cf49.rmeta --extern na=/home/daniel/git/testinghypermine/hypermine/target/release/deps/libnalgebra-615d0479649eed30.rmeta --extern png=/home/daniel/git/testinghypermine/hypermine/target/release/deps/libpng-089627b494fcd05d.rmeta --extern quinn=/home/daniel/git/testinghypermine/hypermine/target/release/deps/libquinn-a3a9da2f9e31ef73.rmeta --extern rcgen=/home/daniel/git/testinghypermine/hypermine/target/release/deps/librcgen-be435a028b2dfce2.rmeta --extern rustls=/home/daniel/git/testinghypermine/hypermine/target/release/deps/librustls-825cbe6b34245b7a.rmeta --extern serde=/home/daniel/git/testinghypermine/hypermine/target/release/deps/libserde-fbaa982a65688d4a.rmeta --extern server=/home/daniel/git/testinghypermine/hypermine/target/release/deps/libserver-8fd1f97c8fb84524.rmeta --extern tokio=/home/daniel/git/testinghypermine/hypermine/target/release/deps/libtokio-aa180fba706594fa.rmeta --extern toml=/home/daniel/git/testinghypermine/hypermine/target/release/deps/libtoml-f7f6190f25a07c8e.rmeta --extern tracing=/home/daniel/git/testinghypermine/hypermine/target/release/deps/libtracing-e0bf1ed6aae74c7d.rmeta --extern vk_shader_macros=/home/daniel/git/testinghypermine/hypermine/target/release/deps/libvk_shader_macros-d2fb7777a36a4f79.rmeta --extern webpki=/home/daniel/git/testinghypermine/hypermine/target/release/deps/libwebpki-66dacb92629212d5.rmeta --extern whoami=/home/daniel/git/testinghypermine/hypermine/target/release/deps/libwhoami-05f81a39d2c15f6f.rmeta --extern winit=/home/daniel/git/testinghypermine/hypermine/target/release/deps/libwinit-0172bcc7807380da.rmeta -L native=/home/daniel/git/testinghypermine/hypermine/target/release/build/ring-86d47ac3f01cd916/out`
    Building [=====================================================> ] 255/257: client

Notes

I was able to reproduce this behaviour in Github CI here: https://github.com/danieldeankon/hypermine/pull/2/checks?check_run_id=1138809646

CI / test (ubuntu-latest, stable) finished in 17m 39s. CI / test (ubuntu-latest, beta) has been going for two hours and counting at time of writing, and similarly for CI / test (ubuntu-latest, nightly).

Output of cargo version:
cargo 1.48.0-nightly (8777a6b1e 2020-09-15)

My operating system is Linux Mint 19.2 Cinnamon.

@ehuss ehuss transferred this issue from rust-lang/cargo Sep 20, 2020
@jonas-schievink jonas-schievink added the regression-from-stable-to-nightly Performance or correctness regression from stable to nightly. label Sep 20, 2020
@rustbot rustbot added the I-prioritize Issue: Indicates that prioritization has been requested for this issue. label Sep 20, 2020
@ehuss
Copy link
Contributor

ehuss commented Sep 20, 2020

Transferred the issue to rust-lang/rust. This is a rustc issue, and I don't think there is a lot that Cargo can do about it. rustc is doing work, it just never finishes.

I have bisected this to #73526. cc @cuviper, are there any known issues with the LLVM 11 upgrade causing rustc to go into any kind of infinite loops? When I attached with gdb, it looked like it was deep inside a proc macro (vk-shader-macros).

@LeSeulArtichaut LeSeulArtichaut added I-hang Issue: The compiler never terminates, due to infinite loops, deadlock, livelock, etc. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Sep 20, 2020
@cuviper cuviper added regression-from-stable-to-beta Performance or correctness regression from stable to beta. and removed I-hang Issue: The compiler never terminates, due to infinite loops, deadlock, livelock, etc. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. regression-from-stable-to-nightly Performance or correctness regression from stable to nightly. labels Sep 20, 2020
@cuviper
Copy link
Member

cuviper commented Sep 20, 2020

I'm not aware of any LLVM 11 issues like that.

@LeSeulArtichaut

This comment has been minimized.

@cuviper

This comment has been minimized.

@LeSeulArtichaut

This comment has been minimized.

@LeSeulArtichaut LeSeulArtichaut added I-hang Issue: The compiler never terminates, due to infinite loops, deadlock, livelock, etc. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. A-LLVM Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues. labels Sep 20, 2020
@LeSeulArtichaut
Copy link
Contributor

@rustbot ping llvm

@rustbot rustbot added the ICEBreaker-LLVM Bugs identified for the LLVM ICE-breaker group label Sep 20, 2020
@rustbot
Copy link
Collaborator

rustbot commented Sep 20, 2020

Hey LLVM ICE-breakers! This bug has been identified as a good
"LLVM ICE-breaking candidate". In case it's useful, here are some
instructions for tackling these sorts of bugs. Maybe take a look?
Thanks! <3

cc @camelid @comex @cuviper @DutchGhost @hdhoang @heyrutvik @higuoxing @JOE1994 @jryans @mmilenko @nagisa @nikic @Noah-Kennedy @SiavoshZarrasvand @spastorino @vertexclique

@jyn514 jyn514 added the E-needs-mcve Call for participation: This issue has a repro, but needs a Minimal Complete and Verifiable Example label Sep 20, 2020
@apiraino
Copy link
Contributor

Assigning P-high as discussed as part of the Prioritization Working Group procedure and removing I-prioritize.

@apiraino apiraino added P-high High priority and removed I-prioritize Issue: Indicates that prioritization has been requested for this issue. labels Sep 23, 2020
@LeSeulArtichaut LeSeulArtichaut added I-nominated E-needs-bisection Call for participation: This issue needs bisection: https://github.com/rust-lang/cargo-bisect-rustc and removed E-needs-bisection Call for participation: This issue needs bisection: https://github.com/rust-lang/cargo-bisect-rustc labels Sep 24, 2020
@LeSeulArtichaut
Copy link
Contributor

I couldn't reproduce this issue with my MacOS laptop (x86_64-apple-darwin). I tried compiling with stable (1.46.0), nightly-2020-09-18 and latest nightly (nightly-2020-09-23), resulting in a successful compilation in less than 15 minutes. Unless I missed something, I guess this probably means this is a Linux-only issue?
@ehuss Did you also use a Linux machine for your previous bisection?

It would be great if other people could take a look and eventually find an MCVE which would be very helpful to resolve this bug.
@rustbot ping cleanup

@rustbot
Copy link
Collaborator

rustbot commented Sep 24, 2020

Hey Cleanup Crew ICE-breakers! This bug has been identified as a good
"Cleanup ICE-breaking candidate". In case it's useful, here are some
instructions for tackling these sorts of bugs. Maybe take a look?
Thanks! <3

cc @AminArria @camelid @chrissimpkins @contrun @DutchGhost @elshize @ethanboxx @h-michael @HallerPatrick @hdhoang @hellow554 @imtsuki @JamesPatrickGill @kanru @KarlK90 @LeSeulArtichaut @MAdrianMattocks @matheus-consoli @mental32 @nmccarty @Noah-Kennedy @pard68 @PeytonT @pierreN @Redblueflame @RobbieClarken @RobertoSnap @robjtede @SarthakSingh31 @senden9 @shekohex @sinato @smmalis37 @spastorino @Stupremee @turboladen @woshilapin @yerke

@rustbot rustbot added the ICEBreaker-Cleanup-Crew Helping to "clean up" bugs with minimal examples and bisections label Sep 24, 2020
@Mark-Simulacrum Mark-Simulacrum added regression-from-stable-to-nightly Performance or correctness regression from stable to nightly. and removed regression-from-stable-to-beta Performance or correctness regression from stable to beta. labels Nov 5, 2020
@Mark-Simulacrum Mark-Simulacrum changed the title Build hang on Linux with nightly and beta toolchains Build hang on Linux with nightly toolchain Nov 5, 2020
@Mark-Simulacrum Mark-Simulacrum changed the title Build hang on Linux with nightly toolchain Build hang on Linuux Nov 5, 2020
@Mark-Simulacrum Mark-Simulacrum changed the title Build hang on Linuux Build hang on Linux Nov 5, 2020
@pnkfelix
Copy link
Member

pnkfelix commented Nov 6, 2020

ping @rust-lang/wg-llvm : is there anyone able to assist with investigating what change between LLVM 10 and LLVM 11 actually caused this, as described in #76980 (comment)

@jethrogb
Copy link
Contributor

jethrogb commented Nov 6, 2020

@pnkfelix I think that's a red herring, see #76980 (comment)

@Mark-Simulacrum
Copy link
Member

@jethrogb Are there any potential downsides to passing -fno-gnu-unique when compiling LLVM? It seems like out of the options we have available that's the easiest and least invasive one, right?

@jethrogb
Copy link
Contributor

jethrogb commented Nov 6, 2020

I'm not aware of any downsides, but I also don't know what the purpose is of this unique type of symbols and why GCC added this special class. It is now my preferred option as well.

I'm hoping I can coerce x.py to use that flag sometime soon. Happy to mentor someone on that as well.

@cuviper
Copy link
Member

cuviper commented Nov 6, 2020

Do we need that flag in the build of libstdc++ as well? (Feasible since we do build our own toolchain for dist...)

@jethrogb
Copy link
Contributor

How does the statically-linked version of libLLVM get built? I ran the dist-x86_64-linux builder but that results in a dynamically-linked libstdc++.

@jethrogb
Copy link
Contributor

@mati865
Copy link
Contributor

mati865 commented Nov 10, 2020

@jethrogb
Copy link
Contributor

The GCC version in the build container is new enough to generate unique symbols but too old to have -fno-gnu-unique 😭

@jethrogb
Copy link
Contributor

PR: #78946

@bors bors closed this as completed in 5404efc Nov 11, 2020
@Mark-Simulacrum
Copy link
Member

Reopening as E-needs-test, but dropping the regression label. See discussion on #78946 for some context there.

@Mark-Simulacrum Mark-Simulacrum added E-hard Call for participation: Hard difficulty. Experience needed to fix: A lot. E-needs-test Call for participation: An issue has been fixed and does not reproduce, but no test has been added. P-medium Medium priority and removed O-linux Operating system: Linux P-high High priority regression-from-stable-to-nightly Performance or correctness regression from stable to nightly. labels Nov 11, 2020
@tesuji
Copy link
Contributor

tesuji commented Nov 12, 2020

Not opened yet.

@spastorino spastorino reopened this Nov 12, 2020
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this issue Nov 13, 2020
Pkgsrc changes:
 * Remove patches now integrated upstream, many related to SunOS / Illumos.
 * The LLVM fix for powerpc is also now integrated upstream.
 * Adapt those patches where the source has moved or parts are integrated.
 * The randomness patches no longer applies, and I could not find
   where those files went...
 * Provide a separate bootstrap for NetBSD/powerpc 9.0, since apparently
   the C++ ABI is different from 8.0.  Yes, this appears to be specific to
   the NetBSD powerpc ports.

Upstream changes:

Version 1.47.0 (2020-10-08)
==========================

Language
--------
- [Closures will now warn when not used.][74869]

Compiler
--------
- [Stabilized the `-C control-flow-guard` codegen option][73893], which enables
  [Control Flow Guard][1.47.0-cfg] for Windows platforms, and is ignored on
  other platforms.
- [Upgraded to LLVM 11.][73526]
- [Added tier 3\* support for the `thumbv4t-none-eabi` target.][74419]
- [Upgrade the FreeBSD toolchain to version 11.4][75204]
- [`RUST_BACKTRACE`'s output is now more compact.][75048]

\* Refer to Rust's [platform support page][forge-platform-support] for more
information on Rust's tiered platform support.

Libraries
---------
- [`CStr` now implements `Index<RangeFrom<usize>>`.][74021]
- [Traits in `std`/`core` are now implemented for arrays of any length, not just
  those of length less than 33.][74060]
- [`ops::RangeFull` and `ops::Range` now implement Default.][73197]
- [`panic::Location` now implements `Copy`, `Clone`, `Eq`, `Hash`, `Ord`,
  `PartialEq`, and `PartialOrd`.][73583]

Stabilized APIs
---------------
- [`Ident::new_raw`]
- [`Range::is_empty`]
- [`RangeInclusive::is_empty`]
- [`Result::as_deref`]
- [`Result::as_deref_mut`]
- [`Vec::leak`]
- [`pointer::offset_from`]
- [`f32::TAU`]
- [`f64::TAU`]

The following previously stable APIs have now been made const.

- [The `new` method for all `NonZero` integers.][73858]
- [The `checked_add`,`checked_sub`,`checked_mul`,`checked_neg`, `checked_shl`,
  `checked_shr`, `saturating_add`, `saturating_sub`, and `saturating_mul`
  methods for all integers.][73858]
- [The `checked_abs`, `saturating_abs`, `saturating_neg`, and `signum`  for all
  signed integers.][73858]
- [The `is_ascii_alphabetic`, `is_ascii_uppercase`, `is_ascii_lowercase`,
  `is_ascii_alphanumeric`, `is_ascii_digit`, `is_ascii_hexdigit`,
  `is_ascii_punctuation`, `is_ascii_graphic`, `is_ascii_whitespace`, and
  `is_ascii_control` methods for `char` and `u8`.][73858]

Cargo
-----
- [`build-dependencies` are now built with opt-level 0 by default.][cargo/8500]
  You can override this by setting the following in your `Cargo.toml`.
  ```toml
  [profile.release.build-override]
  opt-level = 3
  ```
- [`cargo-help` will now display man pages for commands rather just the
  `--help` text.][cargo/8456]
- [`cargo-metadata` now emits a `test` field indicating if a target has
  tests enabled.][cargo/8478]
- [`workspace.default-members` now respects `workspace.exclude`.][cargo/8485]
- [`cargo-publish` will now use an alternative registry by default if it's the
  only registry specified in `package.publish`.][cargo/8571]

Misc
----
- [Added a help button beside Rustdoc's searchbar that explains rustdoc's
  type based search.][75366]
- [Added the Ayu theme to rustdoc.][71237]

Compatibility Notes
-------------------
- [Bumped the minimum supported Emscripten version to 1.39.20.][75716]
- [Fixed a regression parsing `{} && false` in tail expressions.][74650]
- [Added changes to how proc-macros are expanded in `macro_rules!` that should
  help to preserve more span information.][73084] These changes may cause
  compiliation errors if your macro was unhygenic or didn't correctly handle
  `Delimiter::None`.
- [Moved support for the CloudABI target to tier 3.][75568]
- [`linux-gnu` targets now require minimum kernel 2.6.32 and glibc 2.11.][74163]

Internal Only
--------
- [Improved default settings for bootstrapping in `x.py`.][73964]
  You can read details about this change in the ["Changes to `x.py`
  defaults"](https://blog.rust-lang.org/inside-rust/2020/08/30/changes-to-x-py-defaults.html)
  post on the Inside Rust blog.

- [Added the `rustc-docs` component.][75560] This allows you to install
  and read the documentation for the compiler internal APIs. (Currently only
  available for `x86_64-unknown-linux-gnu`.)

[1.47.0-cfg]: https://docs.microsoft.com/en-us/windows/win32/secbp/control-flow-guard
[76980]: rust-lang/rust#76980
[75048]: rust-lang/rust#75048
[74163]: rust-lang/rust#74163
[71237]: rust-lang/rust#71237
[74869]: rust-lang/rust#74869
[73858]: rust-lang/rust#73858
[75716]: rust-lang/rust#75716
[75908]: rust-lang/rust#75908
[75516]: rust-lang/rust#75516
[75560]: rust-lang/rust#75560
[75568]: rust-lang/rust#75568
[75366]: rust-lang/rust#75366
[75204]: rust-lang/rust#75204
[74650]: rust-lang/rust#74650
[74419]: rust-lang/rust#74419
[73964]: rust-lang/rust#73964
[74021]: rust-lang/rust#74021
[74060]: rust-lang/rust#74060
[73893]: rust-lang/rust#73893
[73526]: rust-lang/rust#73526
[73583]: rust-lang/rust#73583
[73084]: rust-lang/rust#73084
[73197]: rust-lang/rust#73197
[72488]: rust-lang/rust#72488
[cargo/8456]: rust-lang/cargo#8456
[cargo/8478]: rust-lang/cargo#8478
[cargo/8485]: rust-lang/cargo#8485
[cargo/8500]: rust-lang/cargo#8500
[cargo/8571]: rust-lang/cargo#8571
[`Ident::new_raw`]:  https://doc.rust-lang.org/nightly/proc_macro/struct.Ident.html#method.new_raw
[`Range::is_empty`]: https://doc.rust-lang.org/nightly/std/ops/struct.Range.html#method.is_empty
[`RangeInclusive::is_empty`]: https://doc.rust-lang.org/nightly/std/ops/struct.RangeInclusive.html#method.is_empty
[`Result::as_deref_mut`]: https://doc.rust-lang.org/nightly/std/result/enum.Result.html#method.as_deref_mut
[`Result::as_deref`]: https://doc.rust-lang.org/nightly/std/result/enum.Result.html#method.as_deref
[`TypeId::of`]: https://doc.rust-lang.org/nightly/std/any/struct.TypeId.html#method.of
[`Vec::leak`]: https://doc.rust-lang.org/nightly/std/vec/struct.Vec.html#method.leak
[`f32::TAU`]: https://doc.rust-lang.org/nightly/std/f32/consts/constant.TAU.html
[`f64::TAU`]: https://doc.rust-lang.org/nightly/std/f64/consts/constant.TAU.html
[`pointer::offset_from`]: https://doc.rust-lang.org/nightly/std/primitive.pointer.html#method.offset_from
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
E-hard Call for participation: Hard difficulty. Experience needed to fix: A lot. E-needs-test Call for participation: An issue has been fixed and does not reproduce, but no test has been added. I-hang Issue: The compiler never terminates, due to infinite loops, deadlock, livelock, etc. P-medium Medium priority T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet