troubles building deno in case of slow download of librusty_v8.a

[mash-graz]

there already a few bug reports available documenting this error: failed to add native library /tmp/deno/target/release/gn_out/obj/librusty_v8.a: truncated or malformed archive (offset to next archive member past the end of the archive after member /) issue (eg. #7115, #7517, #8385), but i think, most of them could be caused by some kind of strange race condition during the build process, if your internet connection is very slow and the download of librusty_v8.a (~50MB in release mode and 150MB in case of the debug version) takes to long, because other parts of the build process already see/use the incomplete archive and trigger this error:

error: failed to add native library /tmp/deno/target/release/gn_out/obj/librusty_v8.a: truncated or malformed archive (offset to next archive member past the end of the archive after member /)

error: aborting due to previous error

error: could not compile `rusty_v8`

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error: build failed

in practice it looks very strange to the user, because there is no warning or progress report about this background task visible and looks more like the build process somehow hangs, because it's still downloading after this fatal error message and doesn't come back...

if you build deno in a local cloned repo, it will at least succeed on a second invocation of cargo build -- as long as you didn't stop the background download by an hard interrupt. but this workaround will not help in case of cargo install deno, because it's always using a different directory for build resp. download on any successive run. in this case you will most likely just reproduce a very similar behavior on every further invocation.

[bnoordhuis]

As a workaround: https://github.com/denoland/rusty_v8#the-rusty_v8_mirror-environment-variable - precache the .a and tell the build script where to find it.

My hunch is that it's not a race condition but that the download finished prematurely (i.e., partial download.)

Perhaps we can fix that on the rusty_v8 side by checksumming the download.

[mash-graz]

As a workaround: https://github.com/denoland/rusty_v8#the-rusty_v8_mirror-environment-variable - precache the .a and tell the build script where to find it.

thanks for this hint, although i wouldn't see it as a proper solution concerning this issue.

My hunch is that it's not a race condition but that the download finished prematurely (i.e., partial download.)

no -- it's very simple to reproduce on any slow connected machine!
the download is still running in the background, when the error happens, etc.

but even on a fast connected machine you should be able to simulate the problem by utilizing some traffic shaping software or similar web testing tools.

the issue doesn't effect the rusty_v8 crate, if you only built it standalone. it's therefore much more likely caused by some kind of concurrency within the deno build, which doesn't satisfactorily watch out for a finished execution resp. download of this particular build dependency.

[mash-graz]

here is a precise log file file of the cargo build output and a timing diagram.

execution-log.txt
cargo-timing-20201121T045004Z.html.zip

in the terminal output we find entries reporting:

...
Compiling rusty_v8 v0.12.0
...
Completed rusty_v8 v0.12.0 custom-build in 0.9s
...
Completed rusty_v8 v0.12.0 custom-build (run) in 0.0s
...

before the error, but also once again at the end:

Completed rusty_v8 v0.12.0 custom-build (run) in 149.7s

[bnoordhuis]

I'm not able to reproduce what you're seeing, even when I slow down the download speed to a few kb/s.

I also don't see how it can happen. rusty_v8's build.rs downloads the .a synchronously.

You're using nightly builds of cargo/rustc, aren't you? Can you try with stable?

[mash-graz]

You're using nightly builds of cargo/rustc, aren't you? Can you try with stable?

no -- i just used nightly for this single run, because the timing reports aren't available in stable, but otherwise i always used the most recent (1.48) stable release. the behavior is exactly the same in both of them.

I also don't see how it can happen. rusty_v8's build.rs downloads the .a synchronously.

yes -- but it's running twice!

I'm not able to reproduce what you're seeing, even when I slow down the download speed to a few kb/s.

hmm -- on my machine it's perfectly reproducible on any run....

the issue looks more obvious, if you look for this significant section on a cargo build -vv run:

     Running `/tmp/deno/target/release/build/rusty_v8-a3138d9cdff67a1a/build-script-build`
[rusty_v8 0.12.0] static lib URL: https://github.com/denoland/rusty_v8/releases/download/v0.12.0/librusty_v8_release_x86_64-unknown-linux-gnu.a
[rusty_v8 0.12.0] cargo:rustc-link-search=/tmp/deno/target/release/gn_out/obj
[rusty_v8 0.12.0] Downloading https://github.com/denoland/rusty_v8/releases/download/v0.12.0/librusty_v8_release_x86_64-unknown-linux-gnu.a
     Running `/tmp/deno/target/release/build/rusty_v8-a3138d9cdff67a1a/build-script-build`
[rusty_v8 0.12.0] static lib URL: https://github.com/denoland/rusty_v8/releases/download/v0.12.0/librusty_v8_release_x86_64-unknown-linux-gnu.a
[rusty_v8 0.12.0] cargo:rustc-link-search=/tmp/deno/target/release/gn_out/obj
[rusty_v8 0.12.0] static lib already exists /tmp/deno/target/release/gn_out/obj/librusty_v8.a
[rusty_v8 0.12.0] To re-download this file, it must be manually deleted.
[rusty_v8 0.12.0] cargo:rustc-link-lib=static=rusty_v8

there are two build script invocations running in parallel, and the second one runs/finishes quite fast, because it assumes, that the lib is already available on the machine, although the file gets still downloaded by the first instance...

this triggers the /tmp/deno/target/release/gn_out/obj/librusty_v8.a: truncated or malformed archive (offset to next a rchive member past the end of the archive after member /) later on the first linking attempts.

i think i found a rather simple solution for this issue by utilizing lockfiles, which will block any further processing of rusty_v8 till the download completes (see: denoland/rusty_v8#543). i hope it is compatible to all platforms.

[mash-graz]

while updating all the installed rust binaries on my machine utilizing cargo install-update i got an reminder again, that the deno build process resp. this particular issue is still not fixed. :(

as demonstrated in denoland/rusty_v8#545 it's a perfectly reproducible flaw in the rusty_v8 build script, which affects nearly anybody using cargo to build deno from source. it's just a typical race problem, which unfortunately isn't visible on first sight resp. reported in a more helpful and unambiguous manner on the terminal.

@bartlomieju
please nominate another more objective reviewer for denoland/rusty_v8#543 to finally fix this annoying issue!

if someone finds a better solution than my lock file suggestion -- e.g. by wrapping the static library download in a additional pseudo crate to utilize the already available cargo build infrastructure and its download monitoring to avoid the present troubles --, that's perfectly acceptable to me, but simply ignoring/deny such a grave bug is IMHO indisputable.

[bartlomieju]

Fix landed in rusty_v8 in denoland/rusty_v8#543, I will keep this issue open until rusty_v8 is upgraded in deno.

[mash-graz]

@bartlomieju
thanks!