-
Notifications
You must be signed in to change notification settings - Fork 317
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Verify checksum of static lib in build script #545
Comments
hmm -- @bnoordhuis has already suggested this kind of workaround in an early answer to my reports about this defect in our current build system. maybe he still hasn't changed his mind about the actual cause of this issue and still denies any explanation, which interpret the troubles as a consequence of inter-process race conditions in our build system. i don't know -- he simply didn't answer my additional analyses and suggested practical solutions resp. patch attempts until now.. i spend a lot of time last weekend to study this symptoms and their causes. it's a rather puzzling and hard to reproduce issue, like most errors caused by parallel running computing processes resp. unsynchronized data access. i still can't give an exact explanation, why cargo is running two instances of utilizing checksums or renaming/moving the completed file by an atomic operation, when the download is done, are IMHO just very unsatisfying workarounds, which do not solve the actual cause of the defect. in fact, they will most likely just result in a second parallel download attempt instead of blocking any further processing until the still ongoing first download finally becomes ready. that's why i suggested and used the utilization of lock files as much more adequate answer to this challenge in my PR (#543). |
I went over this with @bartlomieju yesterday and we concluded that there's no way for the build script to exhibit such behavior barring bugs in cargo. Possible, of course, but not reproducible. Neither of us really liked the idea of landing fixes for a poorly understood problem. Something like I saw the PR you opened but that uses I'm not completely opposed to some kind of mutual exclusion (in a "can't hurt - hopefully" fashion) but I'd use something like |
as already mentioned, there are many(!) reasons, why cargo may invoke the build script twice in parallel. you'll hardly prevent this behavior and foresee all coming changes in rusts build system, which will reintroduce this kind of problem in one or the other way. IMHO it's therefore nearly unavoidable to solve this issue in a proper manner!
yes -- all this available cross platform file locking solutions for rust are somehow questionable, but it's even harder to handle the same task by any new and creative home made invention much better. the rust std lib unfortunately only supports useful mutex/semaphore handling for intra-process synchronization out of the box, but no simple and reliable solution for inter-process barrier/lock handling. i have therefore chosen one of the most simple and small external crates for this purpose, which at least handles the most desirable set of requirements of such a mechanism (e.g. automatically removing the locks in case of interrupted process etc.) as long as you respect a few minor limitations (e.g. that you have to use the same open file handle and not only file path within your application on windows etc.). if you prefer any of the other available and well tested actual cross platform lock file implementations resp. external rust crates for practical reasons, i wouldn't care -- that's up to you! --, but just ignoring the real cause of our particular problem and debating workarounds, which will hardly solve the actual issue in a similar consequent, simple and reliable manner as by utilizing a properly implemented lock file solutions, looks hardly comprehensible to me -- sorry. :( |
Can you link me to discussion or documentation that supports that claim? None of the issues you linked to seem to make that claim, not even as bug reports. If you're referring to pipelining, I don't think that's relevant. Look at |
i think, the quoted 'cargo build -vv' logs reported in denoland/deno#8449 should verify this claim very well. but i could try to build a demonstration in GitLabs CI-system tonight, if you are still not satisfied by my explanations. it's only important, that you compile the package on a machine with very slow network and more than one core, because a |
o.k. here is the demonstration: i wasn't able to reproduce the issue by compiling only the i just utilized a slightly modified repo of https://gitlab.com/mash-graz/rusty_v8_slow_network/-/commit/7b294ef60b0ab8fb86a06b40052944cb0429d0f5 and in a modified repo of the output of this build process resp. the reproduced failure is documented at: https://gitlab.com/mash-graz/deno_slow_network/-/jobs/884293446 it's again this already reported: i hope, this practical demonstration resp. reproducible recipe helps to finally trust my claims. |
@bnoordhuis |
i just added references to a few other issues affected by this bug to this issue ticket resp. the suggested solution/explanation... |
#426 and denoland/deno#7517 seem to be very likely other issues affected by this bug... |
Closing in favor of #543 |
Actually it's still an issue to resolve |
could you please explain the reasons for reopening? at least on may machine it seems to work. nevertheless i also have my doubts, if this issue couldn't be fixed in an even more pleasant and general manner: i.e. but this are ideas for improvements, which i only found on mediating about this case after i wrote this first patch... so it's only a first small step forward, which hopefully eliminates at least one obvious source of troubles. |
File locking and checksumming are orthogonal issues. Checksumming is to detect bit flips and truncated downloads. |
yes, this are indeed two different topics. i was more interested in fixing this actual bug, which is/was only falsely reported as "truncated file" but in fact caused by other reasons, but it would be indeed another useful improvement, to compress and pack the relevant binary payloads in some kind of archive format (zip, rar, 7z etc.), that provide basic checksum validation automatically on decompression. this would also improve the download times drastically! |
As an alternative approach to fixing
truncated or malformed archive (offset to next archive member past the end of the archive after member /)
error (#543, denoland/deno#8449) we shall create checksum files in CI and later verify those checksums against downloaded static lib during build process.The text was updated successfully, but these errors were encountered: