-
Notifications
You must be signed in to change notification settings - Fork 348
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cargo miri test
does not run doc-tests
#584
Comments
Doc tests need rustdoc as driver, while miri needs itself as driver. |
@rust-lang/rustdoc, can rustdoc be used as a library (not a driver) to extract the doctests without running them? |
Or, alternatively, can we set some environment variable to make rustdoc call miri instead of calling rustc and running the binary on each test after extraction? |
Unfortunately, librustdoc was taken out of the sysroot a while back, and i don't think we auto-publish the source for that as a crate either, so there's not really an easy solution for that. Rustdoc itself doesn't really provide functionality to only save off the extracted/converted doctest, either.
Same response - unless the test is manually marked as |
How open would the rustdoc team be to provide a way for Miri to run the extracted tests, and what would be the preferred way of achieving that (librustdoc vs controlling what happens after extraction)? For me it seems like the second option is the simpler one. |
The compilation of tests seems to be performed at https://github.com/rust-lang/rust/blob/17e62f77f954bed97aae839624bfd6dd68342daf/src/librustdoc/test.rs#L181. |
Yeah, now I see what @QuietMisdreavus meant by The "easiest" way to get Miri into this would be to provide a way to call some binary instead, but I am not sure if that is feasible. |
If there were some flag we could provide to just save off all the doctests as standalone rust files, would that be enough to allow miri to attempt to build them? They're meant to be compiled as binaries and link against the regular crate and its dependency tree. |
I guess we could make |
Cargo hands rustdoc all the |
Hm, that raises the interesting question how Currently I wouldn't know how |
Does |
AFAIK rustdoc uses the rustc driver interface directly, it does not run a binary. |
Oh right, I actually read that earlier in the thread today, but forgot about it now that it's Friday evening… My bad. |
With rust-lang/rust#63827, rustdoc now runs rustc out-of-process. That should make it easier to run Miri instead. @andjo403 is there a way to use an env var or so to control the binary that is run? |
not yet working on a new PR where RUSTC env var can be set but have some test to fix |
@Mark-Simulacrum added the |
Is it possible now to somehow trick miri to run doctests? |
|
Redo cargo-miri logic This rewrite the cargo-miri logic for running the requested crate(s) following what we outlined in #739: `cargo miri run/test $FLAGS` (almost) literally invokes `cargo run/test $FLAGS` but with some environment variables set so that we can control what happens: * `RUSTC_WRAPPER` is set so that we get invoked instead of `rustc`. We use that power to mess with the flags being used for the build (things to be interpreted by Miri use a different sysroot), and when we are detecting a binary crate meant to be run by Miri, we grab the info we care about and put it into a JSON file for later use. * `CARGO_TARGET_$TARGET_RUNNER` is set so what we get invoked when cargo wants to run a binary. At that point we take that JSON info from before and use it to invoke Miri. Overall this works great! We get all sorts of cargo magic for free, and do not even need `cargo-metadata` any more. There's one annoying point though, which I have not managed to entirely work around yet: this means we are doing a full build, not just a check-build. Given that our sysroot is MIR-only, I was surprised that this even worked, but still -- this means we are doing more work than we should. So I also added some patching of arguments to make sure `--emit` does not contain `link`, and then more patching was required of the `--extern` flags for the binary because those referenced the `.rlib` files but now only `.rmeta` exists, and that is still not fully working because cargo seems to expect those `.rmeta` files and now triggers a rebuild each time as those files are still missing. My current plan is to make our wrapper create some empty dummy files with the right names, but the amount of hacks we are stacking on top of each other here is getting worrysome.^^ `@ehuss` your input would be welcome on this issue. Due to re-using cargo more literally, this also changes flag parsing to match `cargo`. So `-Zmiri` flags now have to be passed via an environment variable (Cc #1416). This PR is not ready yet, but the parts that are there I think can be reviewed already. TODO: * [x] [Fix Windows](#1540 (comment)). * [x] Figure out how we can do check-only builds without the rebuild problem above. I am also worried about cases where `build.rs` script might detect check-only builds and then do less work; I think some crates in rustc are doing that and if this is a thing in the wider ecosystem we need to find a way to support this as well. * [x] Currently cargo runs doctests as well, and they are not run in Miri. We should at least figure out a way to not run them at all (resolving #584 is left for a future PR). * [x] For some time, detect the old way of passing `-Zmiri` flags and warn that this will need updating. For some simple situations we can probably make it still support the old way, but I plan to remove those hacks after a bit. This is just to give people and me time to go around and send PRs to all projects that use Miri on CI, and update their use of the flags. * [x] Add a test for stdin handling (#1505). This should work now but we should be sure. * [x] Add a test for cargo env vars (#1515). * [x] Check if #1516 is resolved. * [x] Check if #1001 and #1514 are resolved. * [x] Check if #1312 is resolved (not sure if it is wort adding a test). * [x] Check if #1512 is resolved (not sure if it is wort adding a test). Fixes #700. Fixes #739. Fixes #1001. Fixes #1312 (without a test, as we run without cargo's stdin/stdout wrapping now, as the test for stdin confirms). Fixes #1416. Fixes #1505. Fixes #1512 (without a test, as cargo now does all that handling anyway, which various other tests confirm). Fixes #1514. Fixes #1516. Cc `@alecmocatta` who reported many of the bugs above; would be great if you could help with the tests e.g. by providing some small examples I could try. r? `@oli-obk`
Redo cargo-miri logic This rewrite the cargo-miri logic for running the requested crate(s) following what we outlined in #739: `cargo miri run/test $FLAGS` (almost) literally invokes `cargo run/test $FLAGS` but with some environment variables set so that we can control what happens: * `RUSTC_WRAPPER` is set so that we get invoked instead of `rustc`. We use that power to mess with the flags being used for the build (things to be interpreted by Miri use a different sysroot), and when we are detecting a binary crate meant to be run by Miri, we grab the info we care about and put it into a JSON file for later use. * `CARGO_TARGET_$TARGET_RUNNER` is set so what we get invoked when cargo wants to run a binary. At that point we take that JSON info from before and use it to invoke Miri. Overall this works great! We get all sorts of cargo magic for free, and do not even need `cargo-metadata` any more. There's one annoying point though, which I have not managed to entirely work around yet: this means we are doing a full build, not just a check-build. Given that our sysroot is MIR-only, I was surprised that this even worked, but still -- this means we are doing more work than we should. So I also added some patching of arguments to make sure `--emit` does not contain `link`, and then more patching was required of the `--extern` flags for the binary because those referenced the `.rlib` files but now only `.rmeta` exists, and that is still not fully working because cargo seems to expect those `.rmeta` files and now triggers a rebuild each time as those files are still missing. My current plan is to make our wrapper create some empty dummy files with the right names, but the amount of hacks we are stacking on top of each other here is getting worrysome.^^ `@ehuss` your input would be welcome on this issue. Due to re-using cargo more literally, this also changes flag parsing to match `cargo`. So `-Zmiri` flags now have to be passed via an environment variable (Cc #1416). This PR is not ready yet, but the parts that are there I think can be reviewed already. TODO: * [x] [Fix Windows](#1540 (comment)). * [x] Figure out how we can do check-only builds without the rebuild problem above. I am also worried about cases where `build.rs` script might detect check-only builds and then do less work; I think some crates in rustc are doing that and if this is a thing in the wider ecosystem we need to find a way to support this as well. * [x] Currently cargo runs doctests as well, and they are not run in Miri. We should at least figure out a way to not run them at all (resolving #584 is left for a future PR). * [x] For some time, detect the old way of passing `-Zmiri` flags and warn that this will need updating. For some simple situations we can probably make it still support the old way, but I plan to remove those hacks after a bit. This is just to give people and me time to go around and send PRs to all projects that use Miri on CI, and update their use of the flags. * [x] Add a test for stdin handling (#1505). This should work now but we should be sure. * [x] Add a test for cargo env vars (#1515). * [x] Check if #1516 is resolved. * [x] Check if #1001 and #1514 are resolved. * [x] Check if #1312 is resolved (not sure if it is wort adding a test). * [x] Check if #1512 is resolved (not sure if it is wort adding a test). Fixes #700. Fixes #739. Fixes #1001. Fixes #1312 (without a test, as we run without cargo's stdin/stdout wrapping now, as the test for stdin confirms). Fixes #1416. Fixes #1505. Fixes #1512 (without a test, as cargo now does all that handling anyway, which various other tests confirm). Fixes #1514. Fixes #1516. Cc `@alecmocatta` who reported many of the bugs above; would be great if you could help with the tests e.g. by providing some small examples I could try. r? `@oli-obk`
Redo cargo-miri logic This rewrite the cargo-miri logic for running the requested crate(s) following what we outlined in #739: `cargo miri run/test $FLAGS` (almost) literally invokes `cargo run/test $FLAGS` but with some environment variables set so that we can control what happens: * `RUSTC_WRAPPER` is set so that we get invoked instead of `rustc`. We use that power to mess with the flags being used for the build (things to be interpreted by Miri use a different sysroot), and when we are detecting a binary crate meant to be run by Miri, we grab the info we care about and put it into a JSON file for later use. * `CARGO_TARGET_$TARGET_RUNNER` is set so what we get invoked when cargo wants to run a binary. At that point we take that JSON info from before and use it to invoke Miri. Overall this works great! We get all sorts of cargo magic for free, and do not even need `cargo-metadata` any more. There's one annoying point though, which I have not managed to entirely work around yet: this means we are doing a full build, not just a check-build. Given that our sysroot is MIR-only, I was surprised that this even worked, but still -- this means we are doing more work than we should. So I also added some patching of arguments to make sure `--emit` does not contain `link`, and then more patching was required of the `--extern` flags for the binary because those referenced the `.rlib` files but now only `.rmeta` exists, and that is still not fully working because cargo seems to expect those `.rmeta` files and now triggers a rebuild each time as those files are still missing. My current plan is to make our wrapper create some empty dummy files with the right names, but the amount of hacks we are stacking on top of each other here is getting worrysome.^^ `@ehuss` your input would be welcome on this issue. Due to re-using cargo more literally, this also changes flag parsing to match `cargo`. So `-Zmiri` flags now have to be passed via an environment variable (Cc #1416). This PR is not ready yet, but the parts that are there I think can be reviewed already. TODO: * [x] [Fix Windows](#1540 (comment)). * [x] Figure out how we can do check-only builds without the rebuild problem above. ~~I am also worried about cases where `build.rs` script might detect check-only builds and then do less work; I think some crates in rustc are doing that and if this is a thing in the wider ecosystem we need to find a way to support this as well.~~ (postponed that until we have a concrete example) * [x] Currently cargo runs doctests as well, and they are not run in Miri. We should at least figure out a way to not run them at all (resolving #584 is left for a future PR). * [x] For some time, detect the old way of passing `-Zmiri` flags and warn that this will need updating. For some simple situations we can probably make it still support the old way, but I plan to remove those hacks after a bit. This is just to give people and me time to go around and send PRs to all projects that use Miri on CI, and update their use of the flags. * [x] Add a test for stdin handling (#1505). This should work now but we should be sure. * [x] Add a test for cargo env vars (#1515). * [x] Check if #1516 is resolved. * [x] Check if #1001 and #1514 are resolved. * [x] Check if #1312 is resolved (not sure if it is wort adding a test). * [x] Check if #1512 is resolved (not sure if it is wort adding a test). Fixes #700. Fixes #739. Fixes #1001. Fixes #1312 (without a test, as we run without cargo's stdin/stdout wrapping now, as the test for stdin confirms). Fixes #1416. Fixes #1505. Fixes #1512 (without a test, as cargo now does all that handling anyway, which various other tests confirm). Fixes #1514. Fixes #1516. Cc `@alecmocatta` who reported many of the bugs above; would be great if you could help with the tests e.g. by providing some small examples I could try. r? `@oli-obk`
Redo cargo-miri logic This rewrite the cargo-miri logic for running the requested crate(s) following what we outlined in #739: `cargo miri run/test $FLAGS` (almost) literally invokes `cargo run/test $FLAGS` but with some environment variables set so that we can control what happens: * `RUSTC_WRAPPER` is set so that we get invoked instead of `rustc`. We use that power to mess with the flags being used for the build (things to be interpreted by Miri use a different sysroot), and when we are detecting a binary crate meant to be run by Miri, we grab the info we care about and put it into a JSON file for later use. * `CARGO_TARGET_$TARGET_RUNNER` is set so what we get invoked when cargo wants to run a binary. At that point we take that JSON info from before and use it to invoke Miri. Overall this works great! We get all sorts of cargo magic for free, and do not even need `cargo-metadata` any more. There's one annoying point though, which I have not managed to entirely work around yet: this means we are doing a full build, not just a check-build. Given that our sysroot is MIR-only, I was surprised that this even worked, but still -- this means we are doing more work than we should. So I also added some patching of arguments to make sure `--emit` does not contain `link`, and then more patching was required of the `--extern` flags for the binary because those referenced the `.rlib` files but now only `.rmeta` exists, and that is still not fully working because cargo seems to expect those `.rmeta` files and now triggers a rebuild each time as those files are still missing. My current plan is to make our wrapper create some empty dummy files with the right names, but the amount of hacks we are stacking on top of each other here is getting worrysome.^^ `@ehuss` your input would be welcome on this issue. Due to re-using cargo more literally, this also changes flag parsing to match `cargo`. So `-Zmiri` flags now have to be passed via an environment variable (Cc #1416). This PR is not ready yet, but the parts that are there I think can be reviewed already. TODO: * [x] [Fix Windows](#1540 (comment)). * [x] Figure out how we can do check-only builds without the rebuild problem above. ~~I am also worried about cases where `build.rs` script might detect check-only builds and then do less work; I think some crates in rustc are doing that and if this is a thing in the wider ecosystem we need to find a way to support this as well.~~ (postponed that until we have a concrete example) * [x] Currently cargo runs doctests as well, and they are not run in Miri. We should at least figure out a way to not run them at all (resolving #584 is left for a future PR). * [x] For some time, detect the old way of passing `-Zmiri` flags and warn that this will need updating. For some simple situations we can probably make it still support the old way, but I plan to remove those hacks after a bit. This is just to give people and me time to go around and send PRs to all projects that use Miri on CI, and update their use of the flags. * [x] Add a test for stdin handling (#1505). This should work now but we should be sure. * [x] Add a test for cargo env vars (#1515). * [x] Check if #1516 is resolved. * [x] Check if #1001 and #1514 are resolved. * [x] Check if #1312 is resolved (not sure if it is wort adding a test). * [x] Check if #1512 is resolved (not sure if it is wort adding a test). Fixes #700. Fixes #739. Fixes #1001. Fixes #1312 (without a test, as we run without cargo's stdin/stdout wrapping now, as the test for stdin confirms). Fixes #1416. Fixes #1505. Fixes #1512 (without a test, as cargo now does all that handling anyway, which various other tests confirm). Fixes #1514. Fixes #1516. Cc `@alecmocatta` who reported many of the bugs above; would be great if you could help with the tests e.g. by providing some small examples I could try. r? `@oli-obk`
Is anyone actively working on this? I'd be willing to invest a couple of afternoons over the coming week or two, but I could definitely use some pointers on how to get started. I never touched any of the involved projects before. I did read the contributor's guide, though. Based on what's been said here and what I gathered from reading code, this would basically boil down to replacing this error message with a call to a new function The problem is I'm not clear on what that function should do exactly. Invoke |
No, not that I know of.
That's great. :) However, I'm afraid there's not a ton of guidance available on this issue -- part of the problem here is even figuring out how to best do this.
Yes, that sounds right.
I don't know either what exactly it should do. ;) Probably the first step is to figure out the interaction between rustdoc and cargo, which is the thing that we are "hooking" here. Does cargo call rustdoc to build some binaries, and then the binaries are run by cargo, or does cargo run rustdoc and rustdoc runs some binaries? What kind of command-line arguments does cargo pass to rustdoc, and does Miri have to patch them in any way (besides adding |
Alright. I've skimmed what seemed to be the relevant parts of cargo, I guess I'll have to take a closer look then. Thanks for the guiding questions, that's already pretty helpful! |
Awesome. :) Don't hesitate to ask if you have further questions. (But also, I might not always be able to answer so quickly, so when delays happen that's just because I am busy.) Judging from |
Status report! I spent some time reading the source to
instead.
EDIT: Ok, I have a local build of rustdoc that I can actually hook into. Turns out my naive draft didn't work, no big surprise there. The way rustdoc invokes the test builder is indistinguishable from the way cargo invokes rustdoc, at least from the command line arguments. It does, however, set some Does the miri driver support piping code in like that? If so, I could make |
Ah, interesting -- good find.
Well,
Ugh. Yes that will be interesting.^^
Miri uses rustc for input file handling, so stdin should just work. It has never been tested though. |
It also sets a few env vars to fixup the source location shown in error messages. |
You mean
That would probably help in orchestrating the other phases, but I'm testing the Say if I'll give that a shot later today, but first I'll have to add a regression test to rustdoc to get my PR accepted :) |
Why is that needed? This case distinction already exists in master (to emit the error message).
Yes that works. |
It's not, you're right, my mistake. I got confused for a second there. This should work perfectly fine then. |
I seem to have a mostly working prototype! See the full code here and the diff here.
Problem is that EDIT: Ugh, it's a bit more complicated than that. The issue is the early return after writing the JSON file in EDIT 2: I went with a special case, for now. I figure it's easier to refactor later once I have a solution figured out. I can't seem to get the check-only build to work though, Miri keeps complaining about not being able to find the library crate the doc tests came from. I think that's because cargo-miri creates stub @RalfJung? (not sure if you see the edits if I don't mention you) EDIT 3: Okay, patching the
They all hit this exact same error, down to the Now that I think about it, this might just be because I switched from my local rustc/rustdoc build to the newest nightly, and Miri just doesn't work with that yet. |
I got it to work! 🎉 I was missing an I went ahead and opened a PR :D |
That's awesome. :) Thanks a lot for your work @teryror! |
I'm having trouble getting Miri to ignore certain doctests. Namely, I have some doctests that use SIMD. /// # Example
/// ```
/// #[cfg_attr(miri, ignore)]
/// fn main() { The above doctest winds up getting run anyway. Is there any method used to ignore a doctest? |
I don't think Did you try |
Ah, perfect! That works, thank you! |
I am not sure what it would take to also make it build and execute doc-tests. Probably just some tweaks in how we wrap
cargo rustc
?The text was updated successfully, but these errors were encountered: