From 87ac39c8000bec014154d28704177a72a64f7410 Mon Sep 17 00:00:00 2001 From: Eric Huss Date: Sat, 20 Feb 2021 15:34:12 -0800 Subject: [PATCH 1/2] Add a chapter on the test harness. --- library/test/src/cli.rs | 10 +- src/doc/rustc/src/SUMMARY.md | 1 + src/doc/rustc/src/command-line-arguments.md | 3 +- src/doc/rustc/src/tests/index.md | 283 ++++++++++++++++++++ 4 files changed, 288 insertions(+), 9 deletions(-) create mode 100644 src/doc/rustc/src/tests/index.md diff --git a/library/test/src/cli.rs b/library/test/src/cli.rs index c0b5197e997bb..b7791b1b24d45 100644 --- a/library/test/src/cli.rs +++ b/library/test/src/cli.rs @@ -51,14 +51,8 @@ fn optgroups() -> getopts::Options { .optflag("", "test", "Run tests and not benchmarks") .optflag("", "bench", "Run benchmarks instead of tests") .optflag("", "list", "List all tests and benchmarks") - .optflag("h", "help", "Display this message (longer with --help)") - .optopt( - "", - "logfile", - "Write logs to the specified file instead \ - of stdout", - "PATH", - ) + .optflag("h", "help", "Display this message") + .optopt("", "logfile", "Write logs to the specified file", "PATH") .optflag( "", "nocapture", diff --git a/src/doc/rustc/src/SUMMARY.md b/src/doc/rustc/src/SUMMARY.md index dd19861573667..af5834525fa79 100644 --- a/src/doc/rustc/src/SUMMARY.md +++ b/src/doc/rustc/src/SUMMARY.md @@ -11,6 +11,7 @@ - [Deny-by-default lints](lints/listing/deny-by-default.md) - [Codegen options](codegen-options/index.md) - [JSON Output](json.md) +- [Tests](tests/index.md) - [Platform Support](platform-support.md) - [Targets](targets/index.md) - [Built-in Targets](targets/built-in.md) diff --git a/src/doc/rustc/src/command-line-arguments.md b/src/doc/rustc/src/command-line-arguments.md index 3d6579250a014..7d3cad7988c8c 100644 --- a/src/doc/rustc/src/command-line-arguments.md +++ b/src/doc/rustc/src/command-line-arguments.md @@ -204,7 +204,8 @@ out a longer explanation of a given error. ## `--test`: build a test harness When compiling this crate, `rustc` will ignore your `main` function -and instead produce a test harness. +and instead produce a test harness. See the [Tests chapter](tests/index.md) +for more information about tests. ## `--target`: select a target triple to build diff --git a/src/doc/rustc/src/tests/index.md b/src/doc/rustc/src/tests/index.md new file mode 100644 index 0000000000000..1f914da4fc55e --- /dev/null +++ b/src/doc/rustc/src/tests/index.md @@ -0,0 +1,283 @@ +# Tests + +`rustc` has a built-in facility for building and running tests for a crate. +More information about writing and running tests may be found in the [Testing +Chapter] of the Rust Programming Language book. + +Tests are written as free functions with the [`#[test]` +attribute][attribute-test]. For example: + +```rust +#[test] +fn it_works() { + assert_eq!(2 + 2, 4); +} +``` + +Tests "pass" if they return without an error. They "fail" if they [panic], or +return a [`Result`] with an error. + +By passing the [`--test` option] to `rustc`, the compiler will build the crate +in a special mode to construct an executable that will run the tests in the +crate. The `--test` flag will make the following changes: + +* The crate will be built as a `bin` [crate type], forcing it to be an + executable. +* Links the executable with [`libtest`], the test harness that is part of the + standard library, which handles running the tests. +* Synthesizes a [`main` function] which will process command-line arguments + and run the tests. If the crate already has a `main` function, it will be + replaced. +* Enables the [`test` cfg option], which allows your code to use conditional + compilation to detect if it is being built as a test. +* Enables building of functions annotated with the [`test`][attribute-test] + and [`bench`](#benchmarks) attributes, which will be run by the test + harness. + +After the executable is created, you can run it to execute the tests and +receive a report on what passes and fails. If you are using [Cargo] to manage +your project, it has a built-in [`cargo test`] command which handles all of +this automatically. An example of the output looks like this: + +```text +running 4 tests +test it_works ... ok +test check_valid_args ... ok +test invalid_characters ... ok +test walks_the_dog ... ok + +test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s +``` + +> **Note**: Tests must be built with the [`unwind` panic +> strategy][panic-strategy]. This is because all tests run in the same +> process, and they are intended to catch panics, which is not possible with +> the `abort` strategy. See the unstable [`-Z panic-abort-tests`] option for +> experimental support of the `abort` strategy by spawning tests in separate +> processes. + +## Test attributes + +Tests are indicated using attributes on free functions. The following +attributes are used for testing, see the linked documentation for more +details: + +* [`#[test]`][attribute-test] — Indicates a function is a test to be run. +* `#[bench]` — Indicates a function is a benchmark to be + run. Benchmarks are currently unstable and only available in the nightly + channel, see the [unstable docs][bench-docs] for more details. +* [`#[should_panic]`][attribute-should_panic] — Indicates that the test + function will only pass if the function [panics][panic]. +* [`#[ignore]`][attribute-ignore] — Indicates that the test function will be + compiled, but not run by default. See the [`--ignored`](#--ignored) and + [`--include-ignored`](#--include-ignored) options to run these tests. + +## CLI arguments + +The libtest harness has several command-line arguments to control its +behavior. + +> Note: When running with [`cargo test`], the libtest CLI arguments must be +> passed after the `--` argument to differentiate between flags for Cargo and +> those for the harness. For example: `cargo test -- --nocapture` + +### Filters + +Bare arguments (those without a `-` prefix) are treated as filters which will +only run tests whose name matches one of those strings. The filter will match +any substring found in the full path of the test function. For example, if the +test function `it_works` is located in the module `utils::paths::tests`, then +any of the filters `works`, `path`, `utils::`, or +`utils::paths::tests::it_works` will match that test. + +See [Selection options](#selection-options) for more options to control which +tests are run. + +### Action options + +The following options perform different actions other than running tests. + +#### `--list` + +Prints a list of all tests and benchmarks. Does not run any of the tests. +[Filters](#filters) can be used to list only matching tests. + +#### `-h`, `--help` + +Displays usage information and command-line options. + +### Selection options + +The following options change how tests are selected. + +#### `--test` + +This is the default mode where all tests will be run as well as running all +benchmarks with only a single iteration (to ensure the benchmark works, +without taking the time to actually perform benchmarking). This can be +combined with the `--bench` flag to run both tests and perform full +benchmarking. + +#### `--bench` + +This runs in a mode where tests are ignored, and only runs benchmarks. This +can be combined with `--test` to run both benchmarks and tests. + +#### `--exact` + +This forces [filters](#filters) to match the full path of the test exactly. +For example, if the test `it_works` is in the module `utils::paths::tests`, +then only the string `utils::paths::tests::it_works` will match that test. + +#### `--skip` _FILTER_ + +Skips any tests whose name contains the given _FILTER_ string. This flag may +be passed multiple times. + +#### `--ignored` + +Runs only tests that are marked with the [`ignore` +attribute][attribute-ignore]. + +#### `--include-ignored` + +Runs both [ignored](#--ignored) and non-ignored tests. + +#### `--exclude-should-panic` + +Excludes tests marked with the [`should_panic` +attribute][attribute-should_panic]. + +⚠️ 🚧 This option is [unstable](#unstable-options), and requires the `-Z +unstable-options` flag. See [tracking issue +#82348](https://github.com/rust-lang/rust/issues/82348) for more information. + +### Execution options + +The following options affect how tests are executed. + +#### `--test-threads` _NUM_THREADS_ + +Sets the number of threads to use for running tests in parallel. By default, +uses the amount of concurrency available on the hardware as indicated by +[`available_concurrency`]. + +This can also be specified with the `RUST_TEST_THREADS` environment variable. + +#### `--force-run-in-process` + +Forces the tests to run in a single process when using the [`abort` panic +strategy][panic-strategy]. + +⚠️ 🚧 This only works with the unstable [`-Z panic-abort-tests`] option, and +requires the `-Z unstable-options` flag. See [tracking issue +#67650](https://github.com/rust-lang/rust/issues/67650) for more information. + +#### `--ensure-time` + +⚠️ 🚧 This option is [unstable](#unstable-options), and requires the `-Z +unstable-options` flag. See [tracking issue +#64888](https://github.com/rust-lang/rust/issues/64888) and the [unstable +docs](../../unstable-book/compiler-flags/report-time.html) for more information. + +### Output options + +The following options affect the output behavior. + +#### `-q`, `--quiet` + +Displays one character per test instead of one line per test. This is an alias +for [`--format=terse`](#--format-format). + +#### `--nocapture` + +Does not capture the stdout and stderr of the test, and allows tests to print +to the console. Usually the output is captured, and only displayed if the test +fails. + +This may also be specified by setting the `RUST_TEST_NOCAPTURE` environment +variable set to anything but `0`. + +#### `--show-output` + +Displays the stdout and stderr of successful tests after all tests have run. + +Contrast this with [`--nocapture`](#--nocapture) which allows tests to print +*while they are running*, which can cause interleaved output if there are +multiple tests running in parallel, `--show-output` ensures the output is +contiguous, but requires waiting for all tests to finish. + +#### `--color` _COLOR_ + +Control when colored terminal output is used. Valid options: + +* `auto`: Colorize if stdout is a tty and [`--nocapture`](#--nocapture) is not + used. This is the default. +* `always`: Always colorize the output. +* `never`: Never colorize the output. + +#### `--format` _FORMAT_ + +Controls the format of the output. Valid options: + +* `pretty`: This is the default format, with one line per test. +* `terse`: Displays only a single character per test. [`--quiet`](#-q---quiet) + is an alias for this option. +* `json`: Emits JSON objects, one per line. ⚠️ 🚧 This option is + [unstable](#unstable-options), and requires the `-Z unstable-options` flag. + See [tracking issue #49359](https://github.com/rust-lang/rust/issues/49359) + for more information. + +#### `--logfile` _PATH_ + +Writes the results to the tests to the given file. + +#### `--report-time` _FORMAT_ + +⚠️ 🚧 This option is [unstable](#unstable-options), and requires the `-Z +unstable-options` flag. See [tracking issue +#64888](https://github.com/rust-lang/rust/issues/64888) and the [unstable +docs](../../unstable-book/compiler-flags/report-time.html) for more information. + +### Unstable options + +Some CLI options are added in an "unstable" state, where they are intended for +experimentation and testing to determine if the option works correctly, has +the right design, and is useful. The option may not work correctly, break, or +change at at any time. To signal that you acknowledge that you are using an +unstable option, they require passing the `-Z unstable-options` command-line +flag. + +## Benchmarks + +The libtest harness supports running benchmarks for functions annotated with +the `#[bench]` attribute. Benchmarks are currently unstable, and only +available on the [nightly channel]. More information may be found in the +[unstable book][bench-docs]. + +## Custom test frameworks + +Experimental support for using custom test harnesses is available on the +[nightly channel]. See [tracking issue +#50297](https://github.com/rust-lang/rust/issues/50297) and the +[custom_test_frameworks documentation] for more information. + +[`--test` option]: ../command-line-arguments.md#option-test +[`-Z panic-abort-tests`]: https://github.com/rust-lang/rust/issues/67650 +[`available_concurrency`]: ../../std/thread/fn.available_concurrency.html +[`cargo test`]: ../../cargo/commands/cargo-test.html +[`libtest`]: ../../test/index.html +[`main` function]: ../../reference/crates-and-source-files.html#main-functions +[`Result`]: ../../std/result/index.html +[`test` cfg option]: ../../reference/conditional-compilation.html#test +[attribute-ignore]: ../../reference/attributes/testing.html#the-ignore-attribute +[attribute-should_panic]: ../../reference/attributes/testing.html#the-should_panic-attribute +[attribute-test]: ../../reference/attributes/testing.html#the-test-attribute +[bench-docs]: ../../unstable-book/library-features/test.html +[Cargo]: ../../cargo/index.html +[crate type]: ../../reference/linkage.html +[custom_test_frameworks documentation]: ../../unstable-book/language-features/custom-test-frameworks.html +[nightly channel]: ../../book/appendix-07-nightly-rust.html +[panic-strategy]: ../../book/ch09-01-unrecoverable-errors-with-panic.html +[panic]: ../../book/ch09-01-unrecoverable-errors-with-panic.html +[Testing Chapter]: ../../book/ch11-00-testing.html From 7d99d6d10b0246032d613eb9d9c6b5bffe6c2ba6 Mon Sep 17 00:00:00 2001 From: Eric Huss Date: Mon, 22 Feb 2021 13:39:15 -0800 Subject: [PATCH 2/2] Update for review comments. --- src/doc/rustc/src/tests/index.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/src/doc/rustc/src/tests/index.md b/src/doc/rustc/src/tests/index.md index 1f914da4fc55e..ec23d4fe0dbd3 100644 --- a/src/doc/rustc/src/tests/index.md +++ b/src/doc/rustc/src/tests/index.md @@ -26,8 +26,9 @@ crate. The `--test` flag will make the following changes: * Links the executable with [`libtest`], the test harness that is part of the standard library, which handles running the tests. * Synthesizes a [`main` function] which will process command-line arguments - and run the tests. If the crate already has a `main` function, it will be - replaced. + and run the tests. This new `main` function will replace any existing `main` + function as the entry point of the executable, though the existing `main` + will still be compiled. * Enables the [`test` cfg option], which allows your code to use conditional compilation to detect if it is being built as a test. * Enables building of functions annotated with the [`test`][attribute-test] @@ -83,11 +84,11 @@ behavior. ### Filters -Bare arguments (those without a `-` prefix) are treated as filters which will -only run tests whose name matches one of those strings. The filter will match -any substring found in the full path of the test function. For example, if the -test function `it_works` is located in the module `utils::paths::tests`, then -any of the filters `works`, `path`, `utils::`, or +Positional arguments (those without a `-` prefix) are treated as filters which +will only run tests whose name matches one of those strings. The filter will +match any substring found in the full path of the test function. For example, +if the test function `it_works` is located in the module +`utils::paths::tests`, then any of the filters `works`, `path`, `utils::`, or `utils::paths::tests::it_works` will match that test. See [Selection options](#selection-options) for more options to control which @@ -230,7 +231,7 @@ Controls the format of the output. Valid options: #### `--logfile` _PATH_ -Writes the results to the tests to the given file. +Writes the results of the tests to the given file. #### `--report-time` _FORMAT_ @@ -244,7 +245,7 @@ docs](../../unstable-book/compiler-flags/report-time.html) for more information. Some CLI options are added in an "unstable" state, where they are intended for experimentation and testing to determine if the option works correctly, has the right design, and is useful. The option may not work correctly, break, or -change at at any time. To signal that you acknowledge that you are using an +change at any time. To signal that you acknowledge that you are using an unstable option, they require passing the `-Z unstable-options` command-line flag.