cosmos · julienrbrt · Oct 2, 2024 · Oct 2, 2024
@@ -0,0 +1,58 @@
+---
+sidebar_position: 1
+---
+
+# System Tests
+
+System tests provide a framework to write and execute black box tests against a running chain. This adds another level
+of confidence on top of unit, integration, and simulations tests, ensuring that business-critical scenarios
+(like double signing prevention) or scenarios that can't be tested otherwise (like a chain upgrade) are covered.
+
+## Vanilla Go for Flow Control
+
+System tests are vanilla Go tests that interact with the compiled chain binary. The `test runner` component starts a
+local testnet of 4 nodes (by default) and provides convenient helper methods for accessing the
+`system under test (SUT)`.
+A `CLI wrapper` makes it easy to access keys, submit transactions, or execute operations. Together, these components
+enable the replication and validation of complex business scenarios.
+
+Here's an example of a double signing test, where a new node is added with the same key as the first validator:
+[double signing test example](https://github.com/cosmos/cosmos-sdk/blob/v0.52.0-beta.1/tests/systemtests/fraud_test.go)
+
+The [getting started tutorial](https://github.com/cosmos/cosmos-sdk/blob/v0.52.0-beta.1/tests/systemtests/getting_started.md)
+contains a step-by-step guide to building and running your first system test. It covers setting chain state via genesis
+or
+transactions and validation via transaction response or queries.
+
+## Design Principles and Guidelines
+
+System tests are slower compared to unit or integration tests as they interact with a running chain. Therefore, certain
+principles can guide their usage:
+
+- **Perspective:** Tests should mimic a human interacting with the chain from the outside. Initial states can be set via
+  genesis or transactions to support a test scenario.
+- **Roles:** The user can have multiple roles such as validator, delegator, granter, or group admin.
+- **Focus:** Tests should concentrate on happy paths or business-critical workflows. Unit and integration tests are
+  better suited for more fine-grained testing.
+- **Workflows:** Test workflows and scenarios, not individual units. Given the high setup costs, it is reasonable to
+  combine multiple steps and assertions in a single test method.
+- **Genesis Mods:** Genesis modifications can incur additional time costs for resetting dirty states. Reuse existing
+  accounts (node0..n) whenever possible.
+- **Framework:** Continuously improve the framework for better readability and reusability.
+
+## Errors and Debugging
+
+All output is logged to `systemtests/testnet/node{0..n}.out`. Usually, `node0.out` is very noisy as it receives the CLI
+connections. Prefer any other node's log to find stack traces or error messages.
+
+Using system tests for state setup during debugging has become very handy:
+
+- Start the test with one node only and verbose output:
+
+  ```sh
+  go test -v -tags=system_test ./ --run TestAccountCreation --verbose --nodes-count=1
+  ```
+
+- Copy the CLI command for the transaction and modify the test to stop before the command
+- Start the node with `--home=<project-home>/tests/systemtests/testnet/node0/<binary-name>/` in debug mode
+- Execute CLI command from shell and enter breakpoints
@@ -86,38 +86,26 @@ https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/tests/integration/bank
 
 ## Simulations
 
-Simulations uses as well a minimal application, built with [`depinject`](../packages/01-depinject.md):
+Simulations fuzz tests for deterministic message execution. They use a minimal application, built with [`depinject`](../packages/01-depinject.md):
 
 :::note
-You can as well use the `AppConfig` `configurator` for creating an `AppConfig` [inline](https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/x/slashing/app_test.go#L54-L62). There is no difference between those two ways, use whichever you prefer.
+Simulations have been refactored to message factories
 :::
 
-Following is an example for `x/gov/` simulations:
+An example for `x/bank/` simulations:
 
 ```go reference
-https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/x/gov/simulation/operations_test.go#L406-L430
+https://github.com/cosmos/cosmos-sdk/blob/release/v0.52.x/x/bank/simulation/msg_factory.go#L13-L20
 ```
 
-```go reference
-https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/x/gov/simulation/operations_test.go#L90-L132
-```
-
-## End-to-end Tests
+## System Tests
 
-End-to-end tests are at the top of the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html).
-They must test the whole application flow, from the user perspective (for instance, CLI tests). They are located under [`/tests/e2e`](https://github.com/cosmos/cosmos-sdk/tree/main/tests/e2e).
+System tests are at the top of the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html).
+They test the whole application flow as black box, from the user perspective. They are located under [`/tests/systemtests`](https://github.com/cosmos/cosmos-sdk/tree/main/tests/systemtests).
 
-<!-- @julienrbrt: makes more sense to use an app wired app to have 0 simapp dependencies -->
-For that, the SDK is using `simapp` but you should use your own application (`appd`).
-Here are some examples:
+For that, the SDK is using the `simapp` binary, but you should use your own binary.
+More details about system test can be found in [building-apps](https://docs.cosmos.network/main/build/building-apps/06-system-tests.md)
 
-* SDK E2E tests: <https://github.com/cosmos/cosmos-sdk/tree/main/tests/e2e>.
-* Cosmos Hub E2E tests: <https://github.com/cosmos/gaia/tree/main/tests/e2e>.
-* Osmosis E2E tests: <https://github.com/osmosis-labs/osmosis/tree/main/tests/e2e>.
-
-:::note warning
-The SDK is in the process of creating its E2E tests, as defined in [ADR-59](https://docs.cosmos.network/main/build/architecture/adr-059-test-scopes). This page will eventually be updated with better examples.
-:::
 
 ## Learn More