Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs: move CI python testing args into CI section #34264

Merged
merged 1 commit into from
Jul 10, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
114 changes: 62 additions & 52 deletions docs/testing/python.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,58 +85,8 @@ The default_matter_test_main() function is used to run the test on the command
line. These two lines should appear verbatim at the bottom of every python test
file.

## Defining the test arguments

Below is the format:

```
# test-runner-runs: <run_identifier>
# test-runner-run/<run_identifier>/app: ${TYPE_OF_APP}
# test-runner-run/<run_identifier>/factoryreset: <True|False>
# test-runner-run/<run_identifier>/quiet: <True|False>
# test-runner-run/<run_identifier>/app-args: <app_arguments>
# test-runner-run/<run_identifier>/script-args: <script_arguments>
```

### Description of Parameters

- test-runner-runs: Specifies the identifier for the run. This can be any
unique identifier.

- Example: run1

- test-runner-run/<run_identifier>/app: Indicates the application to be used
in the test. Different app types as needed could be referenced from section
[name: Generate an argument environment file ] of the file
[.github/workflows/tests.yaml](https://github.com/project-chip/connectedhomeip/blob/master/.github/workflows/tests.yaml)

- Example: \${TYPE_OF_APP}

- test-runner-run/<run_identifier>/factoryreset: Determines whether a factory
reset should be performed before the test.

- Example: True

- test-runner-run/<run_identifier>/quiet: Sets the verbosity level of the test
run. When set to True, the test run will be quieter.

- Example: True

- test-runner-run/<run_identifier>/app-args: Specifies the arguments to be
passed to the application during the test.

- Example: --discriminator 1234 --KVS kvs1 --trace-to
json:\${TRACE_APP}.json

- test-runner-run/<run_identifier>/script-args: Specifies the arguments to be
passed to the test script.
- Example: --storage-path admin_storage.json --commissioning-method
on-network --discriminator 1234 --passcode 20202021 --trace-to
json:${TRACE_TEST_JSON}.json --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto

This structured format ensures that all necessary configurations are clearly
defined and easily understood, allowing for consistent and reliable test
execution.
The structured comments above the class definition are used to set up the CI for
the tests. Please see [Running tests in CI](#running-tests-in-ci).

## Cluster Codegen

Expand Down Expand Up @@ -610,3 +560,63 @@ example DUT on the host and includes factory reset support
- if there are things in your test that will fail on CI (ex. test vendor
checks), gate them on the PICS_SDK_CI_ONLY
- is_ci = self.check_pics('PICS_SDK_CI_ONLY')

The CI test runner uses a structured environment setup that can be declared
using structured comments at the top of the test file. To use this structured
format, use the --load-from-env flag with the run_python_tests.py runner.

Ex:
`scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_ICDM_2_1.py'`

## Defining the CI test arguments

Below is the format:

```
# test-runner-runs: <run_identifier>
# test-runner-run/<run_identifier>/app: ${TYPE_OF_APP}
# test-runner-run/<run_identifier>/factoryreset: <True|False>
# test-runner-run/<run_identifier>/quiet: <True|False>
# test-runner-run/<run_identifier>/app-args: <app_arguments>
# test-runner-run/<run_identifier>/script-args: <script_arguments>
```

### Description of Parameters

- test-runner-runs: Specifies the identifier for the run. This can be any
unique identifier.

- Example: run1

- test-runner-run/<run_identifier>/app: Indicates the application to be used
in the test. Different app types as needed could be referenced from section
[name: Generate an argument environment file ] of the file
[.github/workflows/tests.yaml](https://github.com/project-chip/connectedhomeip/blob/master/.github/workflows/tests.yaml)

- Example: \${TYPE_OF_APP}

- test-runner-run/<run_identifier>/factoryreset: Determines whether a factory
reset should be performed before the test.

- Example: True

- test-runner-run/<run_identifier>/quiet: Sets the verbosity level of the test
run. When set to True, the test run will be quieter.

- Example: True

- test-runner-run/<run_identifier>/app-args: Specifies the arguments to be
passed to the application during the test.

- Example: --discriminator 1234 --KVS kvs1 --trace-to
json:\${TRACE_APP}.json

- test-runner-run/<run_identifier>/script-args: Specifies the arguments to be
passed to the test script.
- Example: --storage-path admin_storage.json --commissioning-method
on-network --discriminator 1234 --passcode 20202021 --trace-to
json:${TRACE_TEST_JSON}.json --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto

This structured format ensures that all necessary configurations are clearly
defined and easily understood, allowing for consistent and reliable test
execution.
Loading