Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement TC-RR-1.1 #21736

Closed
tcarmelveilleux opened this issue Aug 8, 2022 · 0 comments · Fixed by #22292
Closed

Implement TC-RR-1.1 #21736

tcarmelveilleux opened this issue Aug 8, 2022 · 0 comments · Fixed by #22292
Assignees
@tcarmelveilleux
Labels
automation Tasks that can be automated cert needs certification test python testing tests

Comments

@tcarmelveilleux
Copy link
Contributor

Problem

TC-RR-1.1 (https://github.com/CHIP-Specifications/chip-test-plans/blob/master/src/minimal_resource_reqs.adoc#tc-rr-1-1-minimal-resource-requirements-for-matter-node) is a resources minima test that exercises a large number facets too difficult to manually orchestrate with chip-tool.

Proposed Solution

The test needs to be implemented with Python testing support
@tcarmelveilleux tcarmelveilleux self-assigned this Aug 8, 2022
@andy31415 andy31415 mentioned this issue Aug 18, 2022
Closed
tcarmelveilleux added a commit to tcarmelveilleux/connectedhomeip that referenced this issue Aug 19, 2022
@tcarmelveilleux
Introduce initial TC-RR-1.1
d6f04fb 
- TC-RR-1.1 is a critical test to validate multi-fabric
  behavior is stable and actually works. The test, broadly,
  validates most of the minimas of the core elements of the spec,
  including ACL entries, certificate sizes, number of CASE
  sessions and subscriptions, number of paths, etc.

Issue project-chip#21736

- This PR introduces the core test and all associated minor
  changes to infrastructure to make it work.
- Still TODO:
  - More extensive cert size maximization (closer to 400 TLV bytes)
  - Add controller and commissionee CAT tags (test is 95% equivalent
    to test plan, but a couple ACL fields differ because of this, in
    ways that don't detract from proving what needs proving
  - Validation that local/peer session IDs have not changed. This is
    not technically needed with the SDK as-is based on the methodology
    but it would future-proof the test against some future optimizations
    that may change subscription behavior in a way that the test would
    not validate CASE sessions remain.
  - Clean-up more after the test, so that a factory reset before/after
    is not needed.

Testing done:
- Passes on Linux against all-clusters, all-clusters-minimal and
  lighting app, with both minimal mdns and Avahi.
- Passes on some other platforms (not named here)
tcarmelveilleux added a commit to tcarmelveilleux/connectedhomeip that referenced this issue Aug 19, 2022
@tcarmelveilleux
Introduce initial TC-RR-1.1
1460e73 
- TC-RR-1.1 is a critical test to validate multi-fabric
  behavior is stable and actually works. The test, broadly,
  validates most of the minimas of the core elements of the spec,
  including ACL entries, certificate sizes, number of CASE
  sessions and subscriptions, number of paths, etc.

Issue project-chip#21736

- This PR introduces the core test and all associated minor
  changes to infrastructure to make it work.
- Still TODO:
  - More extensive cert size maximization (closer to 400 TLV bytes)
  - Add controller and commissionee CAT tags (test is 95% equivalent
    to test plan, but a couple ACL fields differ because of this, in
    ways that don't detract from proving what needs proving
  - Validation that local/peer session IDs have not changed. This is
    not technically needed with the SDK as-is based on the methodology
    but it would future-proof the test against some future optimizations
    that may change subscription behavior in a way that the test would
    not validate CASE sessions remain.
  - Clean-up more after the test, so that a factory reset before/after
    is not needed.

Testing done:
- Passes on Linux against all-clusters, all-clusters-minimal and
  lighting app, with both minimal mdns and Avahi.
- Passes on some other platforms (not named here)

To run within SDK (from scratch: the build steps can be skipped thereafter):

- In one terminal:
  - Build chip-lighting-app linux
  - `clear && rm -f kvs1 && out/debug/standalone/chip-lighting-app --discriminator 1234 --KVS kvs1 --trace_decode 1`

- In another terminal:
  - Build
    - `rm -rf out/python*`
    - `scripts/build_python.sh -m platform -i separate`
  - Run
    - `source ./out/python_env/bin/activate`
    - `python3 src/python_testing/TC_RR_1_1.py --commissioning-method on-network --long-discriminator 1234 --passcode 20202021`
      - Add `--bool-arg skip_user_label_cluster_steps:true` to the end of the command line
        if your DUT has broken UserLabel clusters (but if you have those, fix them :)
@tcarmelveilleux tcarmelveilleux mentioned this issue Aug 19, 2022
Merged
@andy31415 andy31415 mentioned this issue Aug 19, 2022
Closed
tcarmelveilleux added a commit that referenced this issue Aug 19, 2022
@tcarmelveilleux @mrjerryjohns
Introduce initial TC-RR-1.1 (#22032)
dfc1a70 
* Introduce initial TC-RR-1.1

- TC-RR-1.1 is a critical test to validate multi-fabric
  behavior is stable and actually works. The test, broadly,
  validates most of the minimas of the core elements of the spec,
  including ACL entries, certificate sizes, number of CASE
  sessions and subscriptions, number of paths, etc.

Issue #21736

- This PR introduces the core test and all associated minor
  changes to infrastructure to make it work.
- Still TODO:
  - More extensive cert size maximization (closer to 400 TLV bytes)
  - Add controller and commissionee CAT tags (test is 95% equivalent
    to test plan, but a couple ACL fields differ because of this, in
    ways that don't detract from proving what needs proving
  - Validation that local/peer session IDs have not changed. This is
    not technically needed with the SDK as-is based on the methodology
    but it would future-proof the test against some future optimizations
    that may change subscription behavior in a way that the test would
    not validate CASE sessions remain.
  - Clean-up more after the test, so that a factory reset before/after
    is not needed.

Testing done:
- Passes on Linux against all-clusters, all-clusters-minimal and
  lighting app, with both minimal mdns and Avahi.
- Passes on some other platforms (not named here)

To run within SDK (from scratch: the build steps can be skipped thereafter):

- In one terminal:
  - Build chip-lighting-app linux
  - `clear && rm -f kvs1 && out/debug/standalone/chip-lighting-app --discriminator 1234 --KVS kvs1 --trace_decode 1`

- In another terminal:
  - Build
    - `rm -rf out/python*`
    - `scripts/build_python.sh -m platform -i separate`
  - Run
    - `source ./out/python_env/bin/activate`
    - `python3 src/python_testing/TC_RR_1_1.py --commissioning-method on-network --long-discriminator 1234 --passcode 20202021`
      - Add `--bool-arg skip_user_label_cluster_steps:true` to the end of the command line
        if your DUT has broken UserLabel clusters (but if you have those, fix them :)

* More work towards CAT tags

* Address review comments

* Fixed CAT tag testing

* Update src/controller/python/chip/utils/CommissioningBuildingBlocks.py

Co-authored-by: Jerry Johns <[email protected]>
tcarmelveilleux added a commit to tcarmelveilleux/connectedhomeip that referenced this issue Aug 22, 2022
@tcarmelveilleux @mrjerryjohns
Introduce initial TC-RR-1.1 (project-chip#22032)
cb23aaa 
* Introduce initial TC-RR-1.1

- TC-RR-1.1 is a critical test to validate multi-fabric
  behavior is stable and actually works. The test, broadly,
  validates most of the minimas of the core elements of the spec,
  including ACL entries, certificate sizes, number of CASE
  sessions and subscriptions, number of paths, etc.

Issue project-chip#21736

- This PR introduces the core test and all associated minor
  changes to infrastructure to make it work.
- Still TODO:
  - More extensive cert size maximization (closer to 400 TLV bytes)
  - Add controller and commissionee CAT tags (test is 95% equivalent
    to test plan, but a couple ACL fields differ because of this, in
    ways that don't detract from proving what needs proving
  - Validation that local/peer session IDs have not changed. This is
    not technically needed with the SDK as-is based on the methodology
    but it would future-proof the test against some future optimizations
    that may change subscription behavior in a way that the test would
    not validate CASE sessions remain.
  - Clean-up more after the test, so that a factory reset before/after
    is not needed.

Testing done:
- Passes on Linux against all-clusters, all-clusters-minimal and
  lighting app, with both minimal mdns and Avahi.
- Passes on some other platforms (not named here)

To run within SDK (from scratch: the build steps can be skipped thereafter):

- In one terminal:
  - Build chip-lighting-app linux
  - `clear && rm -f kvs1 && out/debug/standalone/chip-lighting-app --discriminator 1234 --KVS kvs1 --trace_decode 1`

- In another terminal:
  - Build
    - `rm -rf out/python*`
    - `scripts/build_python.sh -m platform -i separate`
  - Run
    - `source ./out/python_env/bin/activate`
    - `python3 src/python_testing/TC_RR_1_1.py --commissioning-method on-network --long-discriminator 1234 --passcode 20202021`
      - Add `--bool-arg skip_user_label_cluster_steps:true` to the end of the command line
        if your DUT has broken UserLabel clusters (but if you have those, fix them :)

* More work towards CAT tags

* Address review comments

* Fixed CAT tag testing

* Update src/controller/python/chip/utils/CommissioningBuildingBlocks.py

Co-authored-by: Jerry Johns <[email protected]>
@tcarmelveilleux tcarmelveilleux mentioned this issue Aug 22, 2022
Closed
tcarmelveilleux added a commit to tcarmelveilleux/connectedhomeip that referenced this issue Aug 31, 2022
@tcarmelveilleux
Re-enable TC-RR-1.1 in CI
3929384 
- A regression had occured on RR-1.1 in master before CI was attempted
  in project-chip#22143 which prevented the CI test to be enabled

This PR:

- Fixes TC-RR-1.1
- Enables TC-RR-1.1, TC-SC-3.6 and TC-DA-1.7 to keep them fresh and
  improve integration test coverage (TC-RR-1.1 tests a lot)
- Improve run_python_test.py to do better factory reset

Fixes project-chip#21736

Testing done:
- Manually tested the fix of TC-RR-1.1 on ESP32 and Linux
- CI tests of the above is automated
@tcarmelveilleux tcarmelveilleux mentioned this issue Aug 31, 2022
Merged
tcarmelveilleux added a commit that referenced this issue Sep 1, 2022
@tcarmelveilleux
Re-enable TC-RR-1.1 in CI (#22292)
7894428 
* Re-enable TC-RR-1.1 in CI

- A regression had occured on RR-1.1 in master before CI was attempted
  in #22143 which prevented the CI test to be enabled

This PR:

- Fixes TC-RR-1.1
- Enables TC-RR-1.1, TC-SC-3.6 and TC-DA-1.7 to keep them fresh and
  improve integration test coverage (TC-RR-1.1 tests a lot)
- Improve run_python_test.py to do better factory reset

Fixes #21736

Testing done:
- Manually tested the fix of TC-RR-1.1 on ESP32 and Linux
- CI tests of the above is automated

* Attempt to pass CI

* Debug CI

* More debug of CI

* More CI fixes

* Fix more CI
isiu-apple pushed a commit to isiu-apple/connectedhomeip that referenced this issue Sep 16, 2022
@tcarmelveilleux @mrjerryjohns @isiu-apple
Introduce initial TC-RR-1.1 (project-chip#22032)
e9ebe28 
* Introduce initial TC-RR-1.1

- TC-RR-1.1 is a critical test to validate multi-fabric
  behavior is stable and actually works. The test, broadly,
  validates most of the minimas of the core elements of the spec,
  including ACL entries, certificate sizes, number of CASE
  sessions and subscriptions, number of paths, etc.

Issue project-chip#21736

- This PR introduces the core test and all associated minor
  changes to infrastructure to make it work.
- Still TODO:
  - More extensive cert size maximization (closer to 400 TLV bytes)
  - Add controller and commissionee CAT tags (test is 95% equivalent
    to test plan, but a couple ACL fields differ because of this, in
    ways that don't detract from proving what needs proving
  - Validation that local/peer session IDs have not changed. This is
    not technically needed with the SDK as-is based on the methodology
    but it would future-proof the test against some future optimizations
    that may change subscription behavior in a way that the test would
    not validate CASE sessions remain.
  - Clean-up more after the test, so that a factory reset before/after
    is not needed.

Testing done:
- Passes on Linux against all-clusters, all-clusters-minimal and
  lighting app, with both minimal mdns and Avahi.
- Passes on some other platforms (not named here)

To run within SDK (from scratch: the build steps can be skipped thereafter):

- In one terminal:
  - Build chip-lighting-app linux
  - `clear && rm -f kvs1 && out/debug/standalone/chip-lighting-app --discriminator 1234 --KVS kvs1 --trace_decode 1`

- In another terminal:
  - Build
    - `rm -rf out/python*`
    - `scripts/build_python.sh -m platform -i separate`
  - Run
    - `source ./out/python_env/bin/activate`
    - `python3 src/python_testing/TC_RR_1_1.py --commissioning-method on-network --long-discriminator 1234 --passcode 20202021`
      - Add `--bool-arg skip_user_label_cluster_steps:true` to the end of the command line
        if your DUT has broken UserLabel clusters (but if you have those, fix them :)

* More work towards CAT tags

* Address review comments

* Fixed CAT tag testing

* Update src/controller/python/chip/utils/CommissioningBuildingBlocks.py

Co-authored-by: Jerry Johns <[email protected]>
isiu-apple pushed a commit to isiu-apple/connectedhomeip that referenced this issue Sep 16, 2022
@tcarmelveilleux @isiu-apple
Re-enable TC-RR-1.1 in CI (project-chip#22292)
15c11f5 
* Re-enable TC-RR-1.1 in CI

- A regression had occured on RR-1.1 in master before CI was attempted
  in project-chip#22143 which prevented the CI test to be enabled

This PR:

- Fixes TC-RR-1.1
- Enables TC-RR-1.1, TC-SC-3.6 and TC-DA-1.7 to keep them fresh and
  improve integration test coverage (TC-RR-1.1 tests a lot)
- Improve run_python_test.py to do better factory reset

Fixes project-chip#21736

Testing done:
- Manually tested the fix of TC-RR-1.1 on ESP32 and Linux
- CI tests of the above is automated

* Attempt to pass CI

* Debug CI

* More debug of CI

* More CI fixes

* Fix more CI
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tcarmelveilleux tcarmelveilleux

Labels
automation Tasks that can be automated cert needs certification test python testing tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Re-enable TC-RR-1.1 in CI tcarmelveilleux/connectedhomeip
1 participant
@tcarmelveilleux