Introduce initial TC-RR-1.1 #22032
Merged
Introduce initial TC-RR-1.1 #22032
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- TC-RR-1.1 is a critical test to validate multi-fabric behavior is stable and actually works. The test, broadly, validates most of the minimas of the core elements of the spec, including ACL entries, certificate sizes, number of CASE sessions and subscriptions, number of paths, etc. Issue project-chip#21736 - This PR introduces the core test and all associated minor changes to infrastructure to make it work. - Still TODO: - More extensive cert size maximization (closer to 400 TLV bytes) - Add controller and commissionee CAT tags (test is 95% equivalent to test plan, but a couple ACL fields differ because of this, in ways that don't detract from proving what needs proving - Validation that local/peer session IDs have not changed. This is not technically needed with the SDK as-is based on the methodology but it would future-proof the test against some future optimizations that may change subscription behavior in a way that the test would not validate CASE sessions remain. - Clean-up more after the test, so that a factory reset before/after is not needed. Testing done: - Passes on Linux against all-clusters, all-clusters-minimal and lighting app, with both minimal mdns and Avahi. - Passes on some other platforms (not named here) To run within SDK (from scratch: the build steps can be skipped thereafter): - In one terminal: - Build chip-lighting-app linux - `clear && rm -f kvs1 && out/debug/standalone/chip-lighting-app --discriminator 1234 --KVS kvs1 --trace_decode 1` - In another terminal: - Build - `rm -rf out/python*` - `scripts/build_python.sh -m platform -i separate` - Run - `source ./out/python_env/bin/activate` - `python3 src/python_testing/TC_RR_1_1.py --commissioning-method on-network --long-discriminator 1234 --passcode 20202021` - Add `--bool-arg skip_user_label_cluster_steps:true` to the end of the command line if your DUT has broken UserLabel clusters (but if you have those, fix them :)
pullapprove
bot
requested review from
anush-apple,
arkq,
Byungjoo-Lee,
bzbarsky-apple,
carol-apple,
chrisdecenzo,
chshu,
chulspro,
Damian-Nordic,
dhrishi,
electrocucaracha,
emargolis,
erjiaqing,
franck-apple,
gjc13,
harimau-qirex,
harsha-rajendran and
hawk248
pullapprove
bot
requested review from
mrjerryjohns,
msandstedt,
rgoliver,
saurabhst,
selissia,
tecimovic,
turon,
vijs,
vivien-apple,
wbschiller,
woody-apple,
xylophone21 and
yunhanw-google
|
PR #22032: Size comparison from 3f69587 to 9c8c2aa Increases above 0.2%:
Increases (7 builds for cc13x2_26x2, esp32, linux, psoc6)
Decreases (6 builds for bl602, cc13x2_26x2, psoc6, telink)
Full report (43 builds for bl602, cc13x2_26x2, cyw30739, efr32, esp32, k32w, linux, mbed, nrfconnect, psoc6, telink)
|
mrjerryjohns
approved these changes
Aug 19, 2022
|
PR #22032: Size comparison from 3f69587 to 4365ede Increases above 0.2%:
Increases (6 builds for cc13x2_26x2, k32w, linux)
Decreases (5 builds for bl602, cc13x2_26x2, psoc6, telink)
Full report (43 builds for bl602, cc13x2_26x2, cyw30739, efr32, esp32, k32w, linux, mbed, nrfconnect, psoc6, telink)
|
|
PR #22032: Size comparison from 3f69587 to ed87eef Increases above 0.2%:
Increases (10 builds for efr32, linux, psoc6)
Decreases (7 builds for bl602, efr32, nrfconnect, psoc6, telink)
Full report (43 builds for bl602, cc13x2_26x2, cyw30739, efr32, esp32, k32w, linux, mbed, nrfconnect, psoc6, telink)
|
jmartinez-silabs
approved these changes
Aug 19, 2022
ukatira
approved these changes
Aug 19, 2022
tcarmelveilleux
added a commit
to tcarmelveilleux/connectedhomeip
that referenced
this pull request
Aug 22, 2022
* Introduce initial TC-RR-1.1 - TC-RR-1.1 is a critical test to validate multi-fabric behavior is stable and actually works. The test, broadly, validates most of the minimas of the core elements of the spec, including ACL entries, certificate sizes, number of CASE sessions and subscriptions, number of paths, etc. Issue project-chip#21736 - This PR introduces the core test and all associated minor changes to infrastructure to make it work. - Still TODO: - More extensive cert size maximization (closer to 400 TLV bytes) - Add controller and commissionee CAT tags (test is 95% equivalent to test plan, but a couple ACL fields differ because of this, in ways that don't detract from proving what needs proving - Validation that local/peer session IDs have not changed. This is not technically needed with the SDK as-is based on the methodology but it would future-proof the test against some future optimizations that may change subscription behavior in a way that the test would not validate CASE sessions remain. - Clean-up more after the test, so that a factory reset before/after is not needed. Testing done: - Passes on Linux against all-clusters, all-clusters-minimal and lighting app, with both minimal mdns and Avahi. - Passes on some other platforms (not named here) To run within SDK (from scratch: the build steps can be skipped thereafter): - In one terminal: - Build chip-lighting-app linux - `clear && rm -f kvs1 && out/debug/standalone/chip-lighting-app --discriminator 1234 --KVS kvs1 --trace_decode 1` - In another terminal: - Build - `rm -rf out/python*` - `scripts/build_python.sh -m platform -i separate` - Run - `source ./out/python_env/bin/activate` - `python3 src/python_testing/TC_RR_1_1.py --commissioning-method on-network --long-discriminator 1234 --passcode 20202021` - Add `--bool-arg skip_user_label_cluster_steps:true` to the end of the command line if your DUT has broken UserLabel clusters (but if you have those, fix them :) * More work towards CAT tags * Address review comments * Fixed CAT tag testing * Update src/controller/python/chip/utils/CommissioningBuildingBlocks.py Co-authored-by: Jerry Johns <[email protected]>
isiu-apple
pushed a commit
to isiu-apple/connectedhomeip
that referenced
this pull request
Sep 16, 2022
* Introduce initial TC-RR-1.1 - TC-RR-1.1 is a critical test to validate multi-fabric behavior is stable and actually works. The test, broadly, validates most of the minimas of the core elements of the spec, including ACL entries, certificate sizes, number of CASE sessions and subscriptions, number of paths, etc. Issue project-chip#21736 - This PR introduces the core test and all associated minor changes to infrastructure to make it work. - Still TODO: - More extensive cert size maximization (closer to 400 TLV bytes) - Add controller and commissionee CAT tags (test is 95% equivalent to test plan, but a couple ACL fields differ because of this, in ways that don't detract from proving what needs proving - Validation that local/peer session IDs have not changed. This is not technically needed with the SDK as-is based on the methodology but it would future-proof the test against some future optimizations that may change subscription behavior in a way that the test would not validate CASE sessions remain. - Clean-up more after the test, so that a factory reset before/after is not needed. Testing done: - Passes on Linux against all-clusters, all-clusters-minimal and lighting app, with both minimal mdns and Avahi. - Passes on some other platforms (not named here) To run within SDK (from scratch: the build steps can be skipped thereafter): - In one terminal: - Build chip-lighting-app linux - `clear && rm -f kvs1 && out/debug/standalone/chip-lighting-app --discriminator 1234 --KVS kvs1 --trace_decode 1` - In another terminal: - Build - `rm -rf out/python*` - `scripts/build_python.sh -m platform -i separate` - Run - `source ./out/python_env/bin/activate` - `python3 src/python_testing/TC_RR_1_1.py --commissioning-method on-network --long-discriminator 1234 --passcode 20202021` - Add `--bool-arg skip_user_label_cluster_steps:true` to the end of the command line if your DUT has broken UserLabel clusters (but if you have those, fix them :) * More work towards CAT tags * Address review comments * Fixed CAT tag testing * Update src/controller/python/chip/utils/CommissioningBuildingBlocks.py Co-authored-by: Jerry Johns <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
behavior is stable and actually works. The test, broadly,
validates most of the minimas of the core elements of the spec,
including ACL entries, certificate sizes, number of CASE
sessions and subscriptions, number of paths, etc.
Issue #21736
Change overview
changes to infrastructure to make it work.
to test plan, but a couple ACL fields differ because of this, in
ways that don't detract from proving what needs proving
not technically needed with the SDK as-is based on the methodology
but it would future-proof the test against some future optimizations
that may change subscription behavior in a way that the test would
not validate CASE sessions remain.
is not needed.
Testing
lighting app, with both minimal mdns and Avahi.
To run within SDK (from scratch: the build steps can be skipped thereafter):
In one terminal:
clear && rm -f kvs1 && out/debug/standalone/chip-lighting-app --discriminator 1234 --KVS kvs1 --trace_decode 1In another terminal:
rm -rf out/python*scripts/build_python.sh -m platform -i separatesource ./out/python_env/bin/activaterm -f admin_storage.json && python3 src/python_testing/TC_RR_1_1.py --commissioning-method on-network --long-discriminator 1234 --passcode 20202021--bool-arg skip_user_label_cluster_steps:trueto the end of the command lineif your DUT has broken UserLabel clusters (but if you have those, fix them :)