WIP for Issue 266 - automating TC-IDM-2.2 #34003
Open
+671
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
PR #34003: Size comparison from e22266b to 813570f Full report (85 builds for bl602, bl702, bl702l, cc13x4_26x4, cc32xx, cyw30739, efr32, esp32, linux, mbed, nrfconnect, nxp, psoc6, qpg, stm32, telink, tizen)
|
|
PR #34003: Size comparison from f6ac926 to bda833c Full report (85 builds for bl602, bl702, bl702l, cc13x4_26x4, cc32xx, cyw30739, efr32, esp32, linux, mbed, nrfconnect, nxp, psoc6, qpg, stm32, telink, tizen)
|
|
PR #34003: Size comparison from f6ac926 to cbfc258 Increases above 0.2%:
Full report (11 builds for cc32xx, mbed, nrfconnect, qpg, stm32, tizen)
|
|
PR #34003: Size comparison from f6ac926 to 353ca21 Increases above 0.2%:
Full report (6 builds for cc32xx, mbed, stm32, tizen)
|
|
PR #34003: Size comparison from f6ac926 to 41d5607 Increases above 0.2%:
Full report (85 builds for bl602, bl702, bl702l, cc13x4_26x4, cc32xx, cyw30739, efr32, esp32, linux, mbed, nrfconnect, nxp, psoc6, qpg, stm32, telink, tizen)
|
andy31415
changed the title
cecille
reviewed
Jun 27, 2024
|
PR #34003: Size comparison from 8ebe19d to 79fe72a Full report (85 builds for bl602, bl702, bl702l, cc13x4_26x4, cc32xx, cyw30739, efr32, esp32, linux, mbed, nrfconnect, nxp, psoc6, qpg, stm32, telink, tizen)
|
|
PR #34003: Size comparison from 171843d to 2a70fbb Full report (20 builds for cc13x4_26x4, cc32xx, mbed, nrfconnect, nxp, qpg, stm32, tizen)
|
|
PR #34003: Size comparison from 171843d to d026e8d Full report (30 builds for bl602, bl702, bl702l, cc13x4_26x4, cc32xx, mbed, nrfconnect, nxp, qpg, stm32, tizen)
|
|
PR #34003: Size comparison from 171843d to b96daaf Full report (11 builds for cc32xx, mbed, nrfconnect, qpg, stm32, tizen)
|
|
PR #34003: Size comparison from 171843d to e31f858 Full report (85 builds for bl602, bl702, bl702l, cc13x4_26x4, cc32xx, cyw30739, efr32, esp32, linux, mbed, nrfconnect, nxp, psoc6, qpg, stm32, telink, tizen)
|
|
PR #34003: Size comparison from f83d67b to 9dcb4b2 Full report (85 builds for bl602, bl702, bl702l, cc13x4_26x4, cc32xx, cyw30739, efr32, esp32, linux, mbed, nrfconnect, nxp, psoc6, qpg, stm32, telink, tizen)
|
pullapprove
bot
requested review from
andy31415,
andyg-apple,
anush-apple,
arkq,
axelnxp,
bauerschwan,
bzbarsky-apple,
carol-apple,
chapongatien,
chrisdecenzo,
chshu,
chulspro,
cliffamzn,
Damian-Nordic and
dhrishi
|
PR #34003: Size comparison from e402b96 to 4fdf004 Full report (3 builds for cc32xx, stm32)
|
…nto issue_226_temp_2
…nto issue_226_temp_2
…nto issue_226_temp_2
|
PR #34003: Size comparison from 73cc4f1 to 53d222c Full report (69 builds for bl602, bl702, bl702l, cc13x4_26x4, cc32xx, cyw30739, efr32, esp32, linux, nrfconnect, nxp, psoc6, qpg, stm32, telink, tizen)
|
cecille
reviewed
Dec 12, 2024
src/python_testing/TC_IDM_2_2.py
Outdated
| # Get all clusters from device | ||
| for cluster in self.device_clusters: | ||
| all_types = await self.all_type_attributes_for_cluster(cluster, desired_attribute_type) | ||
| all_types = list(set(all_types) & self.device_attributes) |
There was a problem hiding this comment.
This isn't going to get you what you want - this list is just a list of all the device attribute ids for every cluster all smooshed together. This needs to be across clusters.
Ex. limiting this to just one endpoint for now
say you have this
Cluster 1 defines attributes 1, 2 and 3. The device has 2 and 3.
Cluster 2 defines attributes 1, 2, 3 and 4. The device supports 1 and 4
Per the code in all_device_attributes, self.device_attributes will contain {1,2,3,4}. But your dut doesn't support attribute 1 for cluster 1, and that's would be selected here.
src/python_testing/TC_IDM_2_2.py
Outdated
| self.print_step(1, "Send Request Message to read one attribute on a given cluster and endpoint") | ||
| read_request = await self.default_controller.ReadAttribute(self.dut_node_id, [(0, Clusters.Objects.Descriptor.Attributes.ServerList)]) | ||
|
|
||
| asserts.assert_in(Clusters.Objects.Descriptor, read_request[0].keys(), "Descriptor cluster not in output") |
There was a problem hiding this comment.
below you use assert_equal, which makes sense to use here too.
src/python_testing/TC_IDM_2_2.py
Outdated
|
|
||
| for endpoint in read_request: | ||
| for cluster in read_request[endpoint]: | ||
| # Endpoint 1 seems an issue with ModeSelect (ServerList has an extra 4293984257) |
There was a problem hiding this comment.
This needs changes, but I think the easier solution is to just omit this step because full wildcard is properly and thoroughly checked in 10.1
There was a problem hiding this comment.
Commented for now; might remove completely
src/python_testing/TC_IDM_2_2.py
Outdated
|
|
||
| for endpoint in read_request: | ||
| for cluster in read_request[endpoint]: | ||
| # Endpoint 1 seems an issue with ModeSelect (ServerList has an extra 4293984257) |
There was a problem hiding this comment.
Is this a problem with mode select, or with the unit testing cluster?
There was a problem hiding this comment.
When I had an assert_equal statement that looked like this, without filtering out ModeSelect:
asserts.assert_equal(returned_attrs, attr_list, f"Mismatch for {cluster} at endpoint {endpoint}")
...with the following values:
returned_attrs = [0, 1, 2, 3, 4, 5, 65528, 65529, 65531, 65532, 65533]
attr_list = [0, 1, 2, 3, 4, 5, 65528, 65529, 65531, 65532, 65533, 4293984257]
...I got this assertion error:
Second list contains 1 additional elements.
First extra element 11:
4293984257
- [0, 1, 2, 3, 4, 5, 65528, 65529, 65531, 65532, 65533]
+ [0, 1, 2, 3, 4, 5, 65528, 65529, 65531, 65532, 65533, 4293984257]
? ++++++++++++
…nto issue_226_temp_2
…nto issue_226_temp_2
|
PR #34003: Size comparison from cfdaf79 to 738388a Full report (69 builds for bl602, bl702, bl702l, cc13x4_26x4, cc32xx, cyw30739, efr32, esp32, linux, nrfconnect, nxp, psoc6, qpg, stm32, telink, tizen)
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Working on matter-test-scripts/#266 Specifically the first 8 steps (use known mandatory endpoints, clusters and attributes. Suggest Endpoint 0 (root node), descriptor cluster, any mandatory attribute)
Got several combinations of reading attributes, clusters, and endpoints, and capturing the output. This is very much a work in progress so the testing is subject to change!