[matter_yaml] Add a yaml test that exercise anyOf, EqualityCommands and runIf #26567
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vivien-apple
changed the title
pullapprove
bot
requested review from
amitnj,
anush-apple,
arkq,
bzbarsky-apple,
carol-apple,
cecille,
chrisdecenzo,
chshu,
chulspro,
cliffamzn,
Damian-Nordic,
dhrishi,
electrocucaracha,
emargolis,
franck-apple,
gjc13,
harimau-qirex,
harsha-rajendran,
hawk248,
jelderton and
jepenven-silabs
|
PR #26567: Size comparison from a33249d to ec3e876 Increases (18 builds for bl602, bl702, esp32, psoc6, qpg, telink)
Decreases (6 builds for cyw30739, esp32, telink)
Full report (57 builds for bl602, bl702, cc32xx, cyw30739, efr32, esp32, k32w, linux, mbed, nrfconnect, psoc6, qpg, telink)
|
vivien-apple
force-pushed
the
MatterYaml_AddAnyOfEqualityCommandsRunIf
branch
from
ec3e876 to
bac2a2e
Compare
|
PR #26567: Size comparison from a33249d to bac2a2e Increases (12 builds for bl602, bl702, psoc6, telink)
Decreases (5 builds for psoc6, telink)
Full report (57 builds for bl602, bl702, cc32xx, cyw30739, efr32, esp32, k32w, linux, mbed, nrfconnect, psoc6, qpg, telink)
|
|
General question on the state of the yaml - we seem to be quickly making our way towards writing an ad-hoc fully functional language. Is this really what we want to do? Is this actually easier to write and understand than a test written in a standard language? |
vivien-apple
force-pushed
the
MatterYaml_AddAnyOfEqualityCommandsRunIf
branch
from
bac2a2e to
520840a
Compare
|
PR #26567: Size comparison from b4fb147 to 520840a Increases (8 builds for bl602, bl702, cc32xx, psoc6, telink)
Decreases (10 builds for bl702, esp32, psoc6, qpg, telink)
Full report (57 builds for bl602, bl702, cc32xx, cyw30739, efr32, esp32, k32w, linux, mbed, nrfconnect, psoc6, qpg, telink)
|
bzbarsky-apple
approved these changes
May 16, 2023
vivien-apple
force-pushed
the
MatterYaml_AddAnyOfEqualityCommandsRunIf
branch
from
520840a to
4a523ca
Compare
|
PR #26567: Size comparison from 5f1a60b to 4a523ca Increases (12 builds for bl602, cc32xx, cyw30739, nrfconnect, psoc6, telink)
Decreases (9 builds for bl702, esp32, nrfconnect, psoc6, telink)
Full report (57 builds for bl602, bl702, cc32xx, cyw30739, efr32, esp32, k32w, linux, mbed, nrfconnect, psoc6, qpg, telink)
|
Since this particular PR is about adding a mechanism to do a sort of 'if' that is a fair question. | Is this really what we want to do? The answer is obviously no. And, as of today, I have tried to avoid that as much as possible. | Is this actually easier to write and understand than a test written in a standard language? I am not the best one to answer this particular question. I do not have personal strong opinion about that beside the fact that the YAML tests is what is used today by the certification infrastructure and has been written/updated at fast pace by various different people with various different skills and knowledge of the SDK. So to me the question is not really "should we switch/rewrite the tests to something else" but more "what can we bring on to our current tool to enhance the automated coverage". On a more technical side, the current yaml python runner offers the opportunity to add custom "commands" written in python for very specific steps. My current thinking is that we should leverage this supports to support the long tails of unique steps and if we can found some patterns, add dedicated YAML support for the found pattern. It should let us avoid the issue of making our YAML tests format so complex that it ends up as something that can be compared to a programming language (which is not the case today imho). |
lpbeliveau-silabs
approved these changes
May 16, 2023
tianfeng-yang
pushed a commit
to tianfeng-yang/connectedhomeip
that referenced
this pull request
May 17, 2023
…nd runIf (project-chip#26567) * [matter_yamltests] Add anyOf constraint supports * [matter_yamltest] Add EqualityCommands pseudo cluster * [matter_yamltest] Add a runIf keyword to enable/disable a step based on a boolean variable * Add a yaml test that exercise anyOf, EqualityCommands and runIf
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
#26459 (comment) asked if there is a way to condition test step based on variables that have been saved using
saveAs.This PR is an attempt at trying to add the necessary bits to make it possible to fully automate
Test_TC_S_2_2.yamland other tests that may expect different return values for a given step and condition the following steps depending on the effect return value.Concretely it introduce 3 things:
anyOfconstraint that allows multiple values and is fulfilled is one value in the list match the returned valueEqualityClusterthat allow to compute the result of a simple comparaison. The purpose here is to be able to compute a boolean that can be used to enable/disable some other steps.runIfstep keyword which enable/disable the step dynamically based on a boolean variable value.