Tests yaml

[vivien-apple]

Problem

In order to make it easier to create Cluster tests that may exercise the different high level layers of the controller, this PR introduce some tests written in yaml to make them easier to read and understand.

Those are basic tests, one for the Test Cluster in order support adding exotic commands to it and see if they work, the other is some test against the OnOff Cluster in order to validate that sending commands update the attribute appropriately.

Summary of Changes

• Add a TestHelper.js in order to get ZAP to understand those YAML templates
• Add onoff.yaml and test.yaml into a new folder src/app/tests/yaml/
• Add an example using partials about how those yaml files can be used to generate some tests for the given framework
• Generate some tests for the Darwin CI task

[woody-apple]

Thanks @vivien-apple. Agree with @andy31415 's feedback, and is this PR planning to have C++ chip-tool included as well? How about python?

[vivien-apple]

Thanks @vivien-apple. Agree with @andy31415 's feedback, and is this PR planning to have C++ chip-tool included as well? How about python?

I have added chip-tool C++ generation as well. So the same test description is generating tests for the Darwin CI, as well as tests for chip-tool.

Under chip-tool the list of tests can be retrieved with ./out/debug/standalone/chip-tool tests

  +-------------------------------------------------------------------------------------+
  | Commands:                                                                           |
  +-------------------------------------------------------------------------------------+
  | * TestCluster                                                                       |
  | * OnOffCluster                                                                      |
  +-------------------------------------------------------------------------------------+

Those tests can then be runned by adding the test name at the end, e.g chip-tool tests OnOffCluster.

For python support it sounds like the way to add them will be update mobile-device-test.py to have a custom template to generate tests as well. For example https://github.com/project-chip/connectedhomeip/blob/master/src/controller/python/test/test_scripts/mobile-device-test.py#L69 calls hand written tests for the OnOff cluster. Someone would know python better than me should be able to replace the hand written version by the generated version. I would like to not block this PR on python support though as python support can come next.

[woody-apple]

Thanks @vivien-apple !

[andy31415]

@vivien-apple - since this does code generation, will we need to check-in more generated code or is this for manual runs only (and generate manually as needed)?

I would find checking in test sequences/yaml code useful however historically checking in generated code from zap causes very large PRs that tend to hide real code changes between auto-generated code, making them hard to review.

What is the expected flow when using yaml test cases? Can we have a model where we have only yaml files and check in generated files?

[andy31415]

Do we need zap or if we rely only on yaml files could we generate in pure python (easier to integrate in the build system if pure python)?

Could we convert the tests to generic POSIX? We have many developers who only have a linux machine and github charges 10x minutes when using darwin instead of linux backends.

[woody-apple]

The plan is to convert these to POSIX as soon as possible, though I think we should get this in... this will help substantially test the C++ api here.

[woody-apple]

@saurabhst @Damian-Nordic @msandstedt ?

[woody-apple]

Put up this to bring it into CI:
#7381