Add a parser for CSA XML data models #29888
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pullapprove
bot
requested review from
younghak-hwang,
yufengwangca and
yunhanw-google
cecille
approved these changes
Oct 20, 2023
scripts/py_matter_idl/matter_idl/data_model_xml/handlers/parsing.py
Outdated
Show resolved
Hide resolved
scripts/py_matter_idl/matter_idl/data_model_xml/handlers/__init__.py
Outdated
Show resolved
Hide resolved
scripts/py_matter_idl/matter_idl/data_model_xml/handlers/base.py
Outdated
Show resolved
Hide resolved
scripts/py_matter_idl/matter_idl/data_model_xml/handlers/parsing.py
Outdated
Show resolved
Hide resolved
|
PR #29888: Size comparison from 690d9a0 to 5bf9031 Full report (19 builds for cc13x4_26x4, cc32xx, k32w, linux, mbed, nrfconnect, qpg)
|
|
PR #29888: Size comparison from 690d9a0 to 9fe6a63 Full report (4 builds for cc32xx, mbed, qpg)
|
|
PR #29888: Size comparison from 690d9a0 to 7e09032 Full report (54 builds for bl602, bl702, bl702l, cc13x4_26x4, cc32xx, cyw30739, efr32, esp32, k32w, linux, mbed, nrfconnect, psoc6, qpg)
|
|
PR #29888: Size comparison from 690d9a0 to 32f97ec Full report (71 builds for bl602, bl702, bl702l, cc13x4_26x4, cc32xx, cyw30739, efr32, esp32, k32w, linux, mbed, nrfconnect, psoc6, qpg, telink)
|
woody-apple
approved these changes
Oct 20, 2023
HunsupJung
pushed a commit
to HunsupJung/connectedhomeip
that referenced
this pull request
Oct 23, 2023
* Move xml to say zapxml since that is the format * Start defining a data_model_xml parser (no functionality for now) * Start adding some basic support for data model xml parsing * make the DM parser executable * Start having the ability to parse clusters * More updates, we seem to have parsing for features * Hard-code global attributes * Remove some comments * Add enumeration handling * Add bitmap handling * Restyle * Parse structs * Re-organize parsing a bit * Make a linter happy * Another linter fix * Handling of events * More handling and logic on events * Add support for access privilege parsing * XMLs have maybe invalid enum entries. Handle them gracefully * More attribute handling updates * restyle * Support deprecate constraint * Support constraint decoding, apply on attributes for now * Restyle * Restyle * Field handling * Field handling * Some bug fixing * Start adding command handling * Restyle * Name normalization and more parsing updates * Name normalization and more parsing updates * Better messaging and fix constraint types * Restyle * Start creating a IDL codegen so we can self-test parsed output * Start with listing clusters * Enum listing * A lot more things supported * Attribute rendering * Support for string and octet string sizes * Timed command support * Restyle * Add descriptions to clusters * Attempt to fix up alignment of things * Alignment looks slightly better * Better command separation * Align comments * Align and output descriptions including clusters * More work regarding loop structures * Apply hex formatting to bitmaps. output now seems identical except one whitespace change * Identical output for now * Support API maturity. Notice that doccomments are lost on maturity :( * Fix doxygen parsing for api maturity at the cluster level * Restyle * Support endpoints, although that is not 1:1 as hex encoding and ordering for events is lost * Restyle * Add todo note that default value does not string escaping * Default rendering and add to files * More updates on file dependencies * Unit test IDL generator * Add the IDL unit test as a standard unit test * Update for python compatibility * Fix unit testing of builds when GSDK root is defined * Added a readme file * Restyle * Make xml parser use the idl codegen * Restyle * look to fix misspell warnings * Undo repo update * Fix linter errors * Codegen as idl for data_model_xml_parser.py * Parsing closer to matter idl content * more normalization and type processing * More mandatory conformance logic * Fix mandatory conditionals * Make unit test pass * Fix tests a bit more and make parsers better * Restyle * Ignore min/max values while parsing xmls, even though raw data internally contains them * Restyle * Fix space after click annotations * Compare support for human reviews * Restyle * Fix slash * Undo submodule change * fix xml to zapxml naming changes * Restyle * Update dates from 2022 to 2023 * Add note about the complex test input * Remove unused imports * Restyle * Add some commends based on code review * Add heuristic for setting enum and bitmap sizes, to make output from XML much more readable * Add support for timed and fabric scoped commands * Add missing import --------- Co-authored-by: Andrei Litvin <[email protected]>
shripad621git
pushed a commit
to shripad621git/connectedhomeip
that referenced
this pull request
Oct 31, 2023
* Move xml to say zapxml since that is the format * Start defining a data_model_xml parser (no functionality for now) * Start adding some basic support for data model xml parsing * make the DM parser executable * Start having the ability to parse clusters * More updates, we seem to have parsing for features * Hard-code global attributes * Remove some comments * Add enumeration handling * Add bitmap handling * Restyle * Parse structs * Re-organize parsing a bit * Make a linter happy * Another linter fix * Handling of events * More handling and logic on events * Add support for access privilege parsing * XMLs have maybe invalid enum entries. Handle them gracefully * More attribute handling updates * restyle * Support deprecate constraint * Support constraint decoding, apply on attributes for now * Restyle * Restyle * Field handling * Field handling * Some bug fixing * Start adding command handling * Restyle * Name normalization and more parsing updates * Name normalization and more parsing updates * Better messaging and fix constraint types * Restyle * Start creating a IDL codegen so we can self-test parsed output * Start with listing clusters * Enum listing * A lot more things supported * Attribute rendering * Support for string and octet string sizes * Timed command support * Restyle * Add descriptions to clusters * Attempt to fix up alignment of things * Alignment looks slightly better * Better command separation * Align comments * Align and output descriptions including clusters * More work regarding loop structures * Apply hex formatting to bitmaps. output now seems identical except one whitespace change * Identical output for now * Support API maturity. Notice that doccomments are lost on maturity :( * Fix doxygen parsing for api maturity at the cluster level * Restyle * Support endpoints, although that is not 1:1 as hex encoding and ordering for events is lost * Restyle * Add todo note that default value does not string escaping * Default rendering and add to files * More updates on file dependencies * Unit test IDL generator * Add the IDL unit test as a standard unit test * Update for python compatibility * Fix unit testing of builds when GSDK root is defined * Added a readme file * Restyle * Make xml parser use the idl codegen * Restyle * look to fix misspell warnings * Undo repo update * Fix linter errors * Codegen as idl for data_model_xml_parser.py * Parsing closer to matter idl content * more normalization and type processing * More mandatory conformance logic * Fix mandatory conditionals * Make unit test pass * Fix tests a bit more and make parsers better * Restyle * Ignore min/max values while parsing xmls, even though raw data internally contains them * Restyle * Fix space after click annotations * Compare support for human reviews * Restyle * Fix slash * Undo submodule change * fix xml to zapxml naming changes * Restyle * Update dates from 2022 to 2023 * Add note about the complex test input * Remove unused imports * Restyle * Add some commends based on code review * Add heuristic for setting enum and bitmap sizes, to make output from XML much more readable * Add support for timed and fabric scoped commands * Add missing import --------- Co-authored-by: Andrei Litvin <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added a parser that does a best-effort parse of the CSA XML Data model.
Things like DoorLock.xml has very obvious differences, things like Switch.xml is identical to matter files.
Example usage for human diffs:
./scripts/py_matter_idl/matter_idl/data_model_xml_parser.py \ --compare src/controller/data_model/controller-clusters.matter \ --compare-output out/orig.matter \ -o out/from_xml.matter \ data_model/clusters/Switch.xmlgenerates
out/orig.matterandout/from_xml.matterthat are identical. Replacing Switch with DoorLock shows SDK differences for DoorLock (like different enums, unsupported properties etc).If finding differences, the parser may be at fault or spec may not be reperesented in the SDK correctly. So for now identical output is a good indicator of API matching spec, but differences have to be manually checked.