Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix mismatched expectations on index retrieval in Ember cluster lookup #12750

Merged
merged 1 commit into from
Dec 9, 2021

Conversation

mrjerryjohns
Copy link
Contributor

@mrjerryjohns mrjerryjohns commented Dec 8, 2021

AttributePathExpandIterator uses emberAfClusterIndex to look up the index of a cluster within a given endpoint. This in turn calls into emberAfFindClusterInTypeWithMfgCode to find that index. This returns an index that indexes into all clusters in that endpoint regardless of type.

Later on however, it uses emberAfGetNthClusterId to retrieve the cluster ID for a given index. However, this function applies the index as though it's indexing into a list of clusters that match the masking criteria (i.e server or client). This results in it retrieving the wrong cluster in cases where there is a mix of client and server clusters on a given endpoint.

Since most places use this 'scoped index', this PR fixes emberAfFindClusterInTypeWithMfgCode to return a scoped index instead.

Problem:

Marc noted that when reading the WifiDiagnostics cluster, that on the
server, the logs indicated a retrieval of a ClusterId 0x36, but the
subsequent logs in the reporting engine seemed to indicate it was
picking up ClusterId 0x37 oddly enough.

Testing:

Validated that the above bug no longer happens, as well as reading out
the entire device using wildcards in the REPL to ensure everything still
works.

AttributePathExpandIterator uses emberAfClusterIndex to look up the
index of a cluster within a given endpoint. This in turn calls into
emberAfFindClusterInTypeWithMfgCode to find that index. This returns an
index that indexes into all clusters in that endpoint regardless of
type.

Later on however, it uses emberAfGetNthClusterId to retrieve the actual
cluster index for a given index. However, this function applies the
index as though it's indexing into a list of clusters that match the
masking criteria (i.e server or client). This results in it retrieving
the wrong cluster in cases where there is a mix of client and server
clusters on a given endpoint.

Since most places use this 'scoped index', this PR fixes
emberAfFindClusterInTypeWithMfgCode to return a scoped index instead.

Problem:

Marc noted that when reading the WifiDiagnostics cluster, that on the
server, the logs indicated a retrieval of a ClusterId 0x36, but the
subsequent logs in the reporting engine seemed to indicate it was
picking up ClusterId 0x37 oddly enough.

Testing:

Validated that the above bug no longer happens, as well as reading out
the entire device using wildcards in the REPL to ensure everything still
works.
@mrjerryjohns
Copy link
Contributor Author

mrjerryjohns commented Dec 8, 2021

I was puzzled why such a 'index-shifting-bug' would not have resulted in large, obvious failures since the data returned would have been for a different cluster than what was requested. This is especially true since in the large list of clusters in GENERATED_CLUSTERS in the all-clusters-app, there is a client cluster fairly early on in that list, which would have mean that all indices returned for server clusters past that would have been off by 1.

However, we have a variety of Read tests in both YAML and Python to cover this. So why didn't they break?

  1. This only happens with wildcard reads. With non-wildcard reads, we don't resort to using these emberAf* functions to do index look-up, since we already have enough information to create a ConcreteAttributePath:

in AttributePathExpandIterator.cpp:125

            if (!mpClusterInfo->HasAttributeWildcard())
            {
                mOutputPath.mEndpointId  = mpClusterInfo->mEndpointId;
                mOutputPath.mClusterId   = mpClusterInfo->mClusterId;
                mOutputPath.mAttributeId = mpClusterInfo->mAttributeId;

                // Prepare for next iteration
                mEndpointIndex = mEndEndpointIndex = 0;
                return true;
            }
  1. With wildcard reads (which are only supported in Python), we do have tests that validate the data returned matches what the type expected based on the path provided in the data. Since the Engine still returned valid data (except for the wrong cluster than what was requested), that never resulted in incorrect data where the data and path in the response didn't match up. The current Python APIs don't actually validate that the data returned matches up with what was requested. While this is possible for some wildcard combinations, for others, it's not possible to do any real checking.

Filed #12752 to at least handle some of those cases.

@mlepage-google
Copy link
Contributor

Wild, thanks for digging into it!

@github-actions
Copy link

github-actions bot commented Dec 8, 2021

PR #12750: Size comparison from e4f3246 to 6f42abd

Increases (18 builds for esp32, k32w, linux, nrfconnect, p6, qpg, telink)
platform target config section e4f3246 6f42abd change % change
esp32 all-clusters-app c3devkit (read only) 858566 858572 6 0.0
.flash.text 858566 858572 6 0.0
k32w lighting-app k32w061+se05x+release (read/write) 724516 724532 16 0.0
.text 635616 635632 16 0.0
lock-app k32w061+debug (read/write) 615868 615884 16 0.0
.text 536336 536352 16 0.0
linux chip-tool-ipv6only arm64 (read only) 6732084 6732116 32 0.0
.text 5731988 5732020 32 0.0
thermostat-no-ble arm64 (read only) 1932044 1932060 16 0.0
.text 1601088 1601104 16 0.0
nrfconnect lighting-app nrf52840dk_nrf52840 text 610104 610116 12 0.0
nrf52840dk_nrf52840+rpc (read/write) 866079 866095 16 0.0
text 584832 584844 12 0.0
nrf5340dk_nrf5340_cpuapp (read/write) 829046 829062 16 0.0
text 540532 540544 12 0.0
lock-app nrf52840dk_nrf52840 (read/write) 875523 875539 16 0.0
text 589364 589376 12 0.0
nrf5340dk_nrf5340_cpuapp (read/write) 801754 801770 16 0.0
text 519884 519896 12 0.0
pump-app nrf52840dk_nrf52840 (read/write) 880371 880387 16 0.0
text 592880 592892 12 0.0
pump-controller-app nrf52840dk_nrf52840 (read/write) 873599 873615 16 0.0
text 588100 588112 12 0.0
p6 all-clusters-app default (read/write) 2355624 2355640 16 0.0
.text 1313888 1313904 16 0.0
light-app default (read/write) 2290408 2290424 16 0.0
.text 1248672 1248688 16 0.0
lock-app default (read/write) 2266512 2266528 16 0.0
.text 1224776 1224792 16 0.0
qpg lighting-app qpg6100+debug (read only) 517744 517760 16 0.0
.text 512424 512440 16 0.0
lock-app qpg6100+debug (read only) 491944 491960 16 0.0
.text 486624 486640 16 0.0
telink lighting-app tlsr9518adk80d text 563090 563096 6 0.0
Full report (29 builds for esp32, k32w, linux, mbed, nrfconnect, p6, qpg, telink)
platform target config section e4f3246 6f42abd change % change
esp32 all-clusters-app c3devkit (read only) 858566 858572 6 0.0
(read/write) 1306658 1306658 0 0.0
.dram0.bss 67648 67648 0 0.0
.dram0.data 14124 14124 0 0.0
.flash.rodata 171856 171856 0 0.0
.flash.text 858566 858572 6 0.0
.iram0.text 62076 62076 0 0.0
m5stack (read only) 961835 961835 0 0.0
(read/write) 450948 450948 0 0.0
.dram0.bss 75000 75000 0 0.0
.dram0.data 34048 34048 0 0.0
.flash.rodata 210112 210112 0 0.0
.flash.text 956451 956451 0 0.0
.iram0.text 123451 123451 0 0.0
k32w lighting-app k32w061+se05x+release (read/write) 724516 724532 16 0.0
.bss 81248 81248 0 0.0
.data 1852 1852 0 0.0
.text 635616 635632 16 0.0
lock-app k32w061+debug (read/write) 615868 615884 16 0.0
.bss 71912 71912 0 0.0
.data 1820 1820 0 0.0
.text 536336 536352 16 0.0
shell k32w061+debug (read/write) 679124 679124 0 0.0
.bss 81612 81612 0 0.0
.data 1792 1792 0 0.0
.text 589920 589920 0 0.0
linux chip-tool-ipv6only arm64 (read only) 6732084 6732116 32 0.0
(read/write) 310977 310977 0 0.0
.bss 51665 51665 0 0.0
.data 1048 1048 0 0.0
.data.rel.ro 206592 206592 0 0.0
.dynamic 560 560 0 0.0
.got 48040 48040 0 0.0
.init 24 24 0 0.0
.init_array 160 160 0 0.0
.rodata 341796 341796 0 0.0
.text 5731988 5732020 32 0.0
thermostat-no-ble arm64 (read only) 1932044 1932060 16 0.0
(read/write) 136465 136465 0 0.0
.bss 59457 59457 0 0.0
.data 776 776 0 0.0
.data.rel.ro 69840 69840 0 0.0
.dynamic 560 560 0 0.0
.got 3528 3528 0 0.0
.init 24 24 0 0.0
.init_array 256 256 0 0.0
.rodata 124988 124988 0 0.0
.text 1601088 1601104 16 0.0
mbed all-clusters-app CY8CPROTO_062_4343W+release (read only) 6224 6224 0 0.0
(read/write) 2319536 2319536 0 0.0
.bss 186900 186900 0 0.0
.data 5232 5232 0 0.0
.heap 844312 844312 0 0.0
.text 1282112 1282112 0 0.0
lighting-app CY8CPROTO_062_4343W+release (read only) 6224 6224 0 0.0
(read/write) 2305736 2305736 0 0.0
.bss 175712 175712 0 0.0
.data 5488 5488 0 0.0
.heap 855248 855248 0 0.0
.text 1268336 1268336 0 0.0
lock-app CY8CPROTO_062_4343W+release (read only) 6224 6224 0 0.0
(read/write) 2278720 2278720 0 0.0
.bss 174752 174752 0 0.0
.data 5488 5488 0 0.0
.heap 856208 856208 0 0.0
.text 1241320 1241320 0 0.0
pigweed-app CY8CPROTO_062_4343W+release (read only) 6224 6224 0 0.0
(read/write) 1140008 1140008 0 0.0
.bss 11756 11756 0 0.0
.data 4376 4376 0 0.0
.heap 1020312 1020312 0 0.0
.text 103392 103392 0 0.0
shell CY8CPROTO_062_4343W+release (read only) 6224 6224 0 0.0
(read/write) 2047808 2047808 0 0.0
.bss 156564 156564 0 0.0
.data 4864 4864 0 0.0
.heap 875016 875016 0 0.0
.text 1010408 1010408 0 0.0
nrfconnect lighting-app nrf52840dk_nrf52840 (read/write) 903055 903055 0 0.0
bss 116184 116184 0 0.0
rodata 101208 101208 0 0.0
text 610104 610116 12 0.0
nrf52840dk_nrf52840+rpc (read/write) 866079 866095 16 0.0
bss 112532 112532 0 0.0
rodata 92504 92504 0 0.0
text 584832 584844 12 0.0
nrf5340dk_nrf5340_cpuapp (read/write) 829046 829062 16 0.0
bss 117560 117560 0 0.0
rodata 96464 96464 0 0.0
text 540532 540544 12 0.0
lock-app nrf52840dk_nrf52840 (read/write) 875523 875539 16 0.0
bss 113448 113448 0 0.0
rodata 97324 97324 0 0.0
text 589364 589376 12 0.0
nrf5340dk_nrf5340_cpuapp (read/write) 801754 801770 16 0.0
bss 114856 114856 0 0.0
rodata 92612 92612 0 0.0
text 519884 519896 12 0.0
pigweed-app nrf52840dk_nrf52840 (read/write) 497463 497463 0 0.0
bss 51820 51820 0 0.0
rodata 45852 45852 0 0.0
text 339492 339492 0 0.0
pump-app nrf52840dk_nrf52840 (read/write) 880371 880387 16 0.0
bss 113360 113360 0 0.0
rodata 98676 98676 0 0.0
text 592880 592892 12 0.0
pump-controller-app nrf52840dk_nrf52840 (read/write) 873599 873615 16 0.0
bss 113236 113236 0 0.0
rodata 96816 96816 0 0.0
text 588100 588112 12 0.0
shell nrf52840dk_nrf52840 (read/write) 781611 781611 0 0.0
bss 109552 109552 0 0.0
rodata 74280 74280 0 0.0
text 523264 523264 0 0.0
nrf5340dk_nrf5340_cpuapp (read/write) 696686 696686 0 0.0
bss 110532 110532 0 0.0
rodata 68920 68920 0 0.0
text 443904 443904 0 0.0
p6 all-clusters-app default (read/write) 2355624 2355640 16 0.0
.bss 113132 113132 0 0.0
.data 2512 2512 0 0.0
.heap 917696 917696 0 0.0
.text 1313888 1313904 16 0.0
light-app default (read/write) 2290408 2290424 16 0.0
.bss 100904 100904 0 0.0
.data 2328 2328 0 0.0
.heap 930112 930112 0 0.0
.text 1248672 1248688 16 0.0
lock-app default (read/write) 2266512 2266528 16 0.0
.bss 99784 99784 0 0.0
.data 2288 2288 0 0.0
.heap 931272 931272 0 0.0
.text 1224776 1224792 16 0.0
qpg lighting-app qpg6100+debug (read only) 517744 517760 16 0.0
(read/write) 122332 122332 0 0.0
.bss 82624 82624 0 0.0
.data 956 956 0 0.0
.text 512424 512440 16 0.0
lock-app qpg6100+debug (read only) 491944 491960 16 0.0
(read/write) 122336 122336 0 0.0
.bss 81760 81760 0 0.0
.data 912 912 0 0.0
.text 486624 486640 16 0.0
persistent-storage-app qpg6100+debug (read only) 108104 108104 0 0.0
(read/write) 122336 122336 0 0.0
.bss 36152 36152 0 0.0
.data 288 288 0 0.0
.text 102784 102784 0 0.0
telink lighting-app tlsr9518adk80d (read/write) 807090 807090 0 0.0
bss 82764 82764 0 0.0
noinit 37160 37160 0 0.0
text 563090 563096 6 0.0

@andy31415 andy31415 merged commit bba082b into project-chip:master Dec 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants