Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove Test*.py scripts from excluded list and enabling its execution via glob python test runner #35795

Closed
wants to merge 2 commits into from
Closed

Remove Test*.py scripts from excluded list and enabling its execution via glob python test runner #35795

wants to merge 2 commits into from

Conversation

vatsalghelani-csa
Copy link
Contributor

Work for Issue #31317

Follow up to PR #35660

PR 19 - This PR modifies the test infrastructure by removing Test*.py scripts from the excluded list and ensuring they are now automatically included in test runs using the glob pattern in the Python test runner script.

@semanticdiff-com SemanticDiff.com
Copy link

semanticdiff-com bot commented Sep 26, 2024 •
edited
Loading

Review changes with SemanticDiff.

Analyzed 7 of 8 files.

Filename Status
✔️ src/python_testing/TestChoiceConformanceSupport.py Analyzed
✔️ src/python_testing/TestConformanceSupport.py Analyzed
✔️ src/python_testing/TestConformanceTest.py Analyzed
✔️ src/python_testing/TestIdChecks.py Analyzed
✔️ src/python_testing/TestMatterTestingSupport.py Analyzed
✔️ src/python_testing/TestSpecParsingDeviceType.py Analyzed
✔️ src/python_testing/TestSpecParsingSupport.py Analyzed
❔ .github/workflows/tests.yaml Unsupported file format

@github-actions github-actions bot added github workflows tests matter-1.4-te2-script-change Script changes before end of Matter 1.4 TE2 labels Sep 26, 2024
@pullapprove pullapprove bot added github workflows tests matter-1.4-te2-script-change Script changes before end of Matter 1.4 TE2 and removed github workflows tests matter-1.4-te2-script-change Script changes before end of Matter 1.4 TE2 labels Sep 26, 2024
andy31415
andy31415 requested changes Sep 26, 2024
View reviewed changes
Copy link
Contributor

@andy31415 andy31415 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets not do this: the runner should be an integration runner and not a path of "run arbitrary python code" just because it can.

@andy31415
Copy link
Contributor

Lets not do this: the runner should be an integration runner and not a path of "run arbitrary python code" just because it can.

Instead pure python tests should be part of BUILD.gn unit testing, since this is what they are: unit tests. Adding them to a test runner increases the complexity of executing them.

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 26, 2024 •
edited
Loading

PR #35795: Size comparison from bb9363d to 7e40f89

Full report (88 builds for bl602, bl702, bl702l, cc13x4_26x4, cc32xx, cyw30739, efr32, esp32, linux, nrfconnect, nxp, psoc6, qpg, stm32, telink, tizen)
platform target config section bb9363d 7e40f89 change % change
bl602 lighting-app bl602 FLASH 1280532 1280532 0 0.0
RAM 95848 95848 0 0.0
bl602+mfd FLASH 1294398 1294398 0 0.0
RAM 95992 95992 0 0.0
bl602+rpc FLASH 1320482 1320482 0 0.0
RAM 104272 104272 0 0.0
bl702 lighting-app bl702 FLASH 944362 944362 0 0.0
RAM 15145 15145 0 0.0
bl702+mfd FLASH 946762 946762 0 0.0
RAM 15305 15305 0 0.0
bl702+rpc FLASH 1039926 1039926 0 0.0
RAM 24173 24173 0 0.0
bl706-eth FLASH 646824 646824 0 0.0
RAM 25233 25233 0 0.0
bl706-wifi FLASH 894654 894654 0 0.0
RAM 14477 14477 0 0.0
bl702l lighting-app bl702l FLASH 961022 961022 0 0.0
RAM 16804 16804 0 0.0
bl702l+mfd FLASH 963644 963644 0 0.0
RAM 16964 16964 0 0.0
cc13x4_26x4 lighting-app LP_EM_CC1354P10_6 FLASH 829380 829380 0 0.0
RAM 123300 123300 0 0.0
lock-ftd LP_EM_CC1354P10_6 FLASH 814424 814424 0 0.0
RAM 125172 125172 0 0.0
pump-app LP_EM_CC1354P10_6 FLASH 760420 760420 0 0.0
RAM 113664 113664 0 0.0
pump-controller-app LP_EM_CC1354P10_6 FLASH 744656 744656 0 0.0
RAM 113856 113856 0 0.0
cc32xx air-purifier CC3235SF_LAUNCHXL FLASH 616202 616202 0 0.0
RAM 205396 205396 0 0.0
lock CC3235SF_LAUNCHXL FLASH 656258 656258 0 0.0
RAM 205548 205548 0 0.0
cyw30739 light CYW30739B2-P5-EVK-01 unknown 2040 2040 0 0.0
FLASH 677757 677757 0 0.0
RAM 78516 78516 0 0.0
CYW30739B2-P5-EVK-02 unknown 2040 2040 0 0.0
FLASH 697601 697601 0 0.0
RAM 81148 81148 0 0.0
CYW30739B2-P5-EVK-03 unknown 2040 2040 0 0.0
FLASH 697601 697601 0 0.0
RAM 81148 81148 0 0.0
CYW930739M2EVB-02 unknown 2040 2040 0 0.0
FLASH 654537 654537 0 0.0
RAM 73584 73584 0 0.0
light-switch CYW30739B2-P5-EVK-01 unknown 2040 2040 0 0.0
FLASH 613821 613821 0 0.0
RAM 71468 71468 0 0.0
CYW30739B2-P5-EVK-02 unknown 2040 2040 0 0.0
FLASH 633449 633449 0 0.0
RAM 74020 74020 0 0.0
CYW30739B2-P5-EVK-03 unknown 2040 2040 0 0.0
FLASH 633449 633449 0 0.0
RAM 74020 74020 0 0.0
lock CYW30739B2-P5-EVK-01 unknown 2040 2040 0 0.0
FLASH 633117 633117 0 0.0
RAM 74516 74516 0 0.0
CYW30739B2-P5-EVK-02 unknown 2040 2040 0 0.0
FLASH 652825 652825 0 0.0
RAM 77068 77068 0 0.0
CYW30739B2-P5-EVK-03 unknown 2040 2040 0 0.0
FLASH 652825 652825 0 0.0
RAM 77068 77068 0 0.0
thermostat CYW30739B2-P5-EVK-01 unknown 2040 2040 0 0.0
FLASH 608461 608461 0 0.0
RAM 68604 68604 0 0.0
CYW30739B2-P5-EVK-02 unknown 2040 2040 0 0.0
FLASH 628321 628321 0 0.0
RAM 71236 71236 0 0.0
CYW30739B2-P5-EVK-03 unknown 2040 2040 0 0.0
FLASH 628321 628321 0 0.0
RAM 71236 71236 0 0.0
efr32 lighting-app BRD4187C FLASH 933356 933356 0 0.0
RAM 135884 135884 0 0.0
lock-app BRD2605a FLASH 741024 741016 -8 -0.0
RAM 230240 230240 0 0.0
BRD4338a FLASH 741688 741680 -8 -0.0
RAM 230272 230272 0 0.0
window-app BRD4187C FLASH 1016440 1016432 -8 -0.0
RAM 127804 127804 0 0.0
esp32 all-clusters-app c3devkit DRAM 94080 94080 0 0.0
FLASH 1537628 1537628 0 0.0
IRAM 82538 82538 0 0.0
m5stack DRAM 115032 115032 0 0.0
FLASH 1548078 1548078 0 0.0
IRAM 117039 117039 0 0.0
linux air-purifier-app debug unknown 4688 4688 0 0.0
FLASH 2765555 2765555 0 0.0
RAM 129424 129424 0 0.0
all-clusters-app debug unknown 5528 5528 0 0.0
FLASH 6069044 6069044 0 0.0
RAM 507392 507392 0 0.0
all-clusters-minimal-app debug unknown 5424 5424 0 0.0
FLASH 5402374 5402374 0 0.0
RAM 241136 241136 0 0.0
bridge-app debug unknown 5408 5408 0 0.0
FLASH 4735996 4735996 0 0.0
RAM 218304 218304 0 0.0
chip-tool debug unknown 5960 5960 0 0.0
FLASH 12547890 12547890 0 0.0
RAM 555122 555122 0 0.0
chip-tool-ipv6only arm64 unknown 20456 20456 0 0.0
FLASH 11205536 11205536 0 0.0
RAM 605368 605368 0 0.0
fabric-admin debug unknown 5792 5792 0 0.0
FLASH 10909723 10909723 0 0.0
RAM 554266 554266 0 0.0
fabric-bridge-app debug unknown 4640 4640 0 0.0
FLASH 4562608 4562608 0 0.0
RAM 204888 204888 0 0.0
lighting-app debug+rpc+ui unknown 6056 6056 0 0.0
FLASH 5676961 5676961 0 0.0
RAM 228552 228552 0 0.0
lock-app debug unknown 5344 5344 0 0.0
FLASH 4786098 4786098 0 0.0
RAM 204376 204376 0 0.0
ota-provider-app debug unknown 4720 4720 0 0.0
FLASH 4418320 4418320 0 0.0
RAM 198112 198112 0 0.0
ota-requestor-app debug unknown 4656 4656 0 0.0
FLASH 4557100 4557100 0 0.0
RAM 202680 202680 0 0.0
shell debug unknown 4216 4216 0 0.0
FLASH 3096333 3096333 0 0.0
RAM 159184 159184 0 0.0
thermostat-no-ble arm64 unknown 9432 9432 0 0.0
FLASH 4307560 4307560 0 0.0
RAM 242808 242808 0 0.0
tv-app debug unknown 5624 5624 0 0.0
FLASH 6014341 6014341 0 0.0
RAM 580928 580928 0 0.0
tv-casting-app debug unknown 5208 5208 0 0.0
FLASH 10741165 10741165 0 0.0
RAM 643008 643008 0 0.0
nrfconnect all-clusters-app nrf52840dk_nrf52840 FLASH 913440 913440 0 0.0
RAM 142199 142199 0 0.0
nrf7002dk_nrf5340_cpuapp FLASH 885020 885020 0 0.0
RAM 140338 140338 0 0.0
all-clusters-minimal-app nrf52840dk_nrf52840 FLASH 846948 846948 0 0.0
RAM 141093 141093 0 0.0
nxp contact k32w0+release FLASH 581760 581760 0 0.0
RAM 70776 70776 0 0.0
k32w1+release FLASH 595664 595664 0 0.0
RAM 62992 62992 0 0.0
mcxw71+release FLASH 595416 595416 0 0.0
RAM 62992 62992 0 0.0
light k32w0+release FLASH 617812 617812 0 0.0
RAM 70248 70248 0 0.0
k32w1+release FLASH 681624 681624 0 0.0
RAM 48632 48632 0 0.0
mcxw71+release FLASH 681640 681640 0 0.0
RAM 48632 48632 0 0.0
lock k32w1+release FLASH 703944 703944 0 0.0
RAM 67132 67132 0 0.0
mcxw71+release FLASH 703968 703968 0 0.0
RAM 67132 67132 0 0.0
psoc6 all-clusters cy8ckit_062s2_43012 FLASH 1645380 1645380 0 0.0
RAM 210888 210888 0 0.0
all-clusters-minimal cy8ckit_062s2_43012 FLASH 1551364 1551364 0 0.0
RAM 207688 207688 0 0.0
light cy8ckit_062s2_43012 FLASH 1467180 1467180 0 0.0
RAM 200696 200696 0 0.0
lock cy8ckit_062s2_43012 FLASH 1463924 1463924 0 0.0
RAM 225048 225048 0 0.0
qpg lighting-app qpg6105+debug FLASH 659512 659512 0 0.0
RAM 105236 105236 0 0.0
lock-app qpg6105+debug FLASH 617476 617476 0 0.0
RAM 99704 99704 0 0.0
stm32 light STM32WB5MM-DK FLASH 481072 481072 0 0.0
RAM 144692 144692 0 0.0
telink air-quality-sensor-app tlsr9528a_retention FLASH 620082 620082 0 0.0
RAM 50988 50988 0 0.0
all-clusters-app tlsr9118bdk40d FLASH 687314 687314 0 0.0
RAM 148328 148328 0 0.0
all-clusters-minimal-app tlsr9528a FLASH 780248 780248 0 0.0
RAM 110780 110780 0 0.0
bridge-app tlsr9258a FLASH 681084 681084 0 0.0
RAM 91644 91644 0 0.0
contact-sensor-app tlsr9528a_retention FLASH 619844 619844 0 0.0
RAM 50940 50940 0 0.0
light-switch-app-ota-shell-factory-data tlsr9528a FLASH 707786 707786 0 0.0
RAM 74280 74280 0 0.0
lighting-app-ota-factory-data tlsr9118bdk40d FLASH 624806 624806 0 0.0
RAM 144316 144316 0 0.0
lighting-app-ota-rpc-factory-data-4mb tlsr9518adk80d FLASH 810458 810458 0 0.0
RAM 99448 99448 0 0.0
lock-app-dfu tlsr9528a FLASH 655690 655690 0 0.0
RAM 67000 67000 0 0.0
ota-requestor-app tlsr9258a FLASH 697060 697060 0 0.0
RAM 91236 91236 0 0.0
pump-app-usb tlsr9518adk80d FLASH 633468 633468 0 0.0
RAM 55816 55816 0 0.0
pump-controller-app tlsr9518adk80d FLASH 610658 610658 0 0.0
RAM 53060 53060 0 0.0
shell tlsr9518adk80d FLASH 467336 467336 0 0.0
RAM 68668 68668 0 0.0
smoke_co_alarm-app tlsr9528a_retention FLASH 626974 626974 0 0.0
RAM 52660 52660 0 0.0
temperature-measurement-app-mars-ota tlsr9518adk80d FLASH 652794 652794 0 0.0
RAM 56608 56608 0 0.0
thermostat tlsr9518adk80d FLASH 637388 637388 0 0.0
RAM 53452 53452 0 0.0
window-covering tlsr9118bdk40d FLASH 523574 523574 0 0.0
RAM 97284 97284 0 0.0
tizen all-clusters-app arm unknown 4876 4876 0 0.0
FLASH 1722412 1722412 0 0.0
RAM 89076 89076 0 0.0
chip-tool-ubsan arm unknown 10316 10316 0 0.0
FLASH 17352042 17352042 0 0.0
RAM 7560020 7560020 0 0.0

@vatsalghelani-csa
Copy link
Contributor Author

This PR is closed for now, the work will be carried over as a follow-up task to the PR #34785.

@vatsalghelani-csa vatsalghelani-csa deleted the testsAgainstNoAppToRunViaGlobScriptRun branch October 1, 2024 20:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andy31415 andy31415 andy31415 requested changes

@andyg-apple andyg-apple Awaiting requested review from andyg-apple

@anush-apple anush-apple Awaiting requested review from anush-apple

@arkq arkq Awaiting requested review from arkq

@axelnxp axelnxp Awaiting requested review from axelnxp

@bauerschwan bauerschwan Awaiting requested review from bauerschwan

@bzbarsky-apple bzbarsky-apple Awaiting requested review from bzbarsky-apple

@carol-apple carol-apple Awaiting requested review from carol-apple

@cecille cecille Awaiting requested review from cecille

@chapongatien chapongatien Awaiting requested review from chapongatien

@chrisdecenzo chrisdecenzo Awaiting requested review from chrisdecenzo

@chshu chshu Awaiting requested review from chshu

@chulspro chulspro Awaiting requested review from chulspro

@cliffamzn cliffamzn Awaiting requested review from cliffamzn

@Damian-Nordic Damian-Nordic Awaiting requested review from Damian-Nordic

@dhrishi dhrishi Awaiting requested review from dhrishi

@doru91 doru91 Awaiting requested review from doru91

@fessehaeve fessehaeve Awaiting requested review from fessehaeve

@harimau-qirex harimau-qirex Awaiting requested review from harimau-qirex

@harsha-rajendran harsha-rajendran Awaiting requested review from harsha-rajendran

@hawk248 hawk248 Awaiting requested review from hawk248

@hicklin hicklin Awaiting requested review from hicklin

@jepenven-silabs jepenven-silabs Awaiting requested review from jepenven-silabs

@jmartinez-silabs jmartinez-silabs Awaiting requested review from jmartinez-silabs

@jmeg-sfy jmeg-sfy Awaiting requested review from jmeg-sfy

@joonhaengHeo joonhaengHeo Awaiting requested review from joonhaengHeo

@jtov-sfy jtov-sfy Awaiting requested review from jtov-sfy

@jtung-apple jtung-apple Awaiting requested review from jtung-apple

@kiel-apple kiel-apple Awaiting requested review from kiel-apple

@kkasperczyk-no kkasperczyk-no Awaiting requested review from kkasperczyk-no

@kpschoedel kpschoedel Awaiting requested review from kpschoedel

@ksperling-apple ksperling-apple Awaiting requested review from ksperling-apple

@lazarkov lazarkov Awaiting requested review from lazarkov

@lpbeliveau-silabs lpbeliveau-silabs Awaiting requested review from lpbeliveau-silabs

@LuDuda LuDuda Awaiting requested review from LuDuda

@mhazley mhazley Awaiting requested review from mhazley

@mkardous-silabs mkardous-silabs Awaiting requested review from mkardous-silabs

@mspang mspang Awaiting requested review from mspang

@nivi-apple nivi-apple Awaiting requested review from nivi-apple

@p0fi p0fi Awaiting requested review from p0fi

@pidarped pidarped Awaiting requested review from pidarped

@plauric plauric Awaiting requested review from plauric

@rcasallas-silabs rcasallas-silabs Awaiting requested review from rcasallas-silabs

@ReneJosefsen ReneJosefsen Awaiting requested review from ReneJosefsen

@robszewczyk robszewczyk Awaiting requested review from robszewczyk

@saurabhst saurabhst Awaiting requested review from saurabhst

@selissia selissia Awaiting requested review from selissia

@sharadb-amazon sharadb-amazon Awaiting requested review from sharadb-amazon

@tcarmelveilleux tcarmelveilleux Awaiting requested review from tcarmelveilleux

@tecimovic tecimovic Awaiting requested review from tecimovic

@tima-q tima-q Awaiting requested review from tima-q

@tobiasgraf tobiasgraf Awaiting requested review from tobiasgraf

@turon turon Awaiting requested review from turon

@vivien-apple vivien-apple Awaiting requested review from vivien-apple

@wiba-nordic wiba-nordic Awaiting requested review from wiba-nordic

@woody-apple woody-apple Awaiting requested review from woody-apple

@younghak-hwang younghak-hwang Awaiting requested review from younghak-hwang

@yufengwangca yufengwangca Awaiting requested review from yufengwangca

@yunhanw-google yunhanw-google Awaiting requested review from yunhanw-google

Assignees
No one assigned
Labels
github matter-1.4-te2-script-change Script changes before end of Matter 1.4 TE2 review - pending tests workflows
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@vatsalghelani-csa @andy31415