Add compile-time code generation using zap for app-templates (#23962)

* Add compile-time code generation using zap for app-templates - add support for gn builds - add support for cmake builds - update esp32 build logic - update pre-generation script - split out app and idl codegen - update template used so zap_generate_all only generates IDLs for apps - updated CI unit tests * Fix tizen build dependencies for all-clusters and minimal * Remove one more unnedded include directory from tizen * Fix tv casting app include * Restyle * Bump up openiotsdk dockerfile to get zap-cli * Fix include for efr32 Lcd painter * Test out a zap installation workflow in darwin builds * Fix trailing slash * Multiline to make mac install more redable * make sure all subdirs are created when installing zap * Use sudo to create the zap install directory * unzip only zap-cli, to hopefully make the unzip faster * Install zap on all darwin CI runs * Add `"$(TEMP_DIR)/out/gen/src/controller/data_model/zapgen/"` to xcode project header search paths * Do not run clang-format for compile-time generated files * Update code generation documentation a bit * More documentation with examples of how to invoke and set pregeneration variables * Restyle * Fix typo in command for code pregen * Fix typo * Add a runnable check that is intended to codegen * make unit tests with golden images run (but not yet pass) * Update golden copies * Update documentation and restyle * Bump up build apps time... seems like bimodal timing distribution and this occasionally fails at 1h, leave to 1.5hrs for now * Fix spelling * make zap-cli --version run on mac as part of the install since I seem to get package conflict errors periodically * Try to use tempState for mac parallel runs * Restyle * Several code review changes * Fix typo in path for templates * Undo third party submodule changes * Move common python zap logic into one file, to have the zap path logic centralized and to be more easily be able to add version checks * Restyle * Add version check capabilities to zap execution and generate.py * Restyle * Update zap version and set skip_real_version based on merge with master * Also update darwin zap installation path * Update more paths with the new zap version * Restyle * Golden image update for regeneration * Do not restyle test outputs for zap generation tests * Restyle * Restyled by isort * Fix up one more include for obsolete paths * Update generated files for unit tests * Re-generate unit test output data * Fix telink contact sensor build * Update unit test data * Revert wrong diff marker * Bump CI docker to 0.6.31 to use the latest zap * Bump up image versions for non-chip-build images as well * Re-generate using latest zap from 0.6.31 image * Keep codegen for controller clusters, for darwin specifically * patch the matter xcode project to update the include path to use the darwin-spefici codegen for controller cluster paths * Restyle * Update to make template checker happy, update min version for zap execution * One more manual restyle attempt * Ran regen again for tests * Re-generated this time using CI steps for checking * See if CI can run zap tests on mac as well * Do not set skip real version for the version check * Add a better comment of why IMClusterCommandHandler is odd * Restyle * Start developing a simpler ZAP version update script as this is likely often used * Off by one fix and update search logic a bit * Add support for min version update as well * Update a comment and a min version * Comment updates * Another comment regarding version update * Fix gen files for unit tests. We should figure out why environment matters here ... this is odd * Update clang-format version comment: it is 15 not 14 now in pigweed * Force clang-format-15 or clang-format (assuming we use pigweed) * Regenerate in pigweed environment, so clang-format is consistently 15 * Report clang-format version when formatting, to understand any format differences * Re-gen all after clang-format-15 was applied * Remove zzz_generated from restyled.yaml since we clang-format already during zap gen * Zap regen app2 * Regenerated (after restyle changed permissions..) and updated logs to contain what files get formatted * Update CI image for coverage run * Serialize zap execution to work around zap darwin bugs * Fix copy and paste error for cmake: remove extra comma * Update version update to control what files to update: usage or docker * Update scripts/tools/zap/generate.py Co-authored-by: Boris Zbarsky <[email protected]> * Skip ZAP version check when ZAP_DEVELOPMENT_PATH is set * Restyle Co-authored-by: Restyled.io <[email protected]> Co-authored-by: Andrei Litvin <[email protected]> Co-authored-by: Boris Zbarsky <[email protected]>
project-chip · Sep 1, 2023 · 2402194 · 2402194
1 parent e39cee7
commit 2402194
Show file tree

Hide file tree

Showing 380 changed files with 35,539 additions and 93,679 deletions.
diff --git a/.github/workflows/bloat_check.yaml b/.github/workflows/bloat_check.yaml
@@ -30,7 +30,7 @@ jobs:
         runs-on: ubuntu-latest
 
         container:
-            image: connectedhomeip/chip-build:0.6.30
+            image: connectedhomeip/chip-build:0.6.31
 
         steps:
             - uses: Wandalen/[email protected]

diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml
@@ -36,7 +36,7 @@ jobs:
         if: github.actor != 'restyled-io[bot]'
 
         container:
-            image: connectedhomeip/chip-build:0.6.30
+            image: connectedhomeip/chip-build:0.6.31
             volumes:
                 - "/tmp/log_output:/tmp/test_logs"
             options: --privileged --sysctl "net.ipv6.conf.all.disable_ipv6=0
@@ -138,7 +138,7 @@ jobs:
         if: github.actor != 'restyled-io[bot]'
 
         container:
-            image: connectedhomeip/chip-build:0.6.30
+            image: connectedhomeip/chip-build:0.6.31
             volumes:
                 - "/tmp/log_output:/tmp/test_logs"
             options: --privileged --sysctl "net.ipv6.conf.all.disable_ipv6=0
@@ -251,6 +251,14 @@ jobs:
                        --pregen-dir ./zzz_pregenerated \
                        build \
                     "
+            - name: Check no code generation in output
+              run: |
+                  CNT=$(find -name "CHIPClusters.h" out | wc -l)
+                  if [ "${CNT}" != "0" ]; then
+                     echo "ERROR: found unexpected generated files:"
+                     find -name "CHIPClusters.h" out
+                     exit 1
+                  fi
             - name: Undo code pre-generation changes (make compile time codegen work again)
               run: |
                   rm -rf ./zzz_pregenerated
@@ -290,7 +298,7 @@ jobs:
         if: github.actor != 'restyled-io[bot]'
 
         container:
-            image: connectedhomeip/chip-build:0.6.30
+            image: connectedhomeip/chip-build:0.6.31
             volumes:
                 - "/tmp/log_output:/tmp/test_logs"
             options: --sysctl "net.ipv6.conf.all.disable_ipv6=0
@@ -344,6 +352,12 @@ jobs:
         timeout-minutes: 200
         runs-on: macos-latest
         if: github.actor != 'restyled-io[bot]'
+
+        env:
+            # NOTE: Generally kept in sync within the repo using
+            #       scripts/tools/zap/version_update.py
+            ZAP_VERSION: v2023.01.09-nightly
+            ZAP_INSTALL_PATH: /usr/local/zap/zap-v2023.01.09-nightly
 
         steps:
             - uses: Wandalen/[email protected]
@@ -374,6 +388,16 @@ jobs:
                   ls -la /usr/local/Cellar/[email protected]
                   OPEN_SSL_VERSION=`ls -la /usr/local/Cellar/[email protected] | cat | tail -n1 | awk '{print $NF}'`
                   ln -s /usr/local/Cellar/[email protected]/$OPEN_SSL_VERSION/lib/pkgconfig/* .
+            - name: Install zap
+              run: |
+                 sudo mkdir -p $ZAP_INSTALL_PATH
+                 sudo chown `whoami` $ZAP_INSTALL_PATH
+                 curl -L https://github.com/project-chip/zap/releases/download/${ZAP_VERSION}/zap-mac.zip \
+                      --output $ZAP_INSTALL_PATH/zap-mac.zip
+                 cd $ZAP_INSTALL_PATH
+                 unzip zap-mac.zip zap-cli
+                 rm zap-mac.zip
+                 ./zap-cli --version
             - name: Bootstrap
               timeout-minutes: 25
               run: scripts/build/gn_bootstrap.sh
@@ -447,7 +471,7 @@ jobs:
         if: github.actor != 'restyled-io[bot]'
 
         container:
-            image: connectedhomeip/chip-build:0.6.30
+            image: connectedhomeip/chip-build:0.6.31
             volumes:
                 - "/tmp/log_output:/tmp/test_logs"
             options: --privileged --sysctl "net.ipv6.conf.all.disable_ipv6=0
@@ -478,4 +502,4 @@ jobs:
             - name: Run Build Coverage
               timeout-minutes: 20
               run: ./scripts/build_coverage.sh
-
+
diff --git a/.github/workflows/chef.yaml b/.github/workflows/chef.yaml
@@ -29,7 +29,7 @@ jobs:
         if: github.actor != 'restyled-io[bot]'
 
         container:
-            image: connectedhomeip/chip-build:0.6.30
+            image: connectedhomeip/chip-build:0.6.31
             options: --user root
 
         steps:
@@ -57,7 +57,7 @@ jobs:
         if: github.actor != 'restyled-io[bot]'
 
         container:
-            image: connectedhomeip/chip-build-esp32:0.6.30
+            image: connectedhomeip/chip-build-esp32:0.6.31
             options: --user root
 
         steps:
@@ -85,7 +85,7 @@ jobs:
         if: github.actor != 'restyled-io[bot]'
 
         container:
-            image: connectedhomeip/chip-build-nrf-platform:0.6.30
+            image: connectedhomeip/chip-build-nrf-platform:0.6.31
             options: --user root
 
         steps:

diff --git a/.github/workflows/cirque.yaml b/.github/workflows/cirque.yaml
@@ -29,7 +29,7 @@ jobs:
         timeout-minutes: 90
 
         env:
-            DOCKER_RUN_VERSION: 0.6.30
+            DOCKER_RUN_VERSION: 0.6.31
             GITHUB_CACHE_PATH: /tmp/cirque-cache/
 
         runs-on: ubuntu-latest
@@ -38,7 +38,7 @@ jobs:
         # need to run with privilege, which isn't supported by job.XXX.contaner
         #  https://github.com/actions/container-action/issues/2
         #         container:
-        #             image: connectedhomeip/chip-build-cirque:0.6.30
+        #             image: connectedhomeip/chip-build-cirque:0.6.31
         #             volumes:
         #                 - "/tmp:/tmp"
         #                 - "/dev/pts:/dev/pts"

diff --git a/.github/workflows/darwin-tests.yaml b/.github/workflows/darwin-tests.yaml
@@ -39,6 +39,10 @@ jobs:
         env:
             BUILD_VARIANT: ${{matrix.build_variant}}
             LSAN_OPTIONS: detect_leaks=1 malloc_context_size=40 suppressions=scripts/tests/chiptest/lsan-mac-suppressions.txt
+            # NOTE: Generally kept in sync within the repo using
+            #       scripts/tools/zap/version_update.py
+            ZAP_VERSION: v2023.01.09-nightly
+            ZAP_INSTALL_PATH: /usr/local/zap/zap-v2023.01.09-nightly
 
         if: github.actor != 'restyled-io[bot]'
         runs-on: macos-latest
@@ -72,6 +76,16 @@ jobs:
                   ls -la /usr/local/Cellar/[email protected]
                   OPEN_SSL_VERSION=`ls -la /usr/local/Cellar/[email protected] | cat | tail -n1 | awk '{print $NF}'`
                   ln -s /usr/local/Cellar/[email protected]/$OPEN_SSL_VERSION/lib/pkgconfig/* .
+            - name: Install zap
+              run: |
+                 sudo mkdir -p $ZAP_INSTALL_PATH
+                 sudo chown `whoami` $ZAP_INSTALL_PATH
+                 curl -L https://github.com/project-chip/zap/releases/download/${ZAP_VERSION}/zap-mac.zip \
+                      --output $ZAP_INSTALL_PATH/zap-mac.zip
+                 cd $ZAP_INSTALL_PATH
+                 unzip zap-mac.zip zap-cli
+                 rm zap-mac.zip
+                 ./zap-cli --version
             - name: Bootstrap
               timeout-minutes: 25
               run: scripts/build/gn_bootstrap.sh

diff --git a/.github/workflows/darwin.yaml b/.github/workflows/darwin.yaml
@@ -31,6 +31,12 @@ jobs:
         if: github.actor != 'restyled-io[bot]'
         runs-on: macos-latest
 
+        env:
+            # NOTE: Generally kept in sync within the repo using
+            #       scripts/tools/zap/version_update.py
+            ZAP_VERSION: v2023.01.09-nightly
+            ZAP_INSTALL_PATH: /usr/local/zap/zap-v2023.01.09-nightly
+
         steps:
             - uses: Wandalen/[email protected]
               name: Checkout
@@ -52,6 +58,16 @@ jobs:
                   ls -la /usr/local/Cellar/[email protected]
                   OPEN_SSL_VERSION=`ls -la /usr/local/Cellar/[email protected] | cat | tail -n1 | awk '{print $NF}'`
                   ln -s /usr/local/Cellar/[email protected]/$OPEN_SSL_VERSION/lib/pkgconfig/* .
+            - name: Install zap
+              run: |
+                 sudo mkdir -p $ZAP_INSTALL_PATH
+                 sudo chown `whoami` $ZAP_INSTALL_PATH
+                 curl -L https://github.com/project-chip/zap/releases/download/${ZAP_VERSION}/zap-mac.zip \
+                      --output $ZAP_INSTALL_PATH/zap-mac.zip
+                 cd $ZAP_INSTALL_PATH
+                 unzip zap-mac.zip zap-cli
+                 rm zap-mac.zip
+                 ./zap-cli --version
             - name: Bootstrap
               timeout-minutes: 25
               run: scripts/build/gn_bootstrap.sh

diff --git a/.github/workflows/doxygen.yaml b/.github/workflows/doxygen.yaml
@@ -82,7 +82,7 @@ jobs:
 
         runs-on: ubuntu-20.04
         container:
-            image: connectedhomeip/chip-build-doxygen:0.6.30
+            image: connectedhomeip/chip-build-doxygen:0.6.31
 
         if: github.actor != 'restyled-io[bot]'
 

diff --git a/.github/workflows/examples-ameba.yaml b/.github/workflows/examples-ameba.yaml
@@ -35,7 +35,7 @@ jobs:
         if: github.actor != 'restyled-io[bot]'
 
         container:
-            image: connectedhomeip/chip-build-ameba:0.6.30
+            image: connectedhomeip/chip-build-ameba:0.6.31
             options: --user root
 
         steps:

diff --git a/.github/workflows/examples-bouffalolab.yaml b/.github/workflows/examples-bouffalolab.yaml
@@ -35,7 +35,7 @@ jobs:
     if: github.actor != 'restyled-io[bot]'
 
     container:
-      image: connectedhomeip/chip-build-bouffalolab:0.6.30
+      image: connectedhomeip/chip-build-bouffalolab:0.6.31
       volumes:
         - "/tmp/bloat_reports:/tmp/bloat_reports"
     steps:

diff --git a/.github/workflows/examples-cc13x2x7_26x2x7.yaml b/.github/workflows/examples-cc13x2x7_26x2x7.yaml
@@ -37,7 +37,7 @@ jobs:
         if: github.actor != 'restyled-io[bot]'
 
         container:
-            image: connectedhomeip/chip-build-ti:0.6.30
+            image: connectedhomeip/chip-build-ti:0.6.31
             volumes:
                 - "/tmp/bloat_reports:/tmp/bloat_reports"
         steps:

diff --git a/.github/workflows/examples-efr32.yaml b/.github/workflows/examples-efr32.yaml
@@ -38,7 +38,7 @@ jobs:
     if: github.actor != 'restyled-io[bot]'
 
     container:
-      image: connectedhomeip/chip-build-efr32:0.6.30
+      image: connectedhomeip/chip-build-efr32:0.6.31
       volumes:
         - "/tmp/bloat_reports:/tmp/bloat_reports"
     steps:

diff --git a/.github/workflows/examples-esp32.yaml b/.github/workflows/examples-esp32.yaml
@@ -34,7 +34,7 @@ jobs:
         if: github.actor != 'restyled-io[bot]'
 
         container:
-            image: connectedhomeip/chip-build-esp32:0.6.30
+            image: connectedhomeip/chip-build-esp32:0.6.31
             volumes:
                 - "/tmp/bloat_reports:/tmp/bloat_reports"
 
@@ -93,6 +93,14 @@ jobs:
                         build \
                         --copy-artifacts-to out/artifacts \
                      "
+            - name: Check no code generation in output
+              run: |
+                  CNT=$(find -name "CHIPClusters.h" out | wc -l)
+                  if [ "${CNT}" != "0" ]; then
+                     echo "ERROR: found unexpected generated files:"
+                     find -name "CHIPClusters.h" out
+                     exit 1
+                  fi
             - name: Undo code pregeneration changes
               run: |
                   rm -rf ./zzz_pregenerated
@@ -140,7 +148,7 @@ jobs:
         if: github.actor != 'restyled-io[bot]'
 
         container:
-            image: connectedhomeip/chip-build-esp32:0.6.30
+            image: connectedhomeip/chip-build-esp32:0.6.31
             volumes:
                 - "/tmp/bloat_reports:/tmp/bloat_reports"
 

diff --git a/.github/workflows/examples-infineon.yaml b/.github/workflows/examples-infineon.yaml
@@ -35,7 +35,7 @@ jobs:
         if: github.actor != 'restyled-io[bot]'
 
         container:
-            image: connectedhomeip/chip-build-infineon:0.6.30
+            image: connectedhomeip/chip-build-infineon:0.6.31
             volumes:
                 - "/tmp/bloat_reports:/tmp/bloat_reports"
         steps:

diff --git a/.github/workflows/examples-k32w.yaml b/.github/workflows/examples-k32w.yaml
@@ -37,7 +37,7 @@ jobs:
         if: github.actor != 'restyled-io[bot]'
 
         container:
-            image: connectedhomeip/chip-build-k32w:0.6.30
+            image: connectedhomeip/chip-build-k32w:0.6.31
             volumes:
                 - "/tmp/bloat_reports:/tmp/bloat_reports"
         steps:

diff --git a/.github/workflows/examples-linux-arm.yaml b/.github/workflows/examples-linux-arm.yaml
@@ -34,7 +34,7 @@ jobs:
         if: github.actor != 'restyled-io[bot]'
 
         container:
-            image: connectedhomeip/chip-build-crosscompile:0.6.30
+            image: connectedhomeip/chip-build-crosscompile:0.6.31
             volumes:
                 - "/tmp/bloat_reports:/tmp/bloat_reports"
 

diff --git a/.github/workflows/examples-linux-imx.yaml b/.github/workflows/examples-linux-imx.yaml
@@ -34,7 +34,7 @@ jobs:
         if: github.actor != 'restyled-io[bot]'
 
         container:
-            image: connectedhomeip/chip-build-imx:0.6.30
+            image: connectedhomeip/chip-build-imx:0.6.31
 
         steps:
             - uses: Wandalen/[email protected]

diff --git a/.github/workflows/examples-linux-standalone.yaml b/.github/workflows/examples-linux-standalone.yaml
@@ -34,7 +34,7 @@ jobs:
         if: github.actor != 'restyled-io[bot]'
 
         container:
-            image: connectedhomeip/chip-build:0.6.30
+            image: connectedhomeip/chip-build:0.6.31
             volumes:
                 - "/tmp/bloat_reports:/tmp/bloat_reports"
 

diff --git a/.github/workflows/examples-mbed.yaml b/.github/workflows/examples-mbed.yaml
@@ -40,7 +40,7 @@ jobs:
         if: github.actor != 'restyled-io[bot]'
 
         container:
-            image: connectedhomeip/chip-build-mbed-os:0.6.30
+            image: connectedhomeip/chip-build-mbed-os:0.6.31
             volumes:
                 - "/tmp/bloat_reports:/tmp/bloat_reports"
 

diff --git a/.github/workflows/examples-mw320.yaml b/.github/workflows/examples-mw320.yaml
@@ -37,7 +37,7 @@ jobs:
         if: github.actor != 'restyled-io[bot]'
 
         container:
-            image: connectedhomeip/chip-build:0.6.30
+            image: connectedhomeip/chip-build:0.6.31
             volumes:
                 - "/tmp/bloat_reports:/tmp/bloat_reports"
         steps:

diff --git a/.github/workflows/examples-nrfconnect.yaml b/.github/workflows/examples-nrfconnect.yaml
@@ -37,7 +37,7 @@ jobs:
         if: github.actor != 'restyled-io[bot]'
 
         container:
-            image: connectedhomeip/chip-build-nrf-platform:0.6.30
+            image: connectedhomeip/chip-build-nrf-platform:0.6.31
             volumes:
                 - "/tmp/bloat_reports:/tmp/bloat_reports"
 

diff --git a/.github/workflows/examples-openiotsdk.yaml b/.github/workflows/examples-openiotsdk.yaml
@@ -35,7 +35,7 @@ jobs:
         if: github.actor != 'restyled-io[bot]'
 
         container:
-            image: connectedhomeip/chip-build-openiotsdk:0.6.30
+            image: connectedhomeip/chip-build-openiotsdk:0.6.31
             volumes:
                 - "/tmp/bloat_reports:/tmp/bloat_reports"
             options: --privileged

diff --git a/.github/workflows/examples-qpg.yaml b/.github/workflows/examples-qpg.yaml
@@ -37,7 +37,7 @@ jobs:
         if: github.actor != 'restyled-io[bot]'
 
         container:
-            image: connectedhomeip/chip-build:0.6.30
+            image: connectedhomeip/chip-build:0.6.31
             volumes:
                 - "/tmp/bloat_reports:/tmp/bloat_reports"
         steps:

diff --git a/.github/workflows/examples-telink.yaml b/.github/workflows/examples-telink.yaml
@@ -35,7 +35,7 @@ jobs:
         if: github.actor != 'restyled-io[bot]'
 
         container:
-            image: connectedhomeip/chip-build-telink:0.6.30
+            image: connectedhomeip/chip-build-telink:0.6.31
             volumes:
                 - "/tmp/bloat_reports:/tmp/bloat_reports"
 

diff --git a/.github/workflows/examples-tizen.yaml b/.github/workflows/examples-tizen.yaml
@@ -33,7 +33,7 @@ jobs:
         if: github.actor != 'restyled-io[bot]'
 
         container:
-            image: connectedhomeip/chip-build-tizen:0.6.30
+            image: connectedhomeip/chip-build-tizen:0.6.31
             options: --user root
             volumes:
                 - "/tmp/bloat_reports:/tmp/bloat_reports"

diff --git a/.github/workflows/full-android.yaml b/.github/workflows/full-android.yaml
@@ -37,7 +37,7 @@ jobs:
         if: github.actor != 'restyled-io[bot]'
 
         container:
-            image: connectedhomeip/chip-build-android:0.6.30
+            image: connectedhomeip/chip-build-android:0.6.31
             volumes:
                 - "/tmp/log_output:/tmp/test_logs"