Installation failed, unclear why (FAILED: python/gen/matter_build_venv/compiled_requirements.txt) (CON-1215)

[JetForMe]

Describe the bug
I'm running through the installation instructions for Matter support, and ran into an error I don't understand. You can see the lengthy transcript here. Excerpt:

[511/1099] python3 ../../third_party/pigweed/repo/pw_build/py/pw_build/python_runner.py --gn-root ../../ --current-path ../.. --default-toolchain=//build/toolchain/host:mac_arm64_gcc --current-toolchain=//third_party/pigweed/repo/pw_build/python_toolchain:python --capture-output --module piptools --python-virtualenv-config python/gen/matter_build_venv/venv_metadata.json --python-dep-list-files python/gen/matter_build_venv._compile_requirements_metadata_path_list.txt -- compile --resolver=backtracking --allow-unsafe --output-file python/gen/matter_build_venv/compiled_requirements.txt python/gen/matter_build_venv/generated_requirements.txt ../../scripts/setup/requirements.build.txt
FAILED: python/gen/matter_build_venv/compiled_requirements.txt 
python3 ../../third_party/pigweed/repo/pw_build/py/pw_build/python_runner.py --gn-root ../../ --current-path ../.. --default-toolchain=//build/toolchain/host:mac_arm64_gcc --current-toolchain=//third_party/pigweed/repo/pw_build/python_toolchain:python --capture-output --module piptools --python-virtualenv-config python/gen/matter_build_venv/venv_metadata.json --python-dep-list-files python/gen/matter_build_venv._compile_requirements_metadata_path_list.txt -- compile --resolver=backtracking --allow-unsafe --output-file python/gen/matter_build_venv/compiled_requirements.txt python/gen/matter_build_venv/generated_requirements.txt ../../scripts/setup/requirements.build.txt
    error: subprocess-exited-with-error
    
    × Getting requirements to build wheel did not run successfully.

Environment

• ESP-Matter Commit Id: d83caf7fefea
• ESP-IDF Commit Id: v5.2.2
• SoC (eg: ESP32 or ESP32-C3):
• Device Logs (Please attach the log file):
• Host Machine OS: macOS 14.5, Xcode 16beta1
• Commissioner app and versions if present:
• Commissioner's logs if present:

Any additional details
...

[9zigen]

I have the same problem. After downgrading Python version from 3.12 to 3.9 it works.

[ntwyman]

I also had to downgrade python 3.12 to 3.9.2 to get past issues installing esp-matter on an M1 Mac.

[farhan-syed]

Also would like to add incase someone sees this and still is getting the issue even though you have 3.9. May need to install Rosetta 2 if you don't already have it. Installing Rosetta fixed it for me.

[shubhamdp]

@JetForMe can you please provide the complete failure logs and what python version are you using?

[JetForMe]

@JetForMe can you please provide the complete failure logs and what python version are you using?

Oh I'm sorry, I didn't realize I had the wrong link in there. I've edited the original comment, but you can find it here: https://gist.github.com/JetForMe/f410ac9057d2a6c0990f2f53c04c6915

% python --version
Python 3.12.3

[retsyx]

Espressif, please fix this. Having to downgrade python to 3.9 is not a solution.

Edit: this seems to be an old known issue in the upstream matter library. Which, in turn, is because of a breaking change in Python 3.11.

In the meantime, this awful workaround in esp-idf prioritizes using older versions of installed python so that it can work.

diff --git a/tools/detect_python.sh b/tools/detect_python.sh
index bc3e65c59a..f98ab37bac 100644
--- a/tools/detect_python.sh
+++ b/tools/detect_python.sh
@@ -12,7 +12,7 @@ OLDEST_PYTHON_SUPPORTED_MINOR=8
 
 ESP_PYTHON=python
 
-for p_cmd in python3 python python3.8 python3.9 python3.10 python3.11 python3.12; do
+for p_cmd in python3.8 python3.9 python3.10 python3.11 python3.12 python3; do
     $p_cmd --version >/dev/null 2>&1 || continue
     echo "Checking \"$p_cmd\" ..."

[sldr]

Another workaround that worked for me is:

echo "cython<3" > /tmp/constraint.txt
export PIP_CONSTRAINT=/tmp/constraint.txt
cd esp-matter
./install.sh

Note:

• This seems to get you down the road to another module using imp instead of importlib. I think egg or maybe python has made me hangry.
• Credit for find is really in pyyaml issue commant

Laters,
SLDR
(Stephen L. De Rudder)

IT WORKED!!! I was able to build the esp-matter/examples/managed_component_light demo and flash it to an esp32h2 and add it to my homeassistant yellow matter over thread network. I am running Ubuntu 24.04 with Python 3.12.3. The esp-idf git clone was switched to 5.2.1 (per doc) and esp-matter is on branch main (specifically 794f30e). I just ignored the other (down the road) error mentioned in previous note. I used the code from here instead of the QR code pic but I expect the pic would have worked. If I missed anything let me know. My plan is to create a sensor (a bunch on a single esp32h2 for my home alarm system).

[shubhamdp]

@JetForMe @retsyx main branch was forked into release/v1.3 and we have updated the submodule for connectedhomeip to ae52f4e597cd79e956142009b82eb700a0c916f1, it has the fix for working with Python 3.12.

Can you folks please give it a try on release/v1.3 branch.

[JetForMe]

Yes, it seems to work. I followed the instructions here, except I did the following:

% git clone  https://github.com/espressif/esp-matter.git 
% git switch release/v1.3
% git submodule update --init --depth 1
% cd ./connectedhomeip/connectedhomeip
% ./scripts/checkout_submodules.py --platform esp32 darwin --shallow
% cd ../..
% ./install.sh

Running Matter Setup


  WELCOME TO...

         █
         █
     ▄   █   ▄                                █     █
     ▀▀█████▀▀      ▄▀▀▀▄ ▄▀▀▀▄    ▄▀▀▀▀▄█  ▀▀█▀▀▀▀▀█▀▀   ▄▀▀▀▀▄    ▄▀▀
   ▀█▄       ▄█▀   █     █     █  █      █    █     █    █▄▄▄▄▄▄█  █   
     ▀█▄   ▄█▀     █     █     █  █      █    █     █    █         █   
  ▄██▀▀█   █▀▀██▄  █     █     █   ▀▄▄▄▄▀█    ▀▄▄   ▀▄▄   ▀▄▄▄▄▀   █   
 ▀▀    █   █    ▀▀

  BOOTSTRAP! Bootstrap may take a few minutes; please be patient.

Downloading and installing packages into local source directory:

  Setting up CIPD package manager...done (35.9s)
  Setting up Project actions........skipped (0.1s)
  Setting up Python environment.....done (1m8.7s)
  Setting up pw packages............skipped (0.1s)
  Setting up Host tools.............done (0.1s)

Activating environment (setting environment variables):

  Setting environment variables for CIPD package manager...done
  Setting environment variables for Project actions........skipped
  Setting environment variables for Python environment.....done
  Setting environment variables for pw packages............skipped
  Setting environment variables for Host tools.............done

Checking the environment:

20240731 01:37:19 INF Environment passes all checks!

Environment looks good, you are ready to go!

To reactivate this environment in the future, run this in your 
terminal:

  source ./activate.sh

To deactivate this environment, run this:

  deactivate

Installing pip requirements for all...

[notice] A new release of pip is available: 23.2.1 -> 24.2
[notice] To update, run: pip install --upgrade pip

Building host tools

Done. Made 7794 targets from 368 files in 332ms
ninja: Entering directory `/Users/rmann/Projects/Personal/ESP32-C6/esp/esp-matter/connectedhomeip/connectedhomeip/out/host'
[885/885] ld ./chip-tool
ld: warning: ignoring duplicate libraries: 'liblogging-stdio.a'

Host tools built at: /Users/rmann/Projects/Personal/ESP32-C6/esp/esp-matter/connectedhomeip/connectedhomeip/out/host


Exit Matter environment


Installing python dependencies for Matter

Collecting lark==1.1.2 (from -r /Users/rmann/Projects/Personal/ESP32-C6/esp/esp-matter/requirements.txt (line 2))
  Downloading lark-1.1.2-py2.py3-none-any.whl.metadata (1.7 kB)
Collecting stringcase==1.2.0 (from -r /Users/rmann/Projects/Personal/ESP32-C6/esp/esp-matter/requirements.txt (line 3))
  Using cached stringcase-1.2.0-py3-none-any.whl
Collecting jinja2==3.0.1 (from -r /Users/rmann/Projects/Personal/ESP32-C6/esp/esp-matter/requirements.txt (line 5))
  Downloading Jinja2-3.0.1-py3-none-any.whl.metadata (3.5 kB)
Requirement already satisfied: urllib3<2 in /Users/rmann/.espressif/python_env/idf5.2_py3.12_env/lib/python3.12/site-packages (from -r /Users/rmann/Projects/Personal/ESP32-C6/esp/esp-matter/requirements.txt (line 8)) (1.26.19)
Collecting esp-matter-mfg-tool (from -r /Users/rmann/Projects/Personal/ESP32-C6/esp/esp-matter/requirements.txt (line 11))
  Downloading esp_matter_mfg_tool-1.0.2-py3-none-any.whl.metadata (1.9 kB)
Collecting MarkupSafe>=2.0 (from jinja2==3.0.1->-r /Users/rmann/Projects/Personal/ESP32-C6/esp/esp-matter/requirements.txt (line 5))
  Downloading MarkupSafe-2.1.5-cp312-cp312-macosx_10_9_universal2.whl.metadata (3.0 kB)
Requirement already satisfied: bitarray>=2.6.0 in /Users/rmann/.espressif/python_env/idf5.2_py3.12_env/lib/python3.12/site-packages (from esp-matter-mfg-tool->-r /Users/rmann/Projects/Personal/ESP32-C6/esp/esp-matter/requirements.txt (line 11)) (2.9.2)
Collecting cryptography==36.0.0 (from esp-matter-mfg-tool->-r /Users/rmann/Projects/Personal/ESP32-C6/esp/esp-matter/requirements.txt (line 11))
  Downloading cryptography-36.0.0-cp36-abi3-macosx_11_0_arm64.whl.metadata (5.4 kB)
Collecting cffi==1.15.0 (from esp-matter-mfg-tool->-r /Users/rmann/Projects/Personal/ESP32-C6/esp/esp-matter/requirements.txt (line 11))
  Downloading cffi-1.15.0.tar.gz (484 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 484.1/484.1 kB 7.4 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting future==0.18.2 (from esp-matter-mfg-tool->-r /Users/rmann/Projects/Personal/ESP32-C6/esp/esp-matter/requirements.txt (line 11))
  Downloading future-0.18.2.tar.gz (829 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 829.2/829.2 kB 27.4 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting pycparser==2.21 (from esp-matter-mfg-tool->-r /Users/rmann/Projects/Personal/ESP32-C6/esp/esp-matter/requirements.txt (line 11))
  Downloading pycparser-2.21-py2.py3-none-any.whl.metadata (1.1 kB)
Collecting pypng==0.0.21 (from esp-matter-mfg-tool->-r /Users/rmann/Projects/Personal/ESP32-C6/esp/esp-matter/requirements.txt (line 11))
  Downloading pypng-0.0.21-py3-none-any.whl.metadata (12 kB)
Collecting PyQRCode==1.2.1 (from esp-matter-mfg-tool->-r /Users/rmann/Projects/Personal/ESP32-C6/esp/esp-matter/requirements.txt (line 11))
  Downloading PyQRCode-1.2.1.zip (41 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.9/41.9 kB 4.6 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting python-stdnum==1.18 (from esp-matter-mfg-tool->-r /Users/rmann/Projects/Personal/ESP32-C6/esp/esp-matter/requirements.txt (line 11))
  Downloading python_stdnum-1.18-py2.py3-none-any.whl.metadata (17 kB)
Collecting esp-secure-cert-tool==1.0.1 (from esp-matter-mfg-tool->-r /Users/rmann/Projects/Personal/ESP32-C6/esp/esp-matter/requirements.txt (line 11))
  Downloading esp-secure-cert-tool-1.0.1.tar.gz (19 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting ecdsa==0.18.0 (from esp-matter-mfg-tool->-r /Users/rmann/Projects/Personal/ESP32-C6/esp/esp-matter/requirements.txt (line 11))
  Downloading ecdsa-0.18.0-py2.py3-none-any.whl.metadata (29 kB)
Requirement already satisfied: six>=1.9.0 in /Users/rmann/.espressif/python_env/idf5.2_py3.12_env/lib/python3.12/site-packages (from ecdsa==0.18.0->esp-matter-mfg-tool->-r /Users/rmann/Projects/Personal/ESP32-C6/esp/esp-matter/requirements.txt (line 11)) (1.16.0)
Downloading lark-1.1.2-py2.py3-none-any.whl (104 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 104.7/104.7 kB 10.5 MB/s eta 0:00:00
Downloading Jinja2-3.0.1-py3-none-any.whl (133 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.7/133.7 kB 9.8 MB/s eta 0:00:00
Downloading esp_matter_mfg_tool-1.0.2-py3-none-any.whl (35 kB)
Downloading cryptography-36.0.0-cp36-abi3-macosx_11_0_arm64.whl (2.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 18.6 MB/s eta 0:00:00
Downloading ecdsa-0.18.0-py2.py3-none-any.whl (142 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 142.9/142.9 kB 14.3 MB/s eta 0:00:00
Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.7/118.7 kB 10.9 MB/s eta 0:00:00
Downloading pypng-0.0.21-py3-none-any.whl (48 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.1/48.1 kB 4.3 MB/s eta 0:00:00
Downloading python_stdnum-1.18-py2.py3-none-any.whl (1.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 19.8 MB/s eta 0:00:00
Downloading MarkupSafe-2.1.5-cp312-cp312-macosx_10_9_universal2.whl (18 kB)
Building wheels for collected packages: cffi, esp-secure-cert-tool, future, PyQRCode
  Building wheel for cffi (pyproject.toml) ... done
  Created wheel for cffi: filename=cffi-1.15.0-cp312-cp312-macosx_14_0_arm64.whl size=174573 sha256=04c5f249094366bc9de205cd91c0328c0d7dfab2832612bf635c56d6d570b037
  Stored in directory: /Users/rmann/Library/Caches/pip/wheels/b9/d6/15/0950847bf7d74ea5f0380b8b23a1d81b45bdf48488b4b8237a
  Building wheel for esp-secure-cert-tool (pyproject.toml) ... done
  Created wheel for esp-secure-cert-tool: filename=esp_secure_cert_tool-1.0.1-py3-none-any.whl size=21492 sha256=4bcceaa5d07f7acde93056a842497e834532b72f912db54a7a6f7f1254151a78
  Stored in directory: /Users/rmann/Library/Caches/pip/wheels/4b/cb/91/eab28142eec270b1541e6ee675431f93fa135a98c64961b03c
  Building wheel for future (pyproject.toml) ... done
  Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=491056 sha256=a878dd4bac86724ba63775c6a26d7a732b38df86f9975a02d239b58e5bc092fd
  Stored in directory: /Users/rmann/Library/Caches/pip/wheels/f3/27/6f/379220bdb8dccf450313f6613ef82e2a3d4093ec8e54aa4103
  Building wheel for PyQRCode (pyproject.toml) ... done
  Created wheel for PyQRCode: filename=PyQRCode-1.2.1-py3-none-any.whl size=36225 sha256=71de20d1767f920149119913061fd867f85ccef19f98bbec6f32228ce10f0056
  Stored in directory: /Users/rmann/Library/Caches/pip/wheels/38/e3/f8/8940ce720556414aefb861a143102b931d8327f6eaea5733e8
Successfully built cffi esp-secure-cert-tool future PyQRCode
Installing collected packages: stringcase, python-stdnum, PyQRCode, pypng, lark, pycparser, MarkupSafe, future, ecdsa, jinja2, cffi, cryptography, esp-secure-cert-tool, esp-matter-mfg-tool
  Attempting uninstall: pycparser
    Found existing installation: pycparser 2.22
    Uninstalling pycparser-2.22:
      Successfully uninstalled pycparser-2.22
  Attempting uninstall: ecdsa
    Found existing installation: ecdsa 0.19.0
    Uninstalling ecdsa-0.19.0:
      Successfully uninstalled ecdsa-0.19.0
  Attempting uninstall: cffi
    Found existing installation: cffi 1.16.0
    Uninstalling cffi-1.16.0:
      Successfully uninstalled cffi-1.16.0
  Attempting uninstall: cryptography
    Found existing installation: cryptography 41.0.7
    Uninstalling cryptography-41.0.7:
      Successfully uninstalled cryptography-41.0.7
Successfully installed MarkupSafe-2.1.5 PyQRCode-1.2.1 cffi-1.15.0 cryptography-36.0.0 ecdsa-0.18.0 esp-matter-mfg-tool-1.0.2 esp-secure-cert-tool-1.0.1 future-0.18.2 jinja2-3.0.1 lark-1.1.2 pycparser-2.21 pypng-0.0.21 python-stdnum-1.18 stringcase-1.2.0

[notice] A new release of pip is available: 24.0 -> 24.2
[notice] To update, run: pip install --upgrade pip
All done! You can now run:

  . ./export.sh

Not sure why main doesn’t include the changes, but at least this works. Hopefully I'll find some time to actually build an example.

[sj82516]

@JetForMe May I ask what is your mac os version ?
I followed your comments but encountered this error

ninja: Entering directory `/Users/yuanchieh-at/Desktop/esp-matter/connectedhomeip/connectedhomeip/out/host'
[3/4] ld ./chip-cert
FAILED: chip-cert chip-cert.map
g++ -target arm64-apple-macos11.0 -O0 -fPIC -Werror -fdiagnostics-color -Wl,-dead_strip -Wl,-map,./chip-cert.map @./chip-cert.rsp -o ./chip-cert
ld: multiple errors: archive member '/' not a mach-o file in '/Users/yuanchieh-at/Desktop/esp-matter/connectedhomeip/connectedhomeip/out/host/obj/src/crypto/lib/libChipCrypto.a'; archive member '/' not a mach-o file i .......

Do you have any idea about that ?

[JetForMe]

Do you have any idea about that ?

Huh, not really. It seems to have failed to build libChipCrypto.a correctly, but I couldn't say why. I'm on macOS 14.5, with Xcode 16b5 (although when I tried this it was probably b2). I would guess the g++ invoked is whatever GCC the Espressif toolchain builds for RISC-V, so I would assume that’s the same for you and me, but I don’t know enough to quickly check, and can’t spend the time to poke through their toolchain right now.

[sj82516]

I see. Thanks for your reply @JetForMe

[shubhamdp]

Not sure why main doesn’t include the changes, but at least this works. Hopefully I'll find some time to actually build an example.

@JetForMe We are in the process of updating the connectedhomeip submodule, and it will be done soon...

[dhrishi]

The connectedhomeip submodule is updated