Python tests fail in CI on macOS

[scpeters]

Environment

• OS Version: macOS
• Source or binary build? binary builds of gz-sim7, gz-sim8

Description

• Expected behavior: python tests should pass
• Actual behavior: python tests fail

gz-sim7: from https://build.osrfoundation.org/job/gz_sim-ci-gz-sim7-homebrew-amd64/4/

test 301
        Start 301: testFixture_TEST

301: Test command: /usr/local/Frameworks/Python.framework/Versions/3.12/bin/python3.12 "/Users/jenkins/jenkins-agent/workspace/gz_sim-ci-gz-sim7-homebrew-amd64/gz-sim/python/test/testFixture_TEST.py"
301: Working Directory: /Users/jenkins/jenkins-agent/workspace/gz_sim-ci-gz-sim7-homebrew-amd64/build/python
301: Environment variables: 
301:  PYTHONPATH=/usr/local/Cellar/gz-sim7/HEAD/lib/python/
301:  LD_LIBRARY_PATH=/usr/local/Cellar/gz-sim7/HEAD/lib:
301: Test timeout computed to be: 1500
301: Traceback (most recent call last):
301:   File "/Users/jenkins/jenkins-agent/workspace/gz_sim-ci-gz-sim7-homebrew-amd64/gz-sim/python/test/testFixture_TEST.py", line 21, in <module>
301:     from gz.math7 import Vector3d
301: ModuleNotFoundError: No module named 'gz.math7'
301/301 Test #301: testFixture_TEST ........................................***Failed    1.07 sec

gz-sim8: from https://build.osrfoundation.org/job/gz_sim-ci-gz-sim8-homebrew-amd64/4/

...
        Start 320: world_TEST

320: Test command: /usr/local/Frameworks/Python.framework/Versions/3.12/bin/python3.12 "/Users/jenkins/jenkins-agent/workspace/gz_sim-ci-gz-sim8-homebrew-amd64/gz-sim/python/test/world_TEST.py"
320: Working Directory: /Users/jenkins/jenkins-agent/workspace/gz_sim-ci-gz-sim8-homebrew-amd64/build/python
320: Environment variables: 
320:  PYTHONPATH=/usr/local/Cellar/gz-sim8/HEAD/lib/python/
320:  LD_LIBRARY_PATH=/usr/local/Cellar/gz-sim8/HEAD/lib:
320: Test timeout computed to be: 1500
320: Traceback (most recent call last):
320:   File "/Users/jenkins/jenkins-agent/workspace/gz_sim-ci-gz-sim8-homebrew-amd64/gz-sim/python/test/world_TEST.py", line 21, in <module>
320:     from gz_test_deps.math import SphericalCoordinates, Vector3d
320:   File "/Users/jenkins/jenkins-agent/workspace/gz_sim-ci-gz-sim8-homebrew-amd64/gz-sim/python/test/gz_test_deps/math.py", line 1, in <module>
320:     from gz.math7 import *
320: ModuleNotFoundError: No module named 'gz.math7'
320/320 Test #320: world_TEST ..............................................***Failed    0.76 sec
...
The following tests FAILED:
	305 - INTEGRATION_python_system_loader (Failed)
	313 - actor_TEST (Failed)
	314 - joint_TEST (Failed)
	315 - light_TEST (Failed)
	316 - link_TEST (Failed)
	318 - sensor_TEST (Failed)
	319 - testFixture_TEST (Failed)
	320 - world_TEST (Failed)

Steps to reproduce

1. Run a macOS CI build

Output

[scpeters]

This may be related to which version of python3 is found by cmake and used to run the tests. Currently the gz-math and sdformat python bindings are built and installed for [email protected], but gz-sim appears to find [email protected], perhaps because it is a dependency of protobuf?

[scpeters]

Looking at the protobuf formula in homebrew-core, I see that it supports multiple python versions:

• https://github.com/Homebrew/homebrew-core/blob/master/Formula/p/protobuf.rb#L25-L27
• https://github.com/Homebrew/homebrew-core/blob/master/Formula/p/protobuf.rb#L33-L37
• https://github.com/Homebrew/homebrew-core/blob/master/Formula/p/protobuf.rb#L64-L75

[Crola1702]

Do you think this is related to PythonSystemLoaderTest.LoadMultipleSystems test failure in homebrew gz-sim8?

[azeey]

@scpeters, if protobuf depends on multiple python versions, is there a way to install pin a specific version of python for all our formulas?

[Crola1702]

This is also happening for ign-gazebo6 (reference build).

This was reported in #1887

Log output:

test 273
        Start 273: testFixture_TEST.py

273: Test command: /usr/local/Frameworks/Python.framework/Versions/3.11/bin/python3.11 "-m" "pytest" "/Users/jenkins/jenkins-agent/workspace/gz_sim-ci-ign-gazebo6-homebrew-amd64/gz-sim/python/test/testFixture_TEST.py" "--junitxml" "/Users/jenkins/jenkins-agent/workspace/gz_sim-ci-ign-gazebo6-homebrew-amd64/build/test_results/UNIT_testFixture_TEST.xml"
273: Working Directory: /Users/jenkins/jenkins-agent/workspace/gz_sim-ci-ign-gazebo6-homebrew-amd64/build/python
273: Environment variables: 
273:  PYTHONPATH=/usr/local/Cellar/gz-sim6/HEAD/lib/python/
273:  LD_LIBRARY_PATH=/usr/local/Cellar/gz-sim6/HEAD/lib:
273: Test timeout computed to be: 1500
273: ============================= test session starts ==============================
273: platform darwin -- Python 3.11.7, pytest-7.4.4, pluggy-1.3.0
273: rootdir: /Users/jenkins/jenkins-agent/workspace/gz_sim-ci-ign-gazebo6-homebrew-amd64
273: collected 0 items / 1 error
273: 
273: ==================================== ERRORS ====================================
273: ___________ ERROR collecting gz-sim/python/test/testFixture_TEST.py ____________
273: ImportError while importing test module '/Users/jenkins/jenkins-agent/workspace/gz_sim-ci-ign-gazebo6-homebrew-amd64/gz-sim/python/test/testFixture_TEST.py'.
273: Hint: make sure your test modules/packages have valid Python names.
273: Traceback:
273: /usr/local/Cellar/[email protected]/3.11.7/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/__init__.py:126: in import_module
273:     return _bootstrap._gcd_import(name[level:], package, level)
273: ../../gz-sim/python/test/testFixture_TEST.py:20: in <module>
273:     from ignition.math import Vector3d
273: E   ModuleNotFoundError: No module named 'ignition.math'
273: - generated xml file: /Users/jenkins/jenkins-agent/workspace/gz_sim-ci-ign-gazebo6-homebrew-amd64/build/test_results/UNIT_testFixture_TEST.xml -
273: =========================== short test summary info ============================
273: ERROR ../../gz-sim/python/test/testFixture_TEST.py
273: !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
273: =============================== 1 error in 0.54s ===============================
273/273 Test #273: testFixture_TEST.py .....................................***Failed    0.74 sec

[scpeters]

@scpeters, if protobuf depends on multiple python versions, is there a way to install pin a specific version of python for all our formulas?

Currently [email protected] is the most official version of homebrew python since it symlinks /usr/local/bin/python3 to python3.11, so most of our homebrew formulae explicitly depend on [email protected] and install bindings to lib/python3.11, though the gz-sim* formulae do not.

Changing the gz-sim* formulae to use [email protected] explicitly may not be enough though since [email protected] is a direct dependency of gz-sim* via gz-common* and gdal, and I have observed that for systems that have both [email protected] and [email protected] installed, cmake appears to always find version 3.12. I haven't figured out why that is the case.

As I noted in #2249 (comment), protobuf installs bindings for multiple versions of python, so that could be possible option for us, but it may be extra work.

[scpeters]

Changing the gz-sim* formulae to use [email protected] explicitly may not be enough though

I just tried updating gz-sim8 to use [email protected] explicitly, but it still creates bindings for 3.12. I'll need to look into how it finds a particular version of python.

We could also just move everything to 3.12 and declare that anyone who wants to use python bindings needs to use 3.12

[scpeters]

On a suggestion from @traversaro, in osrf/homebrew-simulation#2540 I'm setting Python3_EXECUTABLE explicitly in cmake in order to use the default python version.

[Crola1702]

Hey! I noticed the following log output in gz-sim-main and gz-launch-main:

CMake Error at /usr/local/Cellar/cmake/3.28.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Python3 (missing: Development Development.Module
  Development.Embed) (found version "3.12")
Call Stack (most recent call first):
  /usr/local/Cellar/cmake/3.28.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
  /usr/local/Cellar/cmake/3.28.2/share/cmake/Modules/FindPython/Support.cmake:3867 (find_package_handle_standard_args)
  /usr/local/Cellar/cmake/3.28.2/share/cmake/Modules/FindPython3.cmake:545 (include)
  CMakeLists.txt:151 (find_package)

I think this is related to 3.12 -> 3.11 changes, right?

[scpeters]

Hey! I noticed the following log output in gz-sim-main and gz-launch-main:

CMake Error at /usr/local/Cellar/cmake/3.28.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Python3 (missing: Development Development.Module
  Development.Embed) (found version "3.12")
Call Stack (most recent call first):
  /usr/local/Cellar/cmake/3.28.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
  /usr/local/Cellar/cmake/3.28.2/share/cmake/Modules/FindPython/Support.cmake:3867 (find_package_handle_standard_args)
  /usr/local/Cellar/cmake/3.28.2/share/cmake/Modules/FindPython3.cmake:545 (include)
  CMakeLists.txt:151 (find_package)

I think this is related to 3.12 -> 3.11 changes, right?

I think it will be fixed by gazebo-tooling/release-tools#1110, but I'm still waiting for an approval on that PR