Python Version Conflict in Jetson-Containers Tool

[hugomiralles-manta]

Description

While building the l4t-tensorflow:tf2 container using the Jetson-Containers tool, the build process fails due to a mismatch between the Python version referenced by pip and the version referenced by python3. Specifically:

• pip3 is installed and associated with Python 3.8.
• python3 points to Python 3.6, leading to a runtime inconsistency when Python modules installed by pip3 (e.g., numpy) are not available to python3.

This results in errors during package installation and subsequent Python module imports. The critical error observed is:

ModuleNotFoundError: No module named 'numpy'

Steps to Reproduce

1. Clone the jetson-containers repository.
2. Run the following command to build the l4t-tensorflow:tf2 container:

   export L4T_VERSION="32.7.5"
   export PYTHON_VERSION=3.8
   
   jetson-containers build l4t-tensorflow:tf2

3. Observe the logs for errors during the numpy installation and import test.

Observed Behavior

1. During Build:

   • pip3 successfully installs numpy using Python 3.8:

     Using pip 24.3.1 from /usr/local/lib/python3.8/dist-packages/pip (python 3.8)
     Successfully installed numpy-1.24.4
     

   • However, the python3 executable used in subsequent steps does not recognize the installed numpy module:

     Traceback (most recent call last):
       File "<string>", line 1, in <module>
     ModuleNotFoundError: No module named 'numpy'
     

2. System Configuration:

   • pip3 is tied to /usr/local/lib/python3.8.
   • python3 refers to Python 3.6 due to system defaults.

Expected Behavior

• pip3 and python3 should both point to the same Python version (in this case, Python 3.8) to ensure consistency.
• Modules installed via pip3 should be accessible to python3 without issues.

Root Cause

The build environment contains both Python 3.6 and Python 3.8, but the python3 binary defaults to Python 3.6. This leads to:

1. pip3 installing packages under Python 3.8's site-packages directory.
2. python3 (Python 3.6) not finding the installed modules.

Suggested Fix

To resolve this issue, consider one of the following solutions:

1. Update the Dockerfile or Build Script to Remove Python 3.6:
   Ensure only Python 3.8 is installed in the container. Add the following steps to the build script:

   apt-get remove -y python3.6 python3.6-dev
   update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 1
   update-alternatives --config python3
   ln -sf /usr/bin/python3.8 /usr/bin/python

2. Set Explicit Python Version for Pip:
   Adjust the script to explicitly call python3.8 for module installation and testing:

   python3.8 -m pip install <package>
   python3.8 -c "import <module>"

3. Validate Python Environment During Build:
   Include a step in the Dockerfile or build script to check that python3 and pip3 refer to the same Python version:

   python3 --version
   pip3 --version

Environment Details

• Jetson-Containers Version: Latest (as of December 2024)
• L4T Version: 32.7.5
• JetPack Version: 4.6.5
• Python Versions in Conflict: 3.6 and 3.8
• Base Docker Image: nvcr.io/nvidia/l4t-base:r32.7.1

Additional Notes

• The issue also affects dependent packages, as seen with numpy, but is likely to occur with any Python module due to the version mismatch.
• The problem might recur in other container builds where Python 3.6 and 3.8 coexist.

[johnnynunez]

Hello, tensorflow2 build is only for jetpack6.1
because it uses hermetic CUDA

[hugomiralles-manta]

Similar issue for tf1 or pytorch

[johnnynunez]

Similar issue for tf1 or pytorch

see my message.
secondly,
you can do

 PYTHON_VERSION=3.8 jetson-containers build (package)

[hugomiralles-manta]

I understood, but when I run PYTHON_VERSION=3.8 L4T_VERSION="32.7.5" jetson-containers build pytorch I have the same issue
It is not about hermetic CUDA