pyo3-build-config doesn't rebuild when virtualenv Python version changed

[messense]

Bug Description

When recreating a virtualenv to a different Python version/implementation, pyo3-build-config doesn't rebuild which can cause undefined symobl: xxx issue on import.

Steps to Reproduce

root@ubuntu:~# maturin new pyo3-repro
✔ 🤷 Which kind of bindings to use? · pyo3
  ✨ Done! New project created pyo3-repro
root@ubuntu:~# cd pyo3-repro/
root@ubuntu:~/pyo3-repro# virtualenv -p pypy3 venv
created virtual environment PyPy3.8.13.final.0-64 in 1736ms
  creator PyPy3Posix(dest=/root/pyo3-repro/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==22.2.2, setuptools==65.4.1, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
root@ubuntu:~/pyo3-repro# . venv/bin/activate
(venv) root@ubuntu:~/pyo3-repro# maturin develop
🔗 Found pyo3 bindings
🐍 Found PyPy 3.8 at /root/pyo3-repro/venv/bin/python
   Compiling target-lexicon v0.12.5
   Compiling autocfg v1.1.0
   Compiling proc-macro2 v1.0.47
   Compiling once_cell v1.16.0
   Compiling quote v1.0.21
   Compiling libc v0.2.137
   Compiling unicode-ident v1.0.5
   Compiling syn v1.0.103
   Compiling parking_lot_core v0.9.4
   Compiling smallvec v1.10.0
   Compiling scopeguard v1.1.0
   Compiling cfg-if v1.0.0
   Compiling unindent v0.1.10
   Compiling indoc v1.0.7
   Compiling lock_api v0.4.9
   Compiling memoffset v0.6.5
   Compiling pyo3-build-config v0.17.3
   Compiling parking_lot v0.12.1
   Compiling pyo3-ffi v0.17.3
   Compiling pyo3 v0.17.3
   Compiling pyo3-macros-backend v0.17.3
   Compiling pyo3-macros v0.17.3
   Compiling pyo3-repro v0.1.0 (/root/pyo3-repro)
    Finished dev [unoptimized + debuginfo] target(s) in 34.18s
📦 Built wheel for PyPy 3.8 to /tmp/.tmpJbZYXj/pyo3_repro-0.1.0-pp38-pypy38_pp73-linux_x86_64.whl
🛠  Installed pyo3-repro-0.1.0
(venv) root@ubuntu:~/pyo3-repro# deactivate
root@ubuntu:~/pyo3-repro# rm -rf venv/
root@ubuntu:~/pyo3-repro# virtualenv -p python3.10 venv
created virtual environment CPython3.10.6.final.0-64 in 1529ms
  creator CPython3Posix(dest=/root/pyo3-repro/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==22.2.2, setuptools==65.4.1, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
root@ubuntu:~/pyo3-repro# . venv/bin/activate
(venv) root@ubuntu:~/pyo3-repro# maturin develop
🔗 Found pyo3 bindings
🐍 Found CPython 3.10 at /root/pyo3-repro/venv/bin/python
    Finished dev [unoptimized + debuginfo] target(s) in 0.05s
📦 Built wheel for CPython 3.10 to /tmp/.tmp7JKmvP/pyo3_repro-0.1.0-cp310-cp310-linux_x86_64.whl
🛠  Installed pyo3-repro-0.1.0
(venv) root@ubuntu:~/pyo3-repro# python
Python 3.10.6 (main, Nov  2 2022, 18:53:38) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyo3_repro
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/pyo3-repro/venv/lib/python3.10/site-packages/pyo3_repro/__init__.py", line 1, in <module>
    from .pyo3_repro import *
ImportError: /root/pyo3-repro/venv/lib/python3.10/site-packages/pyo3_repro/pyo3_repro.cpython-310-x86_64-linux-gnu.so: undefined symbol: PyPyTuple_Size
(venv) root@ubuntu:~/pyo3-repro# cat target/debug/build/pyo3-build-config-492403e83d77b89c/out/pyo3-build-config.txt
implementation=PyPy
version=3.8
shared=true
abi3=false
lib_name=pypy3-c
lib_dir=/usr/bin
executable=/root/pyo3-repro/venv/bin/python
pointer_width=64
build_flags=
suppress_build_script_link_lines=false

Backtrace

No response

Your operating system and version

Ubuntu 22.04

Your Python version (python --version)

PyPy 3.8 & Python 3.10

Your Rust version (rustc --version)

rustc 1.65.0

Your PyO3 version

0.17.3

How did you install python? Did you use a virtualenv?

apt, use a virtualenv

Additional Info

No response

[messense]

Note that this is harmless in local environment since you can remove target directory easily, but it's a headache for CI cache.

[davidhewitt]

Hmm, interesting problem.

I think the only option would be to give ourselves another environment variable to control it. Maybe something like PYO3_CONFIG_CACHE, if it's set, only rerun pyo3-build-config when it changes?

[messense]

I think we can add PYO3_PYTHON_VERSION and PYO3_PYTHON_IMPLEMENTATION, we already have similar ones for cross compiling. Then we can set them through maturin and setuptools-rust.

pyo3/pyo3-build-config/src/impl_.rs

Lines 827 to 830 in 6746ff8

	/// `PYO3_CROSS_PYTHON_VERSION`
	pyo3_cross_python_version: Option<OsString>,
	/// `PYO3_CROSS_PYTHON_IMPLEMENTATION`
	pyo3_cross_python_implementation: Option<OsString>,

It works well with a local patch:

root@ubuntu:~/pyo3-repro# virtualenv -p pypy3 venv
created virtual environment PyPy3.8.13.final.0-64 in 519ms
  creator PyPy3Posix(dest=/root/pyo3-repro/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==22.2.2, setuptools==65.4.1, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
root@ubuntu:~/pyo3-repro# . venv/bin/activate
(venv) root@ubuntu:~/pyo3-repro# PYO3_PYTHON_VERSION=3.8.13 PYO3_PYTHON_IMPLEMENTATION=pypy maturin develop
🔗 Found pyo3 bindings
🐍 Found PyPy 3.8 at /root/pyo3-repro/venv/bin/python
   Compiling pyo3-build-config v0.17.3 (/root/code/pyo3/pyo3-build-config)
   Compiling pyo3-macros-backend v0.17.3 (/root/code/pyo3/pyo3-macros-backend)
   Compiling pyo3-ffi v0.17.3 (/root/code/pyo3/pyo3-ffi)
   Compiling pyo3 v0.17.3 (/root/code/pyo3)
   Compiling pyo3-macros v0.17.3 (/root/code/pyo3/pyo3-macros)
   Compiling pyo3-repro v0.1.0 (/root/pyo3-repro)
    Finished dev [unoptimized + debuginfo] target(s) in 25.88s
📦 Built wheel for PyPy 3.8 to /tmp/.tmpjNyj0F/pyo3_repro-0.1.0-pp38-pypy38_pp73-linux_x86_64.whl
🛠  Installed pyo3-repro-0.1.0
(venv) root@ubuntu:~/pyo3-repro# python3
Python 3.8.13 (7.3.9+dfsg-1, Apr 01 2022, 21:41:47)
[PyPy 7.3.9 with GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>> import pyo3_repro
>>>> exit()
(venv) root@ubuntu:~/pyo3-repro# deactivate
root@ubuntu:~/pyo3-repro# rm -rf venv/
root@ubuntu:~/pyo3-repro# virtualenv -p python3.10 venv
created virtual environment CPython3.10.6.final.0-64 in 424ms
  creator CPython3Posix(dest=/root/pyo3-repro/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==22.2.2, setuptools==65.4.1, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
root@ubuntu:~/pyo3-repro# . venv/bin/activate
(venv) root@ubuntu:~/pyo3-repro# PYO3_PYTHON_VERSION=3.10.6 PYO3_PYTHON_IMPLEMENTATION=cpython maturin develop
🔗 Found pyo3 bindings
🐍 Found CPython 3.10 at /root/pyo3-repro/venv/bin/python
   Compiling pyo3-build-config v0.17.3 (/root/code/pyo3/pyo3-build-config)
   Compiling pyo3-ffi v0.17.3 (/root/code/pyo3/pyo3-ffi)
   Compiling pyo3 v0.17.3 (/root/code/pyo3)
   Compiling pyo3-repro v0.1.0 (/root/pyo3-repro)
    Finished dev [unoptimized + debuginfo] target(s) in 14.22s
📦 Built wheel for CPython 3.10 to /tmp/.tmpVAZ0Qs/pyo3_repro-0.1.0-cp310-cp310-linux_x86_64.whl
🛠  Installed pyo3-repro-0.1.0
(venv) root@ubuntu:~/pyo3-repro# python3
Python 3.10.6 (main, Nov  2 2022, 18:53:38) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyo3_repro
>>> exit()