sysconfig data inconsistencies caused by the disabling the site initialization on virtual environments

[FFY00]

Bug report

Bug description:

Currently, running the Python interpreter with -S (disabling the site module initialization) causes several inconsistencies sysconfig.get_paths() depends on whether sysconfig was imported before or after the site initialization

Currently, sysconfig calculates the paths at import time and caches them. This results in it resulting incorrect paths if the site initialization happens after sysconfig was imported.

$ python -S
Python 3.14.0a1+ experimental free-threading build (heads/main:de0d5c6e2e1, Oct 23 2024, 15:37:46) [GCC 14.2.1 20240910] on linux
>>> import sysconfig
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.14t', 'platstdlib': '/usr/local/lib/python3.14t', 'purelib': '/usr/local/lib/python3.14t/site-packages', 'platlib': '/usr/local/lib/python3.14t/site-packages', 'include': '/usr/local/include/python3.14td', 'platinclude': '/usr/local/include/python3.14td', 'scripts': '/usr/local/bin', 'data': '/usr/local'}
>>> import site
>>> site.main()
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.14t', 'platstdlib': '/usr/local/lib/python3.14t', 'purelib': '/usr/local/lib/python3.14t/site-packages', 'platlib': '/usr/local/lib/python3.14t/site-packages', 'include': '/usr/local/include/python3.14td', 'platinclude': '/usr/local/include/python3.14td', 'scripts': '/usr/local/bin', 'data': '/usr/local'}

Expected output when the site module has been initialized:

$ python
Python 3.14.0a1+ experimental free-threading build (heads/main:de0d5c6e2e1, Oct 23 2024, 15:37:46) [GCC 14.2.1 20240910] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.14t', 'platstdlib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t', 'purelib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t/site-packages', 'platlib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t/site-packages', 'include': '/usr/local/include/python3.14td', 'platinclude': '/usr/local/include/python3.14td', 'scripts': '/home/anubis/.virtualenvs/test-sysconfig-paths/bin', 'data': '/home/anubis/.virtualenvs/test-sysconfig-paths'}

And just to make sure this is not dependent on the interpreter being run with -S, and rather on the sysconfig import time.

$ python -S
Python 3.14.0a1+ experimental free-threading build (heads/main:de0d5c6e2e1, Oct 23 2024, 15:37:46) [GCC 14.2.1 20240910] on linux
>>> import site
>>> site.main()
>>> import sysconfig
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.14t', 'platstdlib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t', 'purelib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t/site-packages', 'platlib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t/site-packages', 'include': '/usr/local/include/python3.14td', 'platinclude': '/usr/local/include/python3.14td', 'scripts': '/home/anubis/.virtualenvs/test-sysconfig-paths/bin', 'data': '/home/anubis/.virtualenvs/test-sysconfig-paths'}

CPython versions tested on:

3.9, 3.10, 3.11, 3.12, 3.13, 3.14

Operating systems tested on:

Linux

Linked PRs

• GH-126789: fix some sysconfig data on late site initializations #126812
• [3.13] GH-126789: fix some sysconfig data on late site initializations #126918
• [3.12] GH-126789: fix some sysconfig data on late site initializations #126919
• gh-126789: Correct sysconfig test exclusions for iOS and Android. #126941
• [3.13] gh-126789: Correct sysconfig test exclusions for iOS and Android. (GH-126941) #126950
• [3.12] gh-126789: Correct sysconfig test exclusions for iOS and Android. (GH-126941) #126959

[FFY00]

Things are a bit messy, as the virtual environment mechanism is technically part of the site customization. In reality, the pyvenv.cfg detection is implemented on the interpreter initialization, but the customization itself is done by site. I feel like this should probably be reconsidered.

Anyway, to match this model, if site has been initialized, sysconfig.get_paths() should always return the virtual environment paths. So, the correct result should be as follows:

$ python -S
Python 3.14.0a1+ experimental free-threading build (heads/main:de0d5c6e2e1, Oct 23 2024, 15:37:46) [GCC 14.2.1 20240910] on linux
>>> import sysconfig
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.14t', 'platstdlib': '/usr/local/lib/python3.14t', 'purelib': '/usr/local/lib/python3.14t/site-packages', 'platlib': '/usr/local/lib/python3.14t/site-packages', 'include': '/usr/local/include/python3.14td', 'platinclude': '/usr/local/include/python3.14td', 'scripts': '/usr/local/bin', 'data': '/usr/local'}
>>> import site
>>> site.main()
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.14t', 'platstdlib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t', 'purelib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t/site-packages', 'platlib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t/site-packages', 'include': '/usr/local/include/python3.14td', 'platinclude': '/usr/local/include/python3.14td', 'scripts': '/home/anubis/.virtualenvs/test-sysconfig-paths/bin', 'data': '/home/anubis/.virtualenvs/test-sysconfig-paths'}

[FFY00]

Correction: sysconfig does not cache the sysconfig.get_paths(), it caches sysconfig.get_config_vars(), which is used to calculate the paths.

[FFY00]

While fixing the main bug, I have noticed that the prefix and exec_prefix variables get overwritten by _init_posix, so on virtual environments sysconfig.get_config_vars('prefix') and sys.prefix will have different values, the same goes for exec_prefix.

IMO this is worth to fix, but we probably shouldn't backport it as it is a significant behavior change, and there's a possibility it would break existing user code.

[FFY00]

Correction: sysconfig does not cache the sysconfig.get_paths(), it caches sysconfig.get_config_vars(), which is used to calculate the paths.

Just some more background on this, sysconfig.get_paths() uses an incomplete copy of the config vars directory to expand the path placeholder, which is what threw me off initially. This makes it a little bit more tricky to fix the issue, as we need to make sure we update the config vars cache without creating a recursive loop, but wasn't too difficult to get the implementation right.

[hugovk]

macOS: 3.14.0a1

Hmm, I can't reproduce this in macOS with the first 3.14 alpha. I get the same paths in all cases:

❯ python3.14 -S
Python 3.14.0a1 (v3.14.0a1:8cdaca8b25e, Oct 15 2024, 15:51:57) [Clang 16.0.0 (clang-1600.0.26.3)] on darwin
>>> import sysconfig
>>> sysconfig.get_paths()
{'stdlib': '/Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14', 'platstdlib': '/Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14', 'purelib': '/Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14/site-packages', 'platlib': '/Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14/site-packages', 'include': '/Library/Frameworks/Python.framework/Versions/3.14/include/python3.14', 'platinclude': '/Library/Frameworks/Python.framework/Versions/3.14/include/python3.14', 'scripts': '/Library/Frameworks/Python.framework/Versions/3.14/bin', 'data': '/Library/Frameworks/Python.framework/Versions/3.14'}
>>> import site
>>> site.main()
>>> sysconfig.get_paths()
{'stdlib': '/Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14', 'platstdlib': '/Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14', 'purelib': '/Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14/site-packages', 'platlib': '/Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14/site-packages', 'include': '/Library/Frameworks/Python.framework/Versions/3.14/include/python3.14', 'platinclude': '/Library/Frameworks/Python.framework/Versions/3.14/include/python3.14', 'scripts': '/Library/Frameworks/Python.framework/Versions/3.14/bin', 'data': '/Library/Frameworks/Python.framework/Versions/3.14'}

❯ python3.14
Python 3.14.0a1 (v3.14.0a1:8cdaca8b25e, Oct 15 2024, 15:51:57) [Clang 16.0.0 (clang-1600.0.26.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_paths()
{'stdlib': '/Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14', 'platstdlib': '/Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14', 'purelib': '/Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14/site-packages', 'platlib': '/Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14/site-packages', 'include': '/Library/Frameworks/Python.framework/Versions/3.14/include/python3.14', 'platinclude': '/Library/Frameworks/Python.framework/Versions/3.14/include/python3.14', 'scripts': '/Library/Frameworks/Python.framework/Versions/3.14/bin', 'data': '/Library/Frameworks/Python.framework/Versions/3.14'}

❯ python3.14 -S
Python 3.14.0a1 (v3.14.0a1:8cdaca8b25e, Oct 15 2024, 15:51:57) [Clang 16.0.0 (clang-1600.0.26.3)] on darwin
>>> import site
>>> site.main()
>>> import sysconfig
>>> sysconfig.get_paths()
{'stdlib': '/Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14', 'platstdlib': '/Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14', 'purelib': '/Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14/site-packages', 'platlib': '/Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14/site-packages', 'include': '/Library/Frameworks/Python.framework/Versions/3.14/include/python3.14', 'platinclude': '/Library/Frameworks/Python.framework/Versions/3.14/include/python3.14', 'scripts': '/Library/Frameworks/Python.framework/Versions/3.14/bin', 'data': '/Library/Frameworks/Python.framework/Versions/3.14'}

Debian: 3.14.0a1

Similarly when testing the first alpha on Linux using Docker:

❯ docker run -it python:3.14.0a1 /bin/bash
root@85391c98ea86:/# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

root@85391c98ea86:/# python -S
Python 3.14.0a1 (main, Nov 13 2024, 14:15:00) [GCC 12.2.0] on linux
>>> import sysconfig
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.14', 'platstdlib': '/usr/local/lib/python3.14', 'purelib': '/usr/local/lib/python3.14/site-packages', 'platlib': '/usr/local/lib/python3.14/site-packages', 'include': '/usr/local/include/python3.14', 'platinclude': '/usr/local/include/python3.14', 'scripts': '/usr/local/bin', 'data': '/usr/local'}
>>> import site
>>> site.main()
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.14', 'platstdlib': '/usr/local/lib/python3.14', 'purelib': '/usr/local/lib/python3.14/site-packages', 'platlib': '/usr/local/lib/python3.14/site-packages', 'include': '/usr/local/include/python3.14', 'platinclude': '/usr/local/include/python3.14', 'scripts': '/usr/local/bin', 'data': '/usr/local'}

root@85391c98ea86:/# python
Python 3.14.0a1 (main, Nov 13 2024, 14:15:00) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.14', 'platstdlib': '/usr/local/lib/python3.14', 'purelib': '/usr/local/lib/python3.14/site-packages', 'platlib': '/usr/local/lib/python3.14/site-packages', 'include': '/usr/local/include/python3.14', 'platinclude': '/usr/local/include/python3.14', 'scripts': '/usr/local/bin', 'data': '/usr/local'}

Python 3.14.0a1 (main, Nov 13 2024, 14:15:00) [GCC 12.2.0] on linux
>>> import site
>>> site.main()
>>> import sysconfig
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.14', 'platstdlib': '/usr/local/lib/python3.14', 'purelib': '/usr/local/lib/python3.14/site-packages', 'platlib': '/usr/local/lib/python3.14/site-packages', 'include': '/usr/local/include/python3.14', 'platinclude': '/usr/local/include/python3.14', 'scripts': '/usr/local/bin', 'data': '/usr/local'}

macOS: main

And I also get the same set of paths after the PR has been merged, so no regression:

❯ ./python.exe -S
Python 3.14.0a1+ (heads/main:3938fd60c0c, Nov 18 2024, 11:16:39) [Clang 16.0.0 (clang-1600.0.26.4)] on darwin
>>> import sysconfig
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.14', 'platstdlib': '/usr/local/lib/python3.14', 'purelib': '/usr/local/lib/python3.14/site-packages', 'platlib': '/usr/local/lib/python3.14/site-packages', 'include': '/Users/hugo/github/python/cpython/main/Include', 'platinclude': '/Users/hugo/github/python/cpython/main', 'scripts': '/usr/local/bin', 'data': '/usr/local', 'headers': '/usr/local/include/python3.14'}
>>> import site
>>> site.main()
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.14', 'platstdlib': '/usr/local/lib/python3.14', 'purelib': '/usr/local/lib/python3.14/site-packages', 'platlib': '/usr/local/lib/python3.14/site-packages', 'include': '/Users/hugo/github/python/cpython/main/Include', 'platinclude': '/Users/hugo/github/python/cpython/main', 'scripts': '/usr/local/bin', 'data': '/usr/local', 'headers': '/usr/local/include/python3.14'}

❯ ./python.exe
Python 3.14.0a1+ (heads/main:3938fd60c0c, Nov 18 2024, 11:16:39) [Clang 16.0.0 (clang-1600.0.26.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.14', 'platstdlib': '/usr/local/lib/python3.14', 'purelib': '/usr/local/lib/python3.14/site-packages', 'platlib': '/usr/local/lib/python3.14/site-packages', 'include': '/Users/hugo/github/python/cpython/main/Include', 'platinclude': '/Users/hugo/github/python/cpython/main', 'scripts': '/usr/local/bin', 'data': '/usr/local', 'headers': '/usr/local/include/python3.14'}

❯ ./python.exe -S
Python 3.14.0a1+ (heads/main:3938fd60c0c, Nov 18 2024, 11:16:39) [Clang 16.0.0 (clang-1600.0.26.4)] on darwin
>>> import site
>>> site.main()
>>> import sysconfig
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.14', 'platstdlib': '/usr/local/lib/python3.14', 'purelib': '/usr/local/lib/python3.14/site-packages', 'platlib': '/usr/local/lib/python3.14/site-packages', 'include': '/Users/hugo/github/python/cpython/main/Include', 'platinclude': '/Users/hugo/github/python/cpython/main', 'scripts': '/usr/local/bin', 'data': '/usr/local', 'headers': '/usr/local/include/python3.14'} 

[FFY00]

@hugovk, sorry I just say this now. I should have mentioned it in the issue text, but as per the title this only affects virtual environments.

The issue is that currently site is responsible for the virtual environment initialization (setting sys.prefix and sys.exec_prefix), so when you run a virtual environment with -S, sys.prefix and sys.exec_prefix point to the base installation, meaning that, canonically, you are not in a virtual environment. In site.main(), sys.prefix and sys.exec_prefix are updated to point to the virtual environment.
If the sysconfig.get_config_vars() cache, which is used when calculating sysconfig.get_paths(), is created before the site initialization, it still holds the old prefix values, resulting in sysconfig reporting different paths depending on whether the cache was generated before or after the site initialization.
The issue only affects virtual environments, because that's the only case where sys.prefix and sys.exec_prefix change after initialization.

I hope that helps clarify things. Let me know if you have any more questions.

[hugovk]

Thanks for the clarification!