Failed to install pydantic-core from sources

[n-bes]

pydantic-core build script ignores virtual-enviroment:

$ pip3 install typing-extensions
Requirement already satisfied: typing-extensions in /venv/lib/python3.11/site-packages (4.12.2)

$ poetry install -v --no-root
...
     Compiling pydantic-core v2.20.0 (/tmp/tmp556rg92p/pydantic_core-2.20.0)
  error: failed to run custom build command for `pydantic-core v2.20.0 (/tmp/tmp556rg92p/pydantic_core-2.20.0)`

  Caused by:
    process didn't exit successfully: `/tmp/tmp556rg92p/pydantic_core-2.20.0/target/release/build/pydantic-core-e9fa49663ca28b67/build-script-build` (exit status: 101)
    --- stdout
    cargo:rustc-cfg=Py_3_6
    cargo:rustc-cfg=Py_3_7
    cargo:rustc-cfg=Py_3_8
    cargo:rustc-cfg=Py_3_9
    cargo:rustc-cfg=Py_3_10
    cargo:rustc-cfg=Py_3_11
    cargo:rerun-if-changed=python/pydantic_core/core_schema.py
    cargo:rerun-if-changed=generate_self_schema.py

    --- stderr
    Traceback (most recent call last):
      File "/tmp/tmp556rg92p/pydantic_core-2.20.0/generate_self_schema.py", line 19, in <module>
        from typing_extensions import TypedDict, get_args, get_origin, is_typeddict
    ModuleNotFoundError: No module named 'typing_extensions'
    thread 'main' panicked at build.rs:29:9:
    generate_self_schema.py failed with exit status: 1
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
  warning: build failed, waiting for other jobs to finish...
  💥 maturin failed
    Caused by: Failed to build a native library through cargo
    Caused by: Cargo build finished with "exit status: 101": `env -u CARGO PYO3_ENVIRONMENT_SIGNATURE="cpython-3.11-64bit" PYO3_PYTHON="/usr/bin/python3.11" PYTHON_SYS_EXECUTABLE="/usr/bin/python3.11" "cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "/tmp/tmp556rg92p/pydantic_core-2.20.0/Cargo.toml" "--release" "--lib" "--crate-type" "cdylib"`

Related issues:

• Failed to install pydantic from sources with poetry pydantic#9773
• pip works, but poetry not python-poetry/poetry#9518

[samuelcolvin]

pydantic-core/build.rs

Line 16 in a017bd9

env::var("PYTHON")

You might have some luck setting the PYTHON env var., or looking at the PR that changed that code.

My guess is the poetry is supposed to set the env var, or that's what we assumed.

I think we should probably build the core schema python file and include it in git or sdist, and just check it's right on pre-commit.

[samuelcolvin]

You shouldn't need to install typing-extensions first, it's a build requirement

pydantic-core/pyproject.toml

Line 4 in a017bd9

'typing-extensions >=4.6.0,!=4.7.0'

[dimbleby]

typing-extensions is correctly installed in the isolated build environment per PEP517, but then the pydantic-core build chooses to use a different python to execute that script.

[dimbleby]

per python-poetry/poetry#9518 (comment), this isn't a poetry thing, you can reproduce the same with python -m build

[samuelcolvin]

Yup, like the link I sent above, we use the default python3 executable, or the value of the PYTHON env var to run the script that builds the core schema python definitions.

Happy to accept a PR to improve that logic, or store that definition in version control, or pre-build it for sdist.

[dimbleby]

you do not use the default python3 executable, that would be the one in the isolated build virtual environment.

I assume that pyo3_build_config::get().executable is somehow giving an undesirable answer

[n-bes]

You shouldn't need to install typing-extensions first, it's a build requirement

I know, only showed that build order issue is not exists.

Build with PYTHON env variable works.

$ poetry install --no-root
Installing dependencies from lock file

Package operations: 4 installs, 0 updates, 0 removals

  - Installing typing-extensions (4.12.2)
  - Installing annotated-types (0.7.0)
  - Installing pydantic-core (2.14.5): Failed

  ChefBuildError

  Backend subprocess exited when trying to invoke build_wheel

  Running `maturin pep517 build-wheel -i /usr/bin/python3.11 --compatibility off`
      Updating crates.io index
   Downloading crates ...
    Downloaded equivalent v1.0.1
    Downloaded unindent v0.2.3
    Downloaded strum v0.25.0
    Downloaded tinyvec_macros v0.1.1
    Downloaded pyo3-macros v0.20.0
    Downloaded windows-targets v0.48.1
    Downloaded speedate v0.13.0
    Downloaded once_cell v1.18.0
    Downloaded num-integer v0.1.45
    Downloaded autocfg v1.1.0
    Downloaded cfg-if v1.0.0
    Downloaded indoc v2.0.4
    Downloaded heck v0.4.1
    Downloaded form_urlencoded v1.2.0
    Downloaded memoffset v0.9.0
    Downloaded static_assertions v1.1.0
    Downloaded rustversion v1.0.13
    Downloaded percent-encoding v2.3.0
    Downloaded itoa v1.0.8
    Downloaded redox_syscall v0.3.5
    Downloaded strum_macros v0.25.3
    Downloaded version_check v0.9.4
    Downloaded quote v1.0.29
    Downloaded parking_lot_core v0.9.8
    Downloaded bitflags v1.3.2
    Downloaded wasi v0.11.0+wasi-snapshot-preview1
    Downloaded lexical-parse-integer v0.8.6
    Downloaded smallvec v1.11.1
    Downloaded enum_dispatch v0.3.12
    Downloaded pyo3-build-config v0.20.0
    Downloaded target-lexicon v0.12.9
    Downloaded scopeguard v1.1.0
    Downloaded ahash v0.8.6
    Downloaded lexical-core v0.8.5
    Downloaded getrandom v0.2.10
    Downloaded ryu v1.0.14
    Downloaded proc-macro2 v1.0.69
    Downloaded uuid v1.5.0
    Downloaded unicode-ident v1.0.10
    Downloaded lock_api v0.4.10
    Downloaded parking_lot v0.12.1
    Downloaded serde_derive v1.0.190
    Downloaded indexmap v2.0.0
    Downloaded pyo3-macros-backend v0.20.0
    Downloaded num-traits v0.2.16
    Downloaded lexical-write-integer v0.8.5
    Downloaded unicode-bidi v0.3.13
    Downloaded cc v1.0.79
    Downloaded base64 v0.21.5
    Downloaded tinyvec v1.6.0
    Downloaded python3-dll-a v0.2.9
    Downloaded serde v1.0.190
    Downloaded pyo3-ffi v0.20.0
    Downloaded jiter v0.0.4
    Downloaded lexical-util v0.8.5
    Downloaded url v2.4.1
    Downloaded lexical-write-float v0.8.5
    Downloaded num-bigint v0.4.4
    Downloaded memchr v2.6.3
    Downloaded hashbrown v0.14.0
    Downloaded zerocopy v0.7.20
    Downloaded unicode-normalization v0.1.22
    Downloaded aho-corasick v1.0.2
    Downloaded serde_json v1.0.108
    Downloaded lexical-parse-float v0.8.5
    Downloaded syn v2.0.38
    Downloaded idna v0.4.0
    Downloaded regex v1.10.2
    Downloaded regex-syntax v0.8.2
    Downloaded windows_x86_64_gnullvm v0.48.0
    Downloaded windows_aarch64_gnullvm v0.48.0
    Downloaded pyo3 v0.20.0
    Downloaded zerocopy-derive v0.7.20
    Downloaded windows_x86_64_msvc v0.48.0
    Downloaded windows_aarch64_msvc v0.48.0
    Downloaded windows_i686_msvc v0.48.0
    Downloaded windows_i686_gnu v0.48.0
    Downloaded libc v0.2.147
    Downloaded windows_x86_64_gnu v0.48.0
    Downloaded regex-automata v0.4.3
  📦 Including license file "/tmp/tmph7u3k8qd/pydantic_core-2.14.5/LICENSE"
  🍹 Building a mixed python/rust project
  🔗 Found pyo3 bindings
  🐍 Found CPython 3.11 at /usr/bin/python3.11
  📡 Using build options features, bindings from pyproject.toml
     Compiling autocfg v1.1.0
     Compiling proc-macro2 v1.0.69
     Compiling quote v1.0.29
     Compiling target-lexicon v0.12.9
     Compiling unicode-ident v1.0.10
     Compiling python3-dll-a v0.2.9
     Compiling once_cell v1.18.0
     Compiling libc v0.2.147
     Compiling heck v0.4.1
     Compiling static_assertions v1.1.0
     Compiling version_check v0.9.4
     Compiling cfg-if v1.0.0
     Compiling rustversion v1.0.13
     Compiling parking_lot_core v0.9.8
     Compiling lexical-util v0.8.5
     Compiling tinyvec_macros v0.1.1
     Compiling scopeguard v1.1.0
     Compiling smallvec v1.11.1
     Compiling tinyvec v1.6.0
     Compiling serde v1.0.190
     Compiling memchr v2.6.3
     Compiling unicode-bidi v0.3.13
     Compiling regex-syntax v0.8.2
     Compiling hashbrown v0.14.0
     Compiling serde_json v1.0.108
     Compiling num-traits v0.2.16
     Compiling num-integer v0.1.45
     Compiling lock_api v0.4.10
     Compiling memoffset v0.9.0
     Compiling num-bigint v0.4.4
     Compiling pyo3-build-config v0.20.0
     Compiling syn v2.0.38
     Compiling ahash v0.8.6
     Compiling getrandom v0.2.10
     Compiling aho-corasick v1.0.2
     Compiling parking_lot v0.12.1
     Compiling lexical-write-integer v0.8.5
     Compiling lexical-parse-integer v0.8.6
     Compiling zerocopy v0.7.20
     Compiling indoc v2.0.4
     Compiling percent-encoding v2.3.0
     Compiling equivalent v1.0.1
     Compiling unindent v0.2.3
     Compiling unicode-normalization v0.1.22
     Compiling lexical-write-float v0.8.5
     Compiling lexical-parse-float v0.8.5
     Compiling form_urlencoded v1.2.0
     Compiling indexmap v2.0.0
     Compiling ryu v1.0.14
     Compiling itoa v1.0.8
     Compiling uuid v1.5.0
     Compiling base64 v0.21.5
     Compiling idna v0.4.0
     Compiling lexical-core v0.8.5
     Compiling url v2.4.1
     Compiling regex-automata v0.4.3
     Compiling pyo3-ffi v0.20.0
     Compiling pyo3 v0.20.0
     Compiling pydantic-core v2.14.5 (/tmp/tmph7u3k8qd/pydantic_core-2.14.5)
  error: failed to run custom build command for `pydantic-core v2.14.5 (/tmp/tmph7u3k8qd/pydantic_core-2.14.5)`

  Caused by:
    process didn't exit successfully: `/tmp/tmph7u3k8qd/pydantic_core-2.14.5/target/release/build/pydantic-core-5e5a4926f685a323/build-script-build` (exit status: 101)
    --- stdout
    cargo:rustc-cfg=Py_3_6
    cargo:rustc-cfg=Py_3_7
    cargo:rustc-cfg=Py_3_8
    cargo:rustc-cfg=Py_3_9
    cargo:rustc-cfg=Py_3_10
    cargo:rustc-cfg=Py_3_11
    cargo:rerun-if-changed=python/pydantic_core/core_schema.py
    cargo:rerun-if-changed=generate_self_schema.py

    --- stderr
    Traceback (most recent call last):
      File "/tmp/tmph7u3k8qd/pydantic_core-2.14.5/generate_self_schema.py", line 17, in <module>
        from typing_extensions import TypedDict, get_args, get_origin, is_typeddict
    ModuleNotFoundError: No module named 'typing_extensions'
    thread 'main' panicked at build.rs:29:9:
    generate_self_schema.py failed with exit status: 1
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
  warning: build failed, waiting for other jobs to finish...
  💥 maturin failed
    Caused by: Failed to build a native library through cargo
    Caused by: Cargo build finished with "exit status: 101": `env -u CARGO PYO3_ENVIRONMENT_SIGNATURE="cpython-3.11-64bit" PYO3_PYTHON="/usr/bin/python3.11" PYTHON_SYS_EXECUTABLE="/usr/bin/python3.11" "cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "/tmp/tmph7u3k8qd/pydantic_core-2.14.5/Cargo.toml" "--release" "--lib" "--crate-type" "cdylib"`
  Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/usr/bin/python3.11', '--compatibility', 'off'] returned non-zero exit status 1


  at /opt/poetry/venv/lib/python3.11/site-packages/poetry/installation/chef.py:164 in _prepare
      160│
      161│                 error = ChefBuildError("\n\n".join(message_parts))
      162│
      163│             if error is not None:
    → 164│                 raise error from None
      165│
      166│             return path
      167│
      168│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with pydantic-core (2.14.5) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "pydantic-core (==2.14.5)"'.

$ which python3
/venv/bin/python3

$ PYTHON=/venv/bin/python3 poetry install --no-root
Installing dependencies from lock file

Package operations: 2 installs, 0 updates, 0 removals

  - Installing pydantic-core (2.14.5)
  - Installing pydantic (2.5.2)

$ poetry update --dry-run
Updating dependencies
Resolving dependencies... (0.3s)

Package operations: 0 installs, 0 updates, 0 removals, 4 skipped

  - Installing annotated-types (0.7.0): Skipped for the following reason: Already installed
  - Installing pydantic (2.5.2): Skipped for the following reason: Already installed
  - Installing pydantic-core (2.14.5): Skipped for the following reason: Already installed
  - Installing typing-extensions (4.12.2): Skipped for the following reason: Already installed

[samuelcolvin]

It sounds like the issue is with pyo3_build_config::get().executable then.

Related to #674. @davidhewitt is on pat leave right now, but might have some thoughts when he's back. You could perhaps create an issue on https://github.com/pyo3/pyo3, although I know you've already going quite far down the rabbit whole on this one.

I think that the only fix open to pydantic-core is to pre-build self_schema.py, and thereby avoid the need to involve python at install time at all.

[dimbleby]

maybe these two tries at finding a python should simply be swapped so that the python in the path is found first

pydantic-core/build.rs

Lines 18 to 19 in a017bd9

	.or_else(|| pyo3_build_config::get().executable.clone())
	.unwrap_or_else(|| "python3".to_owned()),

edit: not quite so simple as "simply swapped" of course, would need to check that looking for a python in the path did find something

[samuelcolvin]

I think pyo3_build_config::get().executable will be right more often than just guessing at 'python3'.

Given that #764 was introduced a year ago, and AFAIK this is the first issue reported, I suspect what we have is working pretty well.

I don't want to introduce more and more logic to find the right python executable, let's either fix pyo3_build_config::get().executable or avoid the problem completely by pre-building self_schema.py.

[dimbleby]

I believe it is PyO3/maturin#2094 at the bottom of the rabbit-hole. maturin deliberately looks for the base python interpreter, rather than the one in the virtual environment.

That propagates via PYO3_PYTHON through pyo3 and then into the pydantic-core build script.

My feeling is that the maturin pull request was simply wrong - it directly contradicts this comment that it left unchanged, and seems rather to miss the point of setting up an isolated build environment with specified requirements.

Alternatively perhaps maturin is implicitly requiring that projects do not do as pydantic-core is doing ie actually use that python interpreter. But then they seem to be going to a lot of trouble to pass around an interpreter that should not be used...

[davidhewitt]

Yep this has now been fixed in maturin.