pyo3 0.14.2 build fails: src/types/datetime.rs has use of undeclared crate or module pyo3

[tiran]

🐛 Bug Reports

pyo3 0.14.2 and current master fail to build from source on a clean system without crate build cache.

     Running `/usr/bin/rustc --crate-name pyo3 --edition=2018 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --emit=dep-info,link -C opt-level=3 -C embed-bitcode=no --test --cfg 'feature="default"' --cfg 'feature="indoc"' --cfg 'feature="macros"' --cfg 'feature="paste"' --cfg 'feature="pyo3-macros"' --cfg 'feature="unindent"' -C metadata=0eb7c25a4e2939c1 -C extra-filename=-0eb7c25a4e2939c1 --out-dir /builddir/build/BUILD/pyo3-0.14.2/target/release/deps -L dependency=/builddir/build/BUILD/pyo3-0.14.2/target/release/deps --extern assert_approx_eq=/builddir/build/BUILD/pyo3-0.14.2/target/release/deps/libassert_approx_eq-9ddb982dab36ca2a.rlib --extern cfg_if=/builddir/build/BUILD/pyo3-0.14.2/target/release/deps/libcfg_if-22e8fbf441310aec.rlib --extern criterion=/builddir/build/BUILD/pyo3-0.14.2/target/release/deps/libcriterion-e177cafc80c35efa.rlib --extern indoc=/builddir/build/BUILD/pyo3-0.14.2/target/release/deps/libindoc-1d4fbc4a6ca20cbd.so --extern libc=/builddir/build/BUILD/pyo3-0.14.2/target/release/deps/liblibc-410fc196cf4c7f83.rlib --extern parking_lot=/builddir/build/BUILD/pyo3-0.14.2/target/release/deps/libparking_lot-2c8351e2d49210c4.rlib --extern paste=/builddir/build/BUILD/pyo3-0.14.2/target/release/deps/libpaste-ec6e91df397d2479.so --extern proptest=/builddir/build/BUILD/pyo3-0.14.2/target/release/deps/libproptest-032aaffb917786a7.rlib --extern pyo3_macros=/builddir/build/BUILD/pyo3-0.14.2/target/release/deps/libpyo3_macros-e7b00bd5f4d3f435.so --extern rustversion=/builddir/build/BUILD/pyo3-0.14.2/target/release/deps/librustversion-7baf5f71c39c5144.so --extern serde_json=/builddir/build/BUILD/pyo3-0.14.2/target/release/deps/libserde_json-4573a638a9116d32.rlib --extern trybuild=/builddir/build/BUILD/pyo3-0.14.2/target/release/deps/libtrybuild-0581ea42697f5899.rlib --extern unindent=/builddir/build/BUILD/pyo3-0.14.2/target/release/deps/libunindent-9540eae49115c7d9.rlib -Copt-level=3 -Cdebuginfo=2 -Clink-arg=-Wl,-z,relro,-z,now -Ccodegen-units=1 --cap-lints=warn -L native=/usr/lib64 --cfg Py_3_6 --cfg Py_3_7 --cfg Py_3_8 --cfg Py_3_9 --cfg Py_3_10 --cfg 'py_sys_config="WITH_THREAD"' --cfg min_const_generics --cfg addr_of -l python3.10`
error[E0433]: failed to resolve: use of undeclared crate or module `pyo3`
   --> src/types/datetime.rs:469:17
    |
469 |             use pyo3::types::{PyDateTime, PyTimeAccess};
    |                 ^^^^ use of undeclared crate or module `pyo3`

error[E0433]: failed to resolve: use of undeclared crate or module `pyo3`
   --> src/types/datetime.rs:468:15
    |
468 |         pyo3::Python::with_gil(|py| {
    |               ^^^^^^ not found in `pyo3`
    |
help: consider importing this struct
    |
465 |     use crate::Python;
    |

error[E0433]: failed to resolve: use of undeclared type `PyDateTime`
   --> src/types/datetime.rs:471:21
    |
471 |             let a = PyDateTime::new_with_fold(py, 2021, 1, 23, 20, 32, 40, 341516, None, false);
    |                     ^^^^^^^^^^ not found in this scope
    |
help: consider importing this struct
    |
465 |     use crate::types::PyDateTime;
    |

error[E0433]: failed to resolve: use of undeclared type `PyDateTime`
   --> src/types/datetime.rs:472:21
    |
472 |             let b = PyDateTime::new_with_fold(py, 2021, 1, 23, 20, 32, 40, 341516, None, true);
    |                     ^^^^^^^^^^ not found in this scope
    |
help: consider importing this struct
    |
465 |     use crate::types::PyDateTime;
    |

🌍 Environment

• Your operating system and version: Fedora 34
• Your python version: Python 3.9
• How did you install python (e.g. apt or pyenv)? Did you use a virtualenv?: Fedora system Python
• Your Rust version (rustc --version): rustc 1.54.0 (Fedora 1.54.0-1.fc34)
• Your PyO3 version: 0.14.2
• Have you tried using latest PyO3 main (replace version = "0.x.y" with git = "https://github.com/PyO3/pyo3")?: yes

💥 Reproducing

The issue is fully reproducible on Fedora build system. I was able to reproduce the problem once on my laptop. After I patched pyo3 and had a successful build, I'm no longer able to reproducible the problem locally. Even blowing away ~/.cargo doesn't help.

My fix is:

diff --git a/src/types/datetime.rs b/src/types/datetime.rs
index cad4b79d2..990d82ee2 100644
--- a/src/types/datetime.rs
+++ b/src/types/datetime.rs
@@ -465,8 +465,8 @@ mod tests {
     #[cfg(not(PyPy))]
     #[test]
     fn test_new_with_fold() {
-        pyo3::Python::with_gil(|py| {
-            use pyo3::types::{PyDateTime, PyTimeAccess};
+        crate::Python::with_gil(|py| {
+            use crate::types::{PyDateTime, PyTimeAccess};
 
             let a = PyDateTime::new_with_fold(py, 2021, 1, 23, 20, 32, 40, 341516, None, false);
             let b = PyDateTime::new_with_fold(py, 2021, 1, 23, 20, 32, 40, 341516, None, true);

[tiran]

The fix resolves the build error on our build infra

Unpatched 0.14.2: https://copr.fedorainfracloud.org/coprs/cheimes/rust/build/2526992/
Patched 0.14.2: https://copr.fedorainfracloud.org/coprs/cheimes/rust/build/2527086/

After patching the test suite is failing with error message The Python interpreter is not initalized and the `auto-initialize` feature is not enabled..

[davidhewitt]

This is interesting; in general it sounds like this line in the PyO3 Cargo.toml is being ignored by your build infra?

For sure the patch you submit is a correct bugfix, I'll merge that now 👍

[tiran]

Thanks!

Yes, it appears that our build system ignores that line. I had to add --features auto-initialize,macros to cargo test run, too. You can find the full patch set at https://src.fedoraproject.org/rpms/rust-pyo3/pull-request/1#request_diff

[tiran]

The RPM build system is running cargo check with additional options:

CARGO_HOME=.cargo RUSTC_BOOTSTRAP=1 /usr/bin/cargo test -j2 -Z avoid-dev-deps --release

I guess that the combination of release flag, avoid-dev-deps, and custom cargo home caused cargo test to ignore the dev-dependency line.

[birkenfeld]

Not sure how avoid-dev-deps is supposed to work with cargo test - which is a dev-related subcommand.

[birkenfeld]

In any case, please bring this up on rust-lang/cargo#5133 - they might want to disallow avoid-dev-deps for cargo test, cargo bench and so forth.

[tiran]

@decathorpe and @cuviper, please take a look.

[decathorpe]

Interesting. I've only ever seen projects use [[test]] with required-features = ["foo", "bar"] if there are tests that require certain features to be enabled in the current crate, like here, in nom:

https://github.com/Geal/nom/blob/6.2.1/Cargo.toml#L107

Path-based dependencies are not supported in our build system at all (because there's no way to know what the complete context of a crate's sources is from a downloaded crate from crates.io), but this not at all the culprit here: cargo itself removes this path-based "self-dev-dependency" when running "cargo package" or "cargo publish". You can check this by running "cargo package" locally, or by downloading the crate from crates.io:

https://crates.io/api/v1/crates/pyo3/0.14.2/download#/pyo3-0.14.2.crate

So the missing dev-dependency is not a problem on our end, but an effect of how the pyo3 crate is forcing "self-dependency with certain features enabled" in an unorthodox way.