Update to Metadata 2.3 to create reliable source dist metadata

Source distributions contain a `PKG-INFO` file at root level with metadata in the same format as wheel metadata. Unlike wheel metadata, this file was previously unreliably, the metadata did not need to (and often did not) match the metadata of the built wheel.

[PEP 643](https://peps.python.org/pep-0643/) introduces Metadata 2.2, where fields can either be static (default) or marked as dynamic. When a source distribution uses `Metadata-Version: 2.3` in its PKG-INFO file, all static fields in the source distribution metadata must match the metadata of a built wheel ([spec](https://packaging.python.org/en/latest/specifications/source-distribution-format/#source-distribution-format)). This means that e.g. when requirements and extras are not marked as dynamic, a version resolver can read the metadata directly from the source dist without a PEP 517 invocation. This is both a huge speedup and makes python packaging more reliable.

This PR bumps the metadata version maturin uses to 2.3 ([PEP 685](https://peps.python.org/pep-0685/) – Comparison of extra names for optional distribution dependencies). It does not add support for dynamic, instead maturin writes static metadata for all of its source dists. If possible, i'd like to keep maturin to emit purely static metadata, it has advantages both for the python packaging ecosystem and by enforcing a better project structure. If you want a single source of truth for the package version, i recommend storing the version in pyproject.toml `project.version` and using `importlib.metadata.version` in the python code.

I bump the version to 1.5.0 since it's a significant behaviour change, but doesn't justify a 2.0 release.

Cargo.toml

@@ -1,7 +1,7 @@
 [package]
 authors = ["konstin <[email protected]>", "messense <[email protected]>"]
 name = "maturin"
-version = "1.4.0"
+version = "1.5.0"
 description = "Build and publish crates with pyo3, rust-cpython and cffi bindings as well as rust binaries as python packages"
 exclude = [
     "test-crates/**/*",

Changelog.md

@@ -7,6 +7,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+* Bump metadata version from 2.1 to 2.3 in [#1960](https://github.com/PyO3/maturin/pull/1960). Source distributions created by maturin now have reliable metadata, meaning tool such as pip, uv and poetry could skip building them for version resolution.
+
 ## [1.4.0] - 2023-12-02
 
 * Bump MSRV to 1.67.0 in [#1847](https://github.com/PyO3/maturin/pull/1847)

src/build_context.rs

@@ -11,7 +11,7 @@ use crate::python_interpreter::InterpreterKind;
 use crate::source_distribution::source_distribution;
 use crate::target::{Arch, Os};
 use crate::{
-    compile, pyproject_toml::Format, BuildArtifact, Metadata21, ModuleWriter, PyProjectToml,
+    compile, pyproject_toml::Format, BuildArtifact, Metadata23, ModuleWriter, PyProjectToml,
     PythonInterpreter, Target,
 };
 use anyhow::{anyhow, bail, Context, Result};
@@ -91,17 +91,17 @@ impl Display for BridgeModel {
 /// Insert wasm launcher scripts as entrypoints and the wasmtime dependency
 fn bin_wasi_helper(
     artifacts_and_files: &[(&BuildArtifact, String)],
-    mut metadata21: Metadata21,
-) -> Result<Metadata21> {
+    mut metadata23: Metadata23,
+) -> Result<Metadata23> {
     eprintln!("⚠️  Warning: wasi support is experimental");
     // escaped can contain [\w\d.], but i don't know how we'd handle dots correctly here
-    if metadata21.get_distribution_escaped().contains('.') {
+    if metadata23.get_distribution_escaped().contains('.') {
         bail!(
             "Can't build wasm wheel if there is a dot in the name ('{}')",
-            metadata21.get_distribution_escaped()
+            metadata23.get_distribution_escaped()
         )
     }
-    if !metadata21.entry_points.is_empty() {
+    if !metadata23.entry_points.is_empty() {
         bail!("You can't define entrypoints yourself for a binary project");
     }
 
@@ -114,29 +114,29 @@ fn bin_wasi_helper(
             base_name.to_string(),
             format!(
                 "{}.{}:main",
-                metadata21.get_distribution_escaped(),
+                metadata23.get_distribution_escaped(),
                 base_name.replace('-', "_")
             ),
         );
     }
 
-    metadata21
+    metadata23
         .entry_points
         .insert("console_scripts".to_string(), console_scripts);
 
     // Add our wasmtime default version if the user didn't provide one
-    if !metadata21
+    if !metadata23
         .requires_dist
         .iter()
         .any(|requirement| requirement.name.as_ref() == "wasmtime")
     {
         // Having the wasmtime version hardcoded is not ideal, it's easy enough to overwrite
-        metadata21
+        metadata23
             .requires_dist
             .push(Requirement::from_str("wasmtime>=11.0.0,<12.0.0").unwrap());
     }
 
-    Ok(metadata21)
+    Ok(metadata23)
 }
 
 /// Contains all the metadata required to build the crate
@@ -152,13 +152,13 @@ pub struct BuildContext {
     pub pyproject_toml_path: PathBuf,
     /// Parsed pyproject.toml if any
     pub pyproject_toml: Option<PyProjectToml>,
-    /// Python Package Metadata 2.1
-    pub metadata21: Metadata21,
+    /// Python Package Metadata 2.3
+    pub metadata23: Metadata23,
     /// The name of the crate
     pub crate_name: String,
     /// The name of the module can be distinct from the package name, mostly
     /// because package names normally contain minuses while module names
-    /// have underscores. The package name is part of metadata21
+    /// have underscores. The package name is part of metadata23
     pub module_name: String,
     /// The path to the Cargo.toml. Required for the cargo invocations
     pub manifest_path: PathBuf,
@@ -481,7 +481,7 @@ impl BuildContext {
 
     fn add_pth(&self, writer: &mut WheelWriter) -> Result<()> {
         if self.editable {
-            writer.add_pth(&self.project_layout, &self.metadata21)?;
+            writer.add_pth(&self.project_layout, &self.metadata23)?;
         }
         Ok(())
     }
@@ -508,7 +508,7 @@ impl BuildContext {
                 "{}{}{}-*.tar.gz",
                 self.out.display(),
                 std::path::MAIN_SEPARATOR,
-                &self.metadata21.get_distribution_escaped(),
+                &self.metadata23.get_distribution_escaped(),
             );
             excludes.add(&glob_pattern)?;
         }
@@ -659,7 +659,7 @@ impl BuildContext {
         let mut writer = WheelWriter::new(
             &tag,
             &self.out,
-            &self.metadata21,
+            &self.metadata23,
             &[tag.clone()],
             self.excludes(Format::Wheel)?,
         )?;
@@ -737,7 +737,7 @@ impl BuildContext {
         let mut writer = WheelWriter::new(
             &tag,
             &self.out,
-            &self.metadata21,
+            &self.metadata23,
             &[tag.clone()],
             self.excludes(Format::Wheel)?,
         )?;
@@ -859,7 +859,7 @@ impl BuildContext {
         let mut writer = WheelWriter::new(
             &tag,
             &self.out,
-            &self.metadata21,
+            &self.metadata23,
             &tags,
             self.excludes(Format::Wheel)?,
         )?;
@@ -897,7 +897,7 @@ impl BuildContext {
 
         // Warn if cffi isn't specified in the requirements
         if !self
-            .metadata21
+            .metadata23
             .requires_dist
             .iter()
             .any(|requirement| requirement.name.as_ref() == "cffi")
@@ -925,7 +925,7 @@ impl BuildContext {
         let mut writer = WheelWriter::new(
             &tag,
             &self.out,
-            &self.metadata21,
+            &self.metadata23,
             &tags,
             self.excludes(Format::Wheel)?,
         )?;
@@ -983,7 +983,7 @@ impl BuildContext {
             _ => unreachable!(),
         };
 
-        if !self.metadata21.scripts.is_empty() {
+        if !self.metadata23.scripts.is_empty() {
             bail!("Defining scripts and working with a binary doesn't mix well");
         }
 
@@ -1010,16 +1010,16 @@ impl BuildContext {
             artifacts_and_files.push((artifact, bin_name))
         }
 
-        let metadata21 = if self.target.is_wasi() {
-            bin_wasi_helper(&artifacts_and_files, self.metadata21.clone())?
+        let metadata23 = if self.target.is_wasi() {
+            bin_wasi_helper(&artifacts_and_files, self.metadata23.clone())?
         } else {
-            self.metadata21.clone()
+            self.metadata23.clone()
         };
 
         let mut writer = WheelWriter::new(
             &tag,
             &self.out,
-            &metadata21,
+            &metadata23,
             &tags,
             self.excludes(Format::Wheel)?,
         )?;
@@ -1041,9 +1041,9 @@ impl BuildContext {
         let mut artifacts_ref = Vec::with_capacity(artifacts.len());
         for (artifact, bin_name) in &artifacts_and_files {
             artifacts_ref.push(*artifact);
-            write_bin(&mut writer, &artifact.path, &self.metadata21, bin_name)?;
+            write_bin(&mut writer, &artifact.path, &self.metadata23, bin_name)?;
             if self.target.is_wasi() {
-                write_wasm_launcher(&mut writer, &self.metadata21, bin_name)?;
+                write_wasm_launcher(&mut writer, &self.metadata23, bin_name)?;
             }
         }
         self.add_external_libs(&mut writer, &artifacts_ref, ext_libs)?;

src/build_options.rs

@@ -494,7 +494,7 @@ impl BuildOptions {
             pyproject_toml_path,
             pyproject_toml,
             module_name,
-            metadata21,
+            metadata23,
             mut cargo_options,
             cargo_metadata,
             mut pyproject_toml_maturin_options,
@@ -564,7 +564,7 @@ impl BuildOptions {
                 &bridge,
                 &[],
                 &target,
-                metadata21.requires_python.as_ref(),
+                metadata23.requires_python.as_ref(),
                 generate_import_lib,
             )?
         } else {
@@ -678,7 +678,7 @@ impl BuildOptions {
             project_layout,
             pyproject_toml_path,
             pyproject_toml,
-            metadata21,
+            metadata23,
             crate_name,
             module_name,
             manifest_path: cargo_toml_path,

src/develop.rs

@@ -82,9 +82,9 @@ fn install_dependencies(
     interpreter: &PythonInterpreter,
     pip_path: Option<&Path>,
 ) -> Result<()> {
-    if !build_context.metadata21.requires_dist.is_empty() {
+    if !build_context.metadata23.requires_dist.is_empty() {
         let mut args = vec!["install".to_string()];
-        args.extend(build_context.metadata21.requires_dist.iter().map(|x| {
+        args.extend(build_context.metadata23.requires_dist.iter().map(|x| {
             let mut pkg = x.clone();
             // Remove extra marker to make it installable with pip
             // Keep in sync with `Metadata21::merge_pyproject_toml()`!
@@ -175,7 +175,7 @@ fn fix_direct_url(
     let mut pip_cmd = make_pip_command(python, pip_path);
     let output = pip_cmd
         .args(["show", "--files"])
-        .arg(&build_context.metadata21.name)
+        .arg(&build_context.metadata23.name)
         .output()
         .context(format!(
             "pip show failed (ran {:?} with {:?})",
@@ -292,7 +292,7 @@ pub fn develop(develop_options: DevelopOptions, venv_dir: &Path) -> Result<()> {
             )?;
             eprintln!(
                 "🛠 Installed {}-{}",
-                build_context.metadata21.name, build_context.metadata21.version
+                build_context.metadata23.name, build_context.metadata23.version
             );
         }
     }

src/lib.rs

@@ -28,7 +28,7 @@ pub use crate::build_options::{BuildOptions, CargoOptions};
 pub use crate::cargo_toml::CargoToml;
 pub use crate::compile::{compile, BuildArtifact};
 pub use crate::develop::{develop, DevelopOptions};
-pub use crate::metadata::{Metadata21, WheelMetadata};
+pub use crate::metadata::{Metadata23, WheelMetadata};
 pub use crate::module_writer::{
     write_dist_info, ModuleWriter, PathWriter, SDistWriter, WheelWriter,
 };

src/main.rs

@@ -262,8 +262,8 @@ fn pep517(subcommand: Pep517Command) -> Result<()> {
             };
 
             let mut writer = PathWriter::from_path(metadata_directory);
-            write_dist_info(&mut writer, &context.metadata21, &tags)?;
-            println!("{}", context.metadata21.get_dist_info_dir().display());
+            write_dist_info(&mut writer, &context.metadata23, &tags)?;
+            println!("{}", context.metadata23.get_dist_info_dir().display());
         }
         Pep517Command::BuildWheel {
             build_options,