From 0c229d7813af53faa82405f08eacc089b43fc826 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Wed, 15 Jan 2025 11:28:05 -0500 Subject: [PATCH] Minor: Document the rationale for the lack of Cargo.lock (#14071) * Minor: Document the rationale for the lack of Cargo.lock * Update README.md --------- Co-authored-by: Oleks V --- README.md | 24 ++++++++++++++++++++++++ datafusion-cli/README.md | 4 +++- 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index e0fc6854ecff..2a20faa9e2fa 100644 --- a/README.md +++ b/README.md @@ -146,3 +146,27 @@ stable API, we also improve the API over time. As a result, we typically deprecate methods before removing them, according to the [deprecation guidelines]. [deprecation guidelines]: https://datafusion.apache.org/library-user-guide/api-health.html + +## Dependencies and a `Cargo.lock` + +`datafusion` is intended for use as a library and thus purposely does not have a +`Cargo.lock` file checked in. You can read more about the distinction in the +[Cargo book]. + +CI tests always run against the latest compatible versions of all dependencies +(the equivalent of doing `cargo update`), as suggested in the [Cargo CI guide] +and we rely on Dependabot for other upgrades. This strategy has two problems +that occasionally arise: + +1. CI failures when downstream libraries upgrade in some non compatible way +2. Local development builds that fail when DataFusion inadvertently relies on + a feature in a newer version of a dependency than declared in `Cargo.toml` + (e.g. a new method is added to a trait that we use). + +However, we think the current strategy is the best tradeoff between maintenance +overhead and user experience and ensures DataFusion always works with the latest +compatible versions of all dependencies. If you encounter either of these +problems, please open an issue or PR. + +[cargo book]: https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html +[cargo ci guide]: https://doc.rust-lang.org/cargo/guide/continuous-integration.html#verifying-latest-dependencies diff --git a/datafusion-cli/README.md b/datafusion-cli/README.md index 73a2eb01b76f..ce09c3b345b9 100644 --- a/datafusion-cli/README.md +++ b/datafusion-cli/README.md @@ -41,6 +41,8 @@ The reason `datafusion-cli` is not part of the main workspace in checked in `Cargo.lock` file to ensure reproducible builds. However, the `datafusion` and sub crates are intended for use as libraries and -thus do not have a `Cargo.lock` file checked in. +thus do not have a `Cargo.lock` file checked in, as described in the [main +README] file. [`datafusion cargo.toml`]: https://github.com/apache/datafusion/blob/main/Cargo.toml +[main readme]: ../README.md