tree-wide: convert rust with git deps to importCargoLock #221716

yu-re-ka · 2023-03-17T19:23:38Z

Description of changes

Less invasive alternative to #217084

Things done

winterqt · 2023-03-17T20:07:11Z

0319eccab39512bf55afa1536515c21134a1dbd0 should explain why, at least in the commit message, as it does technically work. I assume you just want to discourage its use?

yu-re-ka · 2023-03-17T21:16:32Z

Well, with cargo 1.68 and above we can not rely on it being reproducible with git dependencies, as there is extended logic involved now which might change in future versions, so we would have to regenerate the FOD hashes on every update.

The assumption is that cargo vendor is still reproducible when no git dependencies are involved.

winterqt · 2023-03-17T21:27:57Z

Please re-read my comment -- that's exactly what I'm trying to say; it technically works, but we don't want anyone to actually use it. This should be made more clear, at least in the commit message, IMO.

zowoq · 2023-03-18T01:46:21Z

There are several lockfiles committed without a corresponding nix package change?

e.g:

squeekboard/Cargo.lock
gnome-podcats/Cargo.lock
emblem/Cargo.lock
rnote/Cargo.lock
citations/Cargo.lock
newsflash/Cargo.lock
warp/Cargo.lock

oxalica · 2023-03-18T10:39:54Z

Nixpkgs size increase of this PR is ~5% before compression, ~6% after compression.

$ fd '^Cargo.lock$' -X cat | wc -c
8.8Mi

$ git archive --format=tar HEAD~3 | wc -c
162600960
$ git archive --format=tar HEAD | wc -c
170997760

$ git archive --format=tar.gz HEAD~3 | wc -c
34881234
$ git archive --format=tar.gz HEAD | wc -c
37023424

yu-re-ka · 2023-03-18T15:33:19Z

There are several lockfiles committed without a corresponding nix package change?

good find, these should be fixed now

pkgs/build-support/rust/import-cargo-lock.nix

yu-re-ka · 2023-03-18T21:23:28Z

I have run nix-review to ensure that everything still builds.

These packages are already broken on master: fedigroups, habitat

figsoda · 2023-03-19T00:56:09Z

Should I move the jumpy changes to #221921? Didn't see it when I was updating jumpy, sorry

figsoda · 2023-03-19T01:07:50Z

oh cool that will work too

pkgs/applications/misc/citations/default.nix

pkgs/tools/system/zenith/default.nix

zowoq · 2023-03-26T00:40:45Z

pkgs/build-support/rust/fetch-cargo-tarball/default.nix

@@ -76,6 +76,17 @@ in stdenv.mkDerivation ({
    # Override the `http.cainfo` option usually specified in `.cargo/config`.
    export CARGO_HTTP_CAINFO=${cacert}/etc/ssl/certs/ca-bundle.crt

+    if grep --pcre2 '^source = "(?!registry)' Cargo.lock; then


grep: unrecognized option '--pcre2'

perhaps we can just do ^source = "git\+ since we don't really know how to deal with non-git deps anyway?

That was my alias grep=rg in my environment 😅

The reason is that we can not expect the extended logic run on git dependencies starting from Cargo 1.68 to be reproducible in future versions, and thus the output hash would not be sufficiently stable. rust-lang/cargo#11414

superherointj · 2023-03-27T22:32:51Z

Noticed a spike averaging ~6.5 seconds in eval times at nixpkgs. On bisecting found:

Preceding commit to this PR:

2f0ff93 Merge pull request #222585 from fabaff/yara-bump
- real 1m8,129s
  user 1m5,506s
  sys 0m2,620s

At this PR:

f8cbc3c tree-wide: convert rust with git deps to importCargoLock
- real 1m14,646s
  user 1m12,097s
  sys 0m2,546s

Command used for testing:

time nix-env --query --available --out-path --file ./. > /dev/null

Mic92 · 2023-03-29T06:22:47Z

Also around 10GB of RAM on x86_64-linux. Haven't compared what we had before.

alyssais · 2023-03-29T09:01:23Z

That's really weird, considering that in #217084 we found that converting all Rust packages rather than just ones with git deps had almost no impact on CPU time, and a similarly low impact on memory. cc @winterqt

alyssais · 2023-03-29T09:49:28Z

OfBorg also found a much bigger evaluation performance difference than we were expecting from Winter's research: https://github.com/NixOS/nixpkgs/runs/12279178729

kirillrdy · 2023-12-28T00:36:55Z

@yu-re-ka apologies if this has been answered somewhere, but is there a reason for not using Cargo.lock from ${src}/Cargo.lock ( where possible of course )

kirillrdy · 2023-12-28T01:22:58Z

@yu-re-ka apologies if this has been answered somewhere, but is there a reason for not using Cargo.lock from ${src}/Cargo.lock ( where possible of course )

I think I've found answer,

during evaluation because the option 'allow-import-from-derivation' is disabled

yu-re-ka requested review from piegamesde, mmahut, RaghavSood, figsoda, jonringer, teto, zowoq, winterqt and FRidh as code owners March 17, 2023 19:23

github-actions bot added 6.topic: python 6.topic: rust 6.topic: vim labels Mar 17, 2023

ofborg bot added 10.rebuild-darwin: 11-100 10.rebuild-linux: 101-500 labels Mar 17, 2023

yu-re-ka changed the base branch from staging to master March 18, 2023 15:37

ofborg bot added 10.rebuild-darwin: 101-500 2.status: merge conflict This PR has merge conflicts with the target branch and removed 10.rebuild-darwin: 11-100 labels Mar 18, 2023

figsoda reviewed Mar 18, 2023

View reviewed changes

pkgs/build-support/rust/import-cargo-lock.nix Outdated Show resolved Hide resolved

ofborg bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Mar 18, 2023

figsoda mentioned this pull request Mar 19, 2023

RFC: Migrate all Rust packages to importCargoLock #217084

Closed

tomberek approved these changes Mar 25, 2023

View reviewed changes

figsoda reviewed Mar 25, 2023

View reviewed changes

pkgs/applications/misc/citations/default.nix Outdated Show resolved Hide resolved

figsoda reviewed Mar 26, 2023

View reviewed changes

pkgs/tools/system/zenith/default.nix Outdated Show resolved Hide resolved

zowoq reviewed Mar 26, 2023

View reviewed changes

yu-re-ka added 3 commits March 26, 2023 01:52

rustPlatform.fetchCargoTarball: fail on git dependencies

4329de1

The reason is that we can not expect the extended logic run on git dependencies starting from Cargo 1.68 to be reproducible in future versions, and thus the output hash would not be sufficiently stable. rust-lang/cargo#11414

docs/rust: add note about git dependencies

f3a1640

tree-wide: convert rust with git deps to importCargoLock

f8cbc3c

zowoq merged commit d94076a into NixOS:master Mar 26, 2023

yu-re-ka deleted the git-deps-import-cargo-lock branch March 26, 2023 06:33

tomhoule mentioned this pull request Mar 29, 2023

prisma-engines: 4.11.0 -> 4.12.0 #223700

Merged

12 tasks

lilyinstarlight mentioned this pull request Mar 30, 2023

matrix-sdk-crypto-nodejs: 0.1.0-beta.2 -> 0.1.0-beta.3 #220317

Merged

12 tasks

justinas mentioned this pull request Apr 2, 2023

teleport: 12.1.0 -> 12.1.5 #224379

Merged

12 tasks

euank mentioned this pull request Apr 5, 2023

anki: 2.1.15 -> 2.1.60 #221229

Merged

11 tasks

huantianad mentioned this pull request Apr 27, 2023

activitywatch: init at 0.12.2 #202935

Merged

17 tasks

kira-bruneau mentioned this pull request May 4, 2023

newsflash: 2.2.4 -> 2.3.0 #229970

Merged

12 tasks

Stunkymonkey mentioned this pull request May 4, 2023

Cargo build without cargoSha256, use Cargo.lock instead #63653

Open

raphaelr mentioned this pull request Jun 6, 2023

treewide: convert remaining rust packages with git dependencies to importCargoLock #236308

Closed

12 tasks

winterqt mentioned this pull request Jul 31, 2023

rustc: 1.67.1 -> 1.68.1 #220373

Merged

12 tasks

pbsds mentioned this pull request Aug 7, 2023

pagefind: init at 1.0.3 #247773

Merged

12 tasks

kira-bruneau mentioned this pull request Aug 22, 2023

texlab: 5.9.0 -> 5.9.2 #250624

Merged

12 tasks

ahirner mentioned this pull request Dec 27, 2023

edgedb: 3.4.0 -> 4.0.2 #277193

Merged

13 tasks

amesgen mentioned this pull request Apr 26, 2024

matrix-synapse-tools.rust-synapse-compress-state: 0.1.3 -> 0.1.4 #303704

Merged

13 tasks

superherointj mentioned this pull request Jun 17, 2024

Nixpkgs eval time is increasing too fast #320528

Open

superherointj mentioned this pull request Jul 14, 2024

Cargo.lock considered harmful #327063

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tree-wide: convert rust with git deps to importCargoLock #221716

tree-wide: convert rust with git deps to importCargoLock #221716

yu-re-ka commented Mar 17, 2023 •

edited

Loading

winterqt commented Mar 17, 2023

yu-re-ka commented Mar 17, 2023

winterqt commented Mar 17, 2023

zowoq commented Mar 18, 2023

oxalica commented Mar 18, 2023 •

edited

Loading

yu-re-ka commented Mar 18, 2023

yu-re-ka commented Mar 18, 2023 •

edited

Loading

figsoda commented Mar 19, 2023 •

edited

Loading

figsoda commented Mar 19, 2023

zowoq Mar 26, 2023

figsoda Mar 26, 2023 •

edited

Loading

yu-re-ka Mar 26, 2023

superherointj commented Mar 27, 2023 •

edited

Loading

Mic92 commented Mar 29, 2023

alyssais commented Mar 29, 2023 •

edited

Loading

alyssais commented Mar 29, 2023

kirillrdy commented Dec 28, 2023

kirillrdy commented Dec 28, 2023

tree-wide: convert rust with git deps to importCargoLock #221716

tree-wide: convert rust with git deps to importCargoLock #221716

Conversation

yu-re-ka commented Mar 17, 2023 • edited Loading

Description of changes

Things done

winterqt commented Mar 17, 2023

yu-re-ka commented Mar 17, 2023

winterqt commented Mar 17, 2023

zowoq commented Mar 18, 2023

oxalica commented Mar 18, 2023 • edited Loading

yu-re-ka commented Mar 18, 2023

yu-re-ka commented Mar 18, 2023 • edited Loading

figsoda commented Mar 19, 2023 • edited Loading

figsoda commented Mar 19, 2023

zowoq Mar 26, 2023

Choose a reason for hiding this comment

figsoda Mar 26, 2023 • edited Loading

Choose a reason for hiding this comment

yu-re-ka Mar 26, 2023

Choose a reason for hiding this comment

superherointj commented Mar 27, 2023 • edited Loading

Mic92 commented Mar 29, 2023

alyssais commented Mar 29, 2023 • edited Loading

alyssais commented Mar 29, 2023

kirillrdy commented Dec 28, 2023

kirillrdy commented Dec 28, 2023

yu-re-ka commented Mar 17, 2023 •

edited

Loading

oxalica commented Mar 18, 2023 •

edited

Loading

yu-re-ka commented Mar 18, 2023 •

edited

Loading

figsoda commented Mar 19, 2023 •

edited

Loading

figsoda Mar 26, 2023 •

edited

Loading

superherointj commented Mar 27, 2023 •

edited

Loading

alyssais commented Mar 29, 2023 •

edited

Loading