IFD does not work with git deps

[Ten0]

With the following nix derivation:

          ((pkgs.callPackage "${crate2nix}/tools.nix" { }).generatedCargoNix {
            name = "proj";
            # Filter src to avoid it being too large - keep Cargo.toml/Cargo.lock and lib.rs/main.rs
            src = lib.fileset.toSource {
              root = ../..;
              fileset = lib.fileset.fromSource (lib.cleanSourceWith {
                src = ../..;
                filter = path: type: (
                  let baseName = baseNameOf (toString path);
                  in
                  (
                    (type == "directory" && baseName != "target" && baseName != "node_modules")
                    || (baseName == "Cargo.toml" || baseName == "Cargo.lock" || baseName == "lib.rs" || baseName == "main.rs")
                  ) && (lib.cleanSourceFilter path type) # + other basic filters
                );
              });
            };
            cargoToml = ".cargo/workspace/Cargo.toml";
          })

Building leads to the following error (truncated with hopefully only relevant information):

@nix { "action": "setPhase", "phase": "unpackPhase" }
Running phase: unpackPhase
unpacking source archive /nix/store/dgd0qsj7p8s8aid2mkm09ynlxl4f9x6d-source
source root is source
@nix { "action": "setPhase", "phase": "buildPhase" }
Running phase: buildPhase
++ crate2nix generate -f ./.cargo/workspace/Cargo.toml -o Cargo-generated.nix -h /nix/store/80m9cr7lmpsrxh79ql70iz763r6xs90j-proj-crate2nix/crate-hashes.json
Error: while retrieving metadata about ./.cargo/workspace/Cargo.toml: `cargo metadata` exited with an error: error: no matching package named `lightgbm-sys` found
location searched: https://github.com/Ten0/lightgbm-rs.git?branch=for_proj_main#ebc52622
required by package `lightgbm v0.2.3 (https://github.com/Ten0/lightgbm-rs.git?branch=for_proj_main#ebc52622)`
    ... which satisfies git dependency `lightgbm` (locked to 0.2.3) of package `p1 v1.0.0 (/build/source/p2/lib/p1)`
    ... which satisfies path dependency `p1` (locked to 1.0.0) of package `p2 v1.0.0 (/build/source/p2/Service)`

crate2nix failed.
== cargo/config (BEGIN)
    [source.crates-io]
    replace-with = "vendored-sources"
    
    [source."https://github.com/Ten0/lightgbm-rs.git?branch=for_proj_main#ebc5262299d75cc770e21416f9e3ef8204427fba"]
    git = "https://github.com/Ten0/lightgbm-rs.git"
    
    
    branch = "for_proj_main"
    replace-with = "vendored-sources"
    
    [source.vendored-sources]
    directory = "/nix/store/zg0br63gy3b9r67kkfzsjqzxwz79vqwk-deps"

== cargo/config (END)

== crate-hashes.json (BEGIN)
    {
      "lightgbm 0.2.3 (git+https://github.com/Ten0/lightgbm-rs.git?branch=for_proj_main#ebc5262299d75cc770e21416f9e3ef8204427fba)": "0nr42asw4sg7q7xrkg565ivl1s9b6g75kflzy9hvl3srqrnxf10z",
      "lightgbm-sys 0.3.0 (git+https://github.com/Ten0/lightgbm-rs.git?branch=for_proj_main#ebc5262299d75cc770e21416f9e3ef8204427fba)": "0nr42asw4sg7q7xrkg565ivl1s9b6g75kflzy9hvl3srqrnxf10z"
    }
== crate-hashes.json (END)

== ls -la (BEGIN)
total 24
drwxr-xr-x  24 nixbld nixbld  4096 Jan  1  1970 .
drwx------   3 nixbld nixbld    36 May  1 22:54 ..
drwxr-xr-x   3 nixbld nixbld    23 Jan  1  1970 .cargo
drwxr-xr-x  12 nixbld nixbld   173 Jan  1  1970 p2
== ls -la (END)

Contents of /nix/store/zg0br63gy3b9r67kkfzsjqzxwz79vqwk-deps:

> cd /nix/store/zg0br63gy3b9r67kkfzsjqzxwz79vqwk-deps
> find . -maxdepth 1 -name "*lightgbm*"
./hway6s5x02wwvd7jqcqikvb8gw4zmsls-lightgbm-sys-0.3.0
./j7yyjdxmv1vfk77b7a7s30rqliq8rm0k-lightgbm-0.2.3

If I do generate Cargo.nix manually via nix run github:nix-community/crate2nix -- generate -f .cargo/workspace/Cargo.toml and import that instead, it works.

Thanks for your help.

[pacak]

I think rust version that gets passed to "${crate2nix}/tools.nix" is different from rust version you get when you nix run, try checking that.

[Ten0]

That seems to not be the source of the issue.

nix run

When I nix run it uses cargo 1.76.0 as pinned in your nixpkgs, since nix derivation show github:nix-community/crate2nix shows /nix/store/8s7fd4c5r06d4x94vqbmgwsv05v2jmlb-cargo-1.76.0.drv, UNLESS there is also another cargo version in path, in which case it uses that one.
Either way (1.76.0 or my 1.77.0) I don't get any error when running crate2nix generate, and build then runs - although the generated files look different in that they don't use the same ID format.

IFD

I was doing (pkgs.callPackage "${crate2nix}/tools.nix" { }).generatedCargoNix, which had me override the pkgs, but I happened to also have 1.76 there (/nix/store/7ld12b9gmdziwrvjyb1lhf5y49b7l0nw-cargo-1.76.0.drv).
I also tried using crate2nix.tools.${system}.generatedCargoNix, which prevents me from overriding and instead uses the version that you're exporting, but that does result in the same error, despite it picking the exact same derivation as non-ifd: /nix/store/8s7fd4c5r06d4x94vqbmgwsv05v2jmlb-cargo-1.76.0.drv.

So it would look like that's not the source of the issue.

I also made sure to be generating the Cargo.lock on 1.76 if that matters.

Side note

• When building afterwards - if I get to that point - and in the associated dev shell, I'm then using latest stable from oxalica:

          {
            buildRustCrateForPkgs = pkgs: pkgs.buildRustCrate.override (
              let rustToolchain = pkgs.rust-bin.stable.latest; in {
                # Use the latest stable rust version from oxalica overlay instead of the one in nixpkgs to build workspace packages
                # https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/rust.section.md
                # https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/rust/build-rust-crate/default.nix#L12-L13
                rustc = rustToolchain.default;
                cargo = rustToolchain.cargo;
                # We need to override some dependencies/env variables... for some crates - that is specified in crate_overrides.nix
                defaultCrateOverrides = pkgs.callPackage ./crate_overrides.nix { inherit flake-inputs packages cargoNix; };
              }
            );
          };

That's currently pinned at 1.77.0 in the flake.
If it matters that this is the same version that we generate the Cargo.nix for (that Cargo.lock is generated with...), then I wonder how I could override tools.nix to have it use that version from my pkgs.rust-bin - there doesn't seem to be a corresponding argument.