python applications propagate python and messes up PATH

[teto]

Describe the bug

I have the equivalent of

stdenv.mkShell {
  buildInputs = with pkgs; [
		myPythonEnvWithSomeLibraries  # my python.withPackages with 2/3 libraries
        yq  # a python application
	];
}

I start my shell with nix develop / nix-shell. Then when I run python I get a basic python != myPythonEnvWithSomeLibraries. I investigated a bit and turns out that yq is a python application and it has python in its propagatedBuildInputs.
When I enter my nix-shell yq's propagated python ends up at the beginning of my PATH, thus shadowing myPythonEnvWithSomeLibraries's interpreter.

I have 2 questions then:

• I think Python applications shouldn't propagate python ? python packaging is so complex that I wonder if I miss something. Sorry if I missed a closed ticket (found python: can not use imperatively installed packages as libraries #10597 that could be related)
• to work around it, how could I remove python from the propagatedBuildInputs of yq ?

Expected behavior

Python applications should not shadow my shell python interpreter.

Notify maintainers

cc @FRidh

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
output here
 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.34, NixOS, 22.05 (Quokka)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.7.0`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

[FRidh]

nix-shell is a utility for creating a development environment. As such it will run hooks. All Python packages propagate the interpreter to the build environment and thereby the hooks. The solution? Use buildEnv or symlinkJoin.

[polygon]

Not sure if this was closed prematurely. If yq is primarily a Python application, I believe it should not live in python-modules and should also use buildPythonApplication instead of buildPythonPackage which would prevent this issue I believe.

[teto]

yq is written as buildPythonPackage but in all-packages.nix, it is defined as yq = python3.pkgs.toPythonApplication python3.pkgs.yq;. I believe toPythonApplication should strip python from the propagated inputs: I dont want to develop with it, I want the application. Otherwise I would have selected the python module yq

[Artturin]

yq is written as buildPythonPackage but in all-packages.nix, it is defined as yq = python3.pkgs.toPythonApplication python3.pkgs.yq;. I believe toPythonApplication should strip python from the propagated inputs: I dont want to develop with it, I want the application. Otherwise I would have selected the python module yq

nixpkgs/pkgs/top-level/python-packages.nix

Lines 82 to 91 in 2707877

	toPythonApplication = drv:
	drv.overrideAttrs( oldAttrs: {
	passthru = (oldAttrs.passthru or {}) // {
	# Remove Python prefix from name so we have a "normal" name.
	# While the prefix shows up in the store path, it won't be
	# used by `nix-env`.
	name = removePythonPrefix oldAttrs.name;
	pythonModule = false;
	};
	});

propagatedBuildInputs is not stripped
adding propagatedBuildInputs = [];

fails with

python3.9-yq> ERROR: Could not find a version that satisfies the requirement toml>=0.10.0 (from yq) (from versions: none)
python3.9-yq> ERROR: No matching distribution found for toml>=0.10.0

[FRidh]

I believe toPythonApplication should strip python from the propagated inputs

That's on purpose not done to avoid a rebuild.

[teto]

That's on purpose not done to avoid a rebuild.

ha I seebut that may be worth a comment then, do you mind if I add one ?

[FRidh]

ha I seebut that may be worth a comment then, do you mind if I add one ?

Not at all, good idea!

[teto]

@FRidh I've opened #179318 . Feel free to push directly to the branch if you are not happy with phrasing.
While writing the PR, I wondered if we could not have the propagatedBuildInputs be appended to the PATH rather than prepended ?

[FRidh]

While writing the PR, I wondered if we could not have the propagatedBuildInputs be appended to the PATH rather than prepended ?

What is the motivation for that? So you can override the search path? This would get tricky if you have a tool installed in your profile, and then work with a new version and surprisingly you get the old version from your profile. OTOH, I also think standard environment variables should typically still function.

[teto]

what I had in mind is :

• for python applications, append "python" to path so that in a nix-shell, my selected python takes precedence (and the python application works regardless of the presence of a python in PATH)
• for python libraries, keep as is, aka prepend "python" to path

but I suppose it wont avoid a rebuild so that's not a good idea anyway, let's forget about it xD