fix: do not assume that /usr/bin/env exists on macOS

[tie]

This changes fixes build error when /usr/bin/env is not available (for example, when building in Nix sandbox). It makes gyp-generated build configurations invoke Python directly on the gyp tools (similar to gyp-win-tool that uses sys.executable on Windows).

See also NixOS/nixpkgs#261820 and NixOS/nixpkgs#248708

[tie]

Force-pushed to fix issues reported by ruff.

[cclauss]

Not a fan of this. Why not just create or symlink to /usr/bin/env ?

[tie]

Why not just create or symlink to /usr/bin/env?

In general, only build directory is writable in sandbox, and a limited set of host directories is readable.

While it should be possible to pass /usr/bin/env as an impure derivation dependency, as far as I know, this is usually used only for bootstrapping stdenv, and not in the packages themselves. Note that for Node.js, /usr/bin/env would also need to be a propagated impure dependency since some packages (e.g. iosevka) depend on vendored node-gyp (in Node.js, gyp exists in deps/npm/node_modules/node-gyp/gyp and tools/gyp, the latter is used to build Node.js itself).

Alternatively, Nixpkgs can patch shebang lines in pylib/*_tool.py to absolute Python executable path (rewrite #!/usr/bin/env python3 → #!/nix/store/g5cm6iik6p4k39cj9k7a6sg2p09hl7wf-python3-3.10.12/bin/python3), e.g. NixOS/nixpkgs#261820 (comment) and NixOS/nixpkgs#248708, but I don’t really like doing that since technically it is a change to the upstream source and not necessary with this PR.

[tie]

@cclauss, hi, are there any updates on this issue? I’d like to move forward with NixOS/nixpkgs#261820 and NixOS/nixpkgs#262124, and ideally I don’t want to apply this patch without getting it merged upstream (see also my previous comment).

[legendecas]

Why not set up /usr/bin/env in the sandbox macOS environment? The default macOS distribution comes with it.

[wegank]

Sandbox in the Nix term means that only packages built by Nix (which are in /nix/store) are available in the build environment. Access to /usr/bin/env is hence prohibited.

[legendecas]

If a sandbox is going to provide a macOS-like environment, it should provide /usr/bin/env as a default macOS does.

[wegank]

It depends on what you think a build environment should look like.

• In a Homebrew build environment, for example, there are a lot of Apple stuffs, including binaries in /usr/bin and pre-installed toolchains in /Library/Developer/CommandLineTools.
• In a Nix sandbox, you won't have access to /usr/bin or /Library/Developer/CommandLineTools, but to LLVM Clang, GNU Core Utilities, and other open-source components, all built from source and isolated in /nix/store.

Both models are valid, tested in practice (165,000 PRs in Homebrew Core, 270,000 PRs in Nixpkgs), and can be used to build a multitude of things.

[tie]

@cclauss @legendecas, thanks for taking a look at this, can you please explain what concerns do you have with this change? I understand that we can disable/relax sandbox to a certain extent, but that is really the last resort unless there is some technical reason why this change is not sound.

Gyp already uses sys.executable on Windows hosts, so it’s not like this change introduces some new behavior, but rather makes it consistent across platforms. Am I missing something here?

E.g. (%(python)s gyp-win-tool)

gyp-next/pylib/gyp/generator/ninja.py

Lines 2147 to 2157 in 4a8e328

	def _AddWinLinkRules(master_ninja, embed_manifest):
	"""Adds link rules for Windows platform to |master_ninja|."""
	def FullLinkCommand(ldcmd, out, binary_type):
	resource_name = {"exe": "1", "dll": "2"}[binary_type]
	return (
	"%(python)s gyp-win-tool link-with-manifests $arch %(embed)s "
	'%(out)s "%(ldcmd)s" %(resname)s $mt $rc "$intermediatemanifest" '
	"$manifests"
	% {
	"python": sys.executable,

[aduh95]

Wouldn’t this change be problematic if the python executable path contains some shell special char (e.g. ; or some white space)?

[tie]

Yes, but I don't think the existing code base handles that well too. Some of the formatting arguments should be using shlex.quote, but they are not. I think it would be rather confusing to correctly escape in this case but not in others.

[cclauss]

I hesitate because the assumption is correct for the vast majority of our Mac users and the others can use the --python option. NixOS should provide a pervasive solution instead of requiring all tools to make Nix-specific patches.

[aduh95]

IMO as long as NixOS community keeps sending patches and having support for their use case is not something that adds maintenance burden for us, there are no downsides for us. AFAICT this patch does not break the common use-case we actively support, it does not make the code less maintainable, so I think we should land it.

[tie]

I hesitate because the assumption is correct for the vast majority of our Mac users and the others can use the --python option. […]

I don’t think node-gyp currently rewrites the shebang line when generating tools (see permalink below), so it’d still be using #!/usr/bin/env python3 instead of the Python interpreter set via --python option.

gyp-next/pylib/gyp/common.py

Lines 529 to 548 in 4a8e328

	# Slurp input file.
	source_path = os.path.join(
	os.path.dirname(os.path.abspath(__file__)), "%s_tool.py" % prefix
	)
	with open(source_path) as source_file:
	source = source_file.readlines()
	# Set custom header flags.
	header = "# Generated by gyp. Do not edit.\n"
	mac_toolchain_dir = generator_flags.get("mac_toolchain_dir", None)
	if flavor == "mac" and mac_toolchain_dir:
	header += "import os;\nos.environ['DEVELOPER_DIR']='%s'\n" % mac_toolchain_dir
	# Add header and write it out.
	tool_path = os.path.join(out_path, "gyp-%s-tool" % prefix)
	with open(tool_path, "w") as tool_file:
	tool_file.write("".join([source[0], header] + source[1:]))
	# Make file executable.
	os.chmod(tool_path, 0o755)

On that note, would it make sense to change source[0] to #!{sys.executable} instead of formatting tools command invocations with sys.executable? The only downside I can think of in this case is that POSIX shells have weird shebang handling when an executable is a text file and there is something wrong with the interpreter (see https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_01_01) and I usually prefer to avoid shebangs if possible for this reason (though that shouldn’t be an issue in this case, and we already use shebangs).
That could make the change in the PR smaller and hopefully easier to review/maintain. What do you think?

I.e. 6d804d9

[cclauss]

Using #!/usr/bin/env python3 discourages the creation of a venv which results in polluting the system Python with dependencies from multiple projects.

We want to encourage the creation of venvs in which case #!/usr/bin/env python (no 3) will work just fine.

[tie]

I’m confused, I don’t understand how venv’s are related here.

Just to be clear, I’m referring to the shebang lines of the tool scripts, e.g.

gyp-next/pylib/gyp/mac_tool.py

Line 1 in 4a8e328

#!/usr/bin/env python3

You’ve said that we can use --python flag to change the Python installation being used. I assume you mean node-gyp flag,

• https://github.com/nodejs/node-gyp/blob/c495083d991e26de74ec998110d6259675328d65/lib/find-python.js
• https://github.com/nodejs/node-gyp/blob/c495083d991e26de74ec998110d6259675328d65/lib/configure.js#L294

When gyp runs, it copies these scripts to the Makefile directory using CopyTool.

gyp-next/pylib/gyp/common.py

Line 513 in 4a8e328

def CopyTool(flavor, out_path, generator_flags={}):

The code adds “Generated by” header, but doesn’t change the hash bang from #!/usr/bin/env python3 to sys.executable. When the build is run, these files are executed and the OS uses the interpreter from hash bang line to execute the script, /usr/bin/env with python3 argument in this case, so essentially PATH lookup for python3.

However, if, for example, PATH is /path/to/python-3-incompatible/bin:/path/to/python-3.12-good/bin where both directories contain python3 executable, and --python option is set to the lower /path/to/python-3.12-good/bin/python3, we’d still use python-3-incompatible because /usr/bin/env causes PATH lookup.

So, I’m suggesting that we can do 6d804d9 which should also fix the Nix sandbox issue on macOS. This is just another approach for the same issue, but may be better than the PR in its current state if we want to continue using shebangs.

Again, I don’t see how venvs change anything. They are not movable/copyable, so we can safely refer to sys.executable of the virtual environment when generating the project.

[tie]

@cclauss, gentle ping 👀