Detect .venv and add maturin run

[konstin]

Currently, maturin develop will pick up an activated venv or conda and will otherwise fail. Inspired by PEP 704 and the discussion around it, i've implemented a new behaviour where it also picks up a virtualenv called .venv in the current directory or any parent directory, i.e. you don't need to manually activate the venv anymore. You can then use maturin run to actually run your python code, e.g. maturin run -m pytest.

I've implemented this in #1462.

CC @mitsuhiko would this solve #284 (comment)?

[hauntsaninja]

Would you also be willing to support discovery of .venv files (not just directories), as per https://discuss.python.org/t/setting-up-some-guidelines-around-discovering-finding-naming-virtual-environments/22922

[konstin]

Could you summarize what the motivation for having .venv files is, i.e. what is the need for the venv stored somewhere else?

Logic itself isn't much overhead, but i'm worried about how this is will confuse inexperienced users when something with their environment goes wrong and that this would make all error reporting and configuration more complex and documentation way more verbose (e.g. instead of "look inside the .venv" it becomes "if .venv is a directory, look inside, if it is a file, look inside the path pointed to by the text inside the .venv file", users most likely want additional options to create the venv else for automatic venv creation, etc.), so for me this would need a strong use case to be included.

[hauntsaninja]

One thing I like about it is that it makes it really easy to switch between venv's that have different Python versions.

[konstin]

couldn't that be done faster by activating another venv (which has precedence over .venv) instead of editing the file?

[hauntsaninja]

The idea is that this would provide a way for all tools — including things that aren't CLIs, like my IDE — to know what environment I want to use. (Also in practice I'm not editing the file by hand, I have some tool that is)

[hauntsaninja]

I don't think the discuss thread was concluded, although the poll looked positive: https://discuss.python.org/t/setting-up-some-guidelines-around-discovering-finding-naming-virtual-environments/22922/32 so if you're not sure this would be good for your users, that's okay! :-)

[mitsuhiko]

I’m not too familiar with how Python virtualenvs work these days but I world try to mirror how pip installed Python tools work. They basically get the shebang updated to the Python interpreter of the venv injected. I think the equivalent would be to try to discover an adjacent Python interpreter from the maturin executable and the activation script? That assumes maturin is installed in the virtualenv of course.

[nils-werner]

Maybe I'm misunderstanding what maturin does, but this idea sounds a bit like feature creep to me... I don't know any other Python build tool or installer that actually tries to detect and activate a virtualenv? I know that setting PIP_REQUIRE_VIRTUALENV=true will block pip from running outside one, but I am not aware of anything that (automatically) activates one.

The only ones that come to mind are poetry and virtualenv itself, and they are the ones that are actually creating and managing virtualenvs, and not just installing, building, or running something in it.

I think I would like to see maturin encourage the use of a virtualenv, but I don't want it to try and manage it for me. Same with maturin run... I don't think I want maturin to manage my virtualenv, or activate it for me, as there are already plenty of ways to do so.

[konstin]

From my perspective, everything that is required to make maturin just work is in scope. You should be able to do maturin develop/maturin build/maturin publish without any setup beyond installing maturin (and even then there's pip install [-e] . and pipx run maturin), ideally without configuration and without any other tools, so maturin needs to manage everything required to do so. Also note that currently maturin does neither manage nor activate the venv (it can't do that, processes can't change the shell's environment variable without source), it uses and installs into an existing venv.

The only ones that come to mind are poetry and virtualenv itself, and they are the ones that are actually creating and managing virtualenvs, and not just installing, building, or running something in it.

The new generation of package managers such a poetry, pdm and hatch all include similar venv management. rye additionally installs shims that make python automatically launches the venv python (docs). PEP 704 was unfortunately withdrawn, otherwise pip would also do that. Unlike e.g. in rust there's no infrastructure we could build on, so maturin has to implement these things too. With PEP 621 it has become feasible for maturin to also create the venv and install the dependencies, but we're currently lacking a crossplatform venv library in rust and the facilities for resolving and installing the (dev) dependencies, even if we ignore the lack of a common lockfile standard.

Ask differently, what problem does it cause if maturin uses an existing the venv where it would otherwise fail?

[nils-werner]

You should be able to do maturin develop/maturin build/maturin publish without any setup beyond installing maturin

But in order to install maturin I've already created a venv and chosen a package installer, no? Do I then have to switch gears and stop using those tools? Doesn't that make installing maturin harder and not easier?

With PEP 621 it has become feasible for maturin to also create the venv and install the dependencies

You mean Python requirements? Why? There are already numerous tools that try to do that. Adding yet another option will make things harder, not easier....

Reading #284 it seems like you heard the criticism, but then went in the opposite direction... To me it reads like people want maturin to

1. assume less about my Python environment
2. play nicely with being called like a console_script (the established shortcut to skip venv activation)

Yet we ended up with maturin now wanting to create and manage my virtualenv?

Ask differently, what problem does it cause if maturin uses an existing the venv where it would otherwise fail?

You'll be limiting yourself to compatible venv structures only. Not having a venv is a structure that is very common in containers! Plus people will get confused, because that's not how Python tools usually operate. Usually they do the two things mentioned above.

[konstin]

But in order to install maturin I've already created a venv and chosen a package installer, no? Do I then have to switch gears and stop using those tools? Doesn't that make installing maturin harder and not easier?

Not necessarily, many people have a global install or use something like pipx run maturin.

You mean Python requirements? Why? There are already numerous tools that try to do that. Adding yet another option will make things harder, not easier....

assume less about my Python environment

We're not adding a standard, on the contrary, there used to be various more-or-less specified format such as as tool.poetry.dependencies, Pipfile, various requirements.txt subsets, but now there's the single PEP 621! This finally allows different tools to do the same action on dependencies without breaking the user experience (just like your IDE can install, add and upgrade packages with or without using the canonical tool behind it, it's abiding the same standards so the interaction is seamless ). maturin is already installing dependencies, it's just doing it through pip atm. I strongly believe that requiring users to use multiple different tools for the develop/build/publish loop of a package is a bad UX and prefer a cargo like approach, and especially new users gain a lot from a .venv convention (even though PEP 704 unfortunately failed). I would have loved for maturin to just be a plugin to the "cargo for python" but such tool or plugin interface didn't exist back when i started maturin, iirc poetry didn't even have a plugin interface back then.

play nicely with being called like a console_script (the established shortcut to skip venv activation)

I would like to do this but this fails when installed through pipx because pipx will install maturin in a venv, but the user definitely doesn't want to have their development package installed into the pipx environment. maturin instead checks for the users active venv first.

You'll be limiting yourself to compatible venv structures only. Not having a venv is a structure that is very common in containers! Plus people will get confused, because that's not how Python tools usually operate. Usually they do the two things mentioned above.

Yeah this is an inconvenience in containers, but only in the edge case where you want to maturin develop in the container (build & pip install should work fine, as do PEP 517 builds which also always have to use an isolated environment), is that a use case you have?

[nils-werner]

I strongly believe that requiring users to use multiple different tools for the develop/build/publish loop of a package is a bad UX and prefer a cargo like approach, and especially new users gain a lot from a .venv convention (even though PEP 704 unfortunately failed)

You're right, but requiring users to break with their tried and trusted tools is bad UX too. Python already has virtualenv and package managers, why force them to break with it? That's like asking people to abandon cargo when they're building a Rust+Python project.

Not necessarily, many people have a global install or use something like pipx run maturin.
[...]
I would like to do this but this fails when installed through pipx because pipx will install maturin in a venv, but the user definitely doesn't want to have their development package installed into the pipx environment. maturin instead checks for the users active venv first.

Why do you want to install maturin using pipx in the first place? pipx is meant to install and run commandline programs in isolation, not create development or deployment environments, or combine commands with another virtualenv. As a Python developer I would never use pipx for any of my development, and would always install maturin in the same virtualenv that will be used for maturin develop output. (I could maybe also imagine people installing maturin using their OS package manager, but I would strongly recommend against it.)

We're not adding a standard, on the contrary, there used to be various more-or-less specified format such as as tool.poetry.dependencies, Pipfile, various requirements.txt subsets, but now there's the single PEP 621!

Sure, a dependency format, but I'm talking about package installers. Reading a dependency list is easy, solving and installing them is the hard part.

but only in the edge case where you want to maturin develop in the container, is that a use case you have?

To me that's not an edgecase but basically every CI pipeline out there.

[flying-sheep]

I don’t like the idea. .venv is just a convention. It’s possible to e.g. use hatch or tox together with maturin, which use different paths. What maturin currently does perfectly integrates with all tools. Not every tool needs a slightly different opinionated command runner.