Where/How to install dependencies in a Shared Virtual Environment? Executors are failing to resolve flake8/pytest

[Tungsten78]

I have a single @nxlv library in my repo and ran the migration generator to centralize the dependencies.

• I've installed dependencies at the root via poetry install.
• nx run my-project:lint and nx run my-project:test fail because poetry cannot find the commands flake8 and pytest respectively
• if these are installed globally things seem to work, but I assume the intent is to use the dev dependencies

From the blog

When you use the Shared Virtual Environment approach, you must activate the root virtual environment in your terminal before running any linting, testing, build, add, update, or delete commands.

• not sure I understand how to use poetry update, I would like to be able to use nx affected:lint and other multi project commands
• at any rate, poetry update my-project doesn't seem to change things

I did try with the default isolated dependencies project.

• once I invoked the install executor on the project, the lint and test executors resolve the dependencies ok

disclaimer: never used poetry before today. Any light that can be shed on what I'm doing wrong would be appreciated.

[Tungsten78]

@lucasvieirasilva,

From the blog again,

When you use the Shared Virtual Environment approach, you must activate the root virtual environment in your terminal before running any linting, testing, build, add, update, or delete commands.

It's not clear to me how to "activate". Does this refer to installing the dependencies and having poetry establish each .venv?

• Is this best done across the repo with nx run-many --target install?

[lucasvieirasilva]

Hey @Tungsten78, actually, I updated the Install Local Dependency section of the blog, it was referencing the old project generator.

Can you re-run the blog steps again and see if get any other issues?

The @nxlv/python:poetry-project is the new version, in the next releases the @nxlv/python:project is going to be removed, I just forget to fix one of the references in the blog page.

It's not clear to me how to "activate". Does this refer to installing the dependencies and having poetry establish each .venv?

it means you have to activate the python virtual environment in your terminal.

There are a lot of ways of doing that, using poetry you can run poetry shell in our terminal (in the root of the workspace) and the virtual environment is gonna be activated.

Got it?

[Tungsten78]

I thought I had used the new generator but will make double sure. We used the single version policy every where so this was the default I attempted. Regarding activation the concern is with CI. We clone, run npm install, then nx affected/runmany. Currently this works without extra steps with typescript and dotnet. Nx-dotnet adds an npm prepare step to call a restore generator. I think this is an analog to the install executor in nxlv/python. Using the non shared mode, I was able to add a “nx run-many -t install” to have all the python projects setup their venv. Is this equivalent?

…

On Thu, Jun 22, 2023 at 2:44 PM Lucas Vieira ***@***.***> wrote: Hey @Tungsten78 <https://github.com/Tungsten78>, actually, I updated the Install Local Dependency section of the blog, it was referencing the old project generator. Can you re-run the blog steps again and see if get any other issues? The @nxlv/python:poetry-project is the new version, in the next releases the @nxlv/python:project is going to be removed, I just forget to fix one of the references in the blog page. It's not clear to me how to "activate". Does this refer to installing the dependencies and having poetry establish each .venv? it means you have to activate the python virtual environment in your terminal. There are a lot of ways of doing that, using poetry you can run poetry shell in our terminal (in the root of the workspace) and the virtual environment is gonna be activated. Got it? — Reply to this email directly, view it on GitHub <#128 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA5BBGVHUVJF34CA25DX2FLXMSVCLANCNFSM6AAAAAAZPN4Z4A> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[lucasvieirasilva]

Using the non-shared mode, I was able to add a “nx run-many -t install” to
have all the Python projects setup their venv. Is this equivalent?

Correct, although, I don't recommend using non-shared environment, because the installation time might increase based on how many projects you have in the workspace.

I've used the shared venv approach with Azure devops and you only need to activate the venv before the affected command.

You don't need to activate for each project, only in the root before running the command.

Something like:

source .venv/bin/activate && npx nx affected ...

Please let me know if that doesn't work in your tests.

[Tungsten78]

I can confirm I had been using the correct project generator (no change in behavior)- the issue seems to be the lack of activation.

My workflow:

1. Open Nx workspace
2. Add nxlv project (my-project) using poetry-project generator
3. Run migrate-to-shared-venv generator
4. nx lint my-project <-- fails to find flake8

If I activate ahead of 4 it works.

What I'm noticing is the activation is scoped (I realize this is the whole point); however, I don't want to change our dev workflow where most devs won't deal directly with the python projects. Nx abstracts this nicely by hiding the details behind the common targets (lint/test/etc). What I want to avoid is the pre-requisite of having an activation at the root.

Is there a way to have this persisted? I would like to embed something in the npm prepare such that post clone / npm i the Nx workspace just works.

[lucasvieirasilva]

Hey @Tungsten78, I've created this issue to implement this feature #132.

[lucasvieirasilva]

Hey, @Tungsten78 I've released a new version that supports auto-activate the shared virtual environment as we are discussing in this thread.

Could you check this PR description please: #138

[Tungsten78]

Backing up a bit more, the same steps without npx nx generate @nxlv/python:migrate-to-shared-venv

• nx lint produces the same flake8 missing

I am using python 3.11.4 (if it matters).
I'm having to change

[tool.poetry.dependencies]
python = ">=3.9,<3.11"

to

  [tool.poetry.dependencies]
  python = ">=3.9"

Sorry for omitting this detail

[lucasvieirasilva]

well, the Python version shouldn't matter for this behavior, I've noticed you are using Windows, could you try with Windows WSL? maybe the way I'm activating the venv programmatically doesn't work properly on windows

[Tungsten78]

@lucasvieirasilva - the version seemed to be the issue.

• There were errors hiding in the command output I hadn't noticed.
• Looks like npx nx generate @nxlv/python:migrate-to-shared-venv is what does the initial activation at the root (installing flake8/pytest)

So the steps as indicated now work, however in ci we still need this initial activation (I think):

• while nx lint my-project does activate the project, it looks to ignore the root dev dependencies
• whereas, poetry install at the root looks to walk the tree activating root + project. flake8 is found subsequently.

I'll leave poetry install in my prepare step. This works and seems mostly benign.

[lucasvieirasilva]

@Tungsten78 got it, well, since you move to a single versioning approach, you always need to run poetry install at the root before running the commands, that's on purpose.