Execute jobs in separate conda environments

[jan-janssen]

Caution: Executing pyiron_base jobs in separate conda environments introduces additional overhead. When possible it is highly recommended to create one conda environment which combines all dependencies.

Feature:

For specific application it is required to execute parts of a given workflow in separate conda environments. For example in BAMresearch/NFDI4IngScientificWorkflowRequirements. For these special cases this pull request adds the functionality to pyiron_base to execute individual jobs in separate conda environments. Additional conda environments can be created with the Project() object using pr.conda_environment.create() and can be assigned to individual jobs using job.server.conda_environment_path = pr.conda_environment.py312.

Example:

pr = Project("Test")
pr.conda_environment.create(env_name="py312", env_file="env.yaml")
pr.create_job_class(
    class_name="PythonVersionJob",
    executable_str="python --version",
)
job = pr.create.job.PythonVersionJob(job_name="job_conda_path")
job.server.conda_environment_path = pr.conda_environment.py312
job.run()

[jan-janssen]

It would be great if we would have python bindings for the creation of conda environments - pyiron/conda_subprocess#4

[niklassiemer]

I see the value of the addition in principle, however, tab completion would only show create and the getitem is hidden. Just using pr.conda_environment would only inform that this is a property.
Therefore, I would add a get method and provide a reasonable repr such that things are clear from the user perspective.

[jan-janssen]

I see the value of the addition in principle, however, tab completion would only show create and the getitem is hidden. Just using pr.conda_environment would only inform that this is a property. Therefore, I would add a get method and provide a reasonable repr such that things are clear from the user perspective.

I agree a __repr__ function should be added for visualisation. Still I do not understand your concerns about the tab completion, as I overload the __dir__ method. I just missed to add the create method in the __dir__ function.

[niklassiemer]

Yes, sorry I misread getattr and getitem and was confused!

[niklassiemer]

I did understand correctly, that you only use the other conda environment for starting the shell script running the actual computation? Or is it required that pyiron(_base) is also installed in the other conda environment?

[jan-janssen]

The creation of conda environments fails on windows, so the windows tests are skipped for now.

[jan-janssen]

I did understand correctly, that you only use the other conda environment for starting the shell script running the actual computation? Or is it required that pyiron(_base) is also installed in the other conda environment?

That is correct, only the external executable is executed in the different environment. This allows the external environments to be small and simplifies the integration on the pyiron side. Calling python functions in other conda environments is currently not supported.