Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[documentation] Compare available executors #253

Closed
jan-janssen opened this issue Jan 23, 2024 · 1 comment
Closed

[documentation] Compare available executors #253

jan-janssen opened this issue Jan 23, 2024 · 1 comment

Comments

@jan-janssen
Copy link
Member

jan-janssen commented Jan 23, 2024

Compare different kinds of executors to unify the interface:

Name Package Scaling Interface Comment
ThreadPoolExecutor standard library single node Python
ProcessPoolExecutor standard library single node Python fast on a single node
MPIPoolExecutor mpi4py MPI Python
FluxExecutor flux Flux Shell
Executor pysqa SLURM Python hdf5 support
PyFluxExecutor pympipool Flux Python
PyMPIExecutor pympipool MPI Python
PySlurmExecutor pympipool SLURM Python
SubprocessExecutor pympipool Subprocess Shell conda support

Open Questions:

  • Is there any SSH support for existing Executors? How to handle transfer of files on remote locations?
  • Can shell tasks and python tasks be combined in a single submission? With flux this should not be an issue and similarly with slurm.
@jan-janssen
Copy link
Member Author

By now all these executors are unified and available via from executorlib import Executor. The different capabilities are now discussed as part of the different modes:

  • Local Mode for running on a workstation computer.
  • HPC Submission Mode for submitting individual Python functions to the job scheduler of an HPC.
  • HPC Allocation Mode for submitting Python functions within the allocation of a given HPC job scheduler.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant