diff --git a/README.md b/README.md index a1516702..54287344 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,7 @@ # pympipool - up-scale python functions for high performance computing [![Unittests](https://github.com/pyiron/pympipool/actions/workflows/unittest-openmpi.yml/badge.svg)](https://github.com/pyiron/pympipool/actions/workflows/unittest-openmpi.yml) [![Coverage Status](https://coveralls.io/repos/github/pyiron/pympipool/badge.svg?branch=main)](https://coveralls.io/github/pyiron/pympipool?branch=main) +[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/pyiron/pympipool/HEAD?labpath=notebooks%2Fexamples.ipynb) Up-scaling python functions for high performance computing (HPC) can be challenging. While the python standard library provides interfaces for multiprocessing and asynchronous task execution, namely diff --git a/binder/environment.yml b/binder/environment.yml new file mode 100644 index 00000000..34f38cda --- /dev/null +++ b/binder/environment.yml @@ -0,0 +1,11 @@ +channels: +- conda-forge +dependencies: +- python +- numpy +- openmpi +- cloudpickle =3.0.0 +- flux-core +- mpi4py =3.1.5 +- tqdm =4.66.1 +- pyzmq =25.1.1 diff --git a/binder/kernel.json b/binder/kernel.json new file mode 100644 index 00000000..16c69cf5 --- /dev/null +++ b/binder/kernel.json @@ -0,0 +1,16 @@ +{ + "argv": [ + "flux", + "start", + "/srv/conda/envs/notebook/bin/python", + "-m", + "ipykernel_launcher", + "-f", + "{connection_file}" + ], + "display_name": "Flux", + "language": "python", + "metadata": { + "debugger": true + } +} \ No newline at end of file diff --git a/binder/postBuild b/binder/postBuild new file mode 100644 index 00000000..f5cee2b7 --- /dev/null +++ b/binder/postBuild @@ -0,0 +1,6 @@ +# jupyter kernel +mkdir -p /home/jovyan/.local/share/jupyter/kernels/flux +cp binder/kernel.json /home/jovyan/.local/share/jupyter/kernels/flux + +# install pympipool +pip install . \ No newline at end of file diff --git a/notebooks/examples.ipynb b/notebooks/examples.ipynb new file mode 100644 index 00000000..90596a0b --- /dev/null +++ b/notebooks/examples.ipynb @@ -0,0 +1 @@ +{"metadata":{"kernelspec":{"name":"flux","display_name":"Flux","language":"python"},"language_info":{"name":"python","version":"3.11.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":5,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Examples\nThe `pympipool.Executor` extends the interface of the [`concurrent.futures.Executor`](https://docs.python.org/3/library/concurrent.futures.html#module-concurrent.futures)\nto simplify the up-scaling of individual functions in a given workflow.","metadata":{},"id":"19bad499-5a97-425c-beec-dcd88d693d4c"},{"cell_type":"markdown","source":"## Compatibility\nStarting with the basic example of `1+1=2`. With the `ThreadPoolExecutor` from the [`concurrent.futures`](https://docs.python.org/3/library/concurrent.futures.html#module-concurrent.futures)\nstandard library this can be written as: ","metadata":{},"id":"f752ec8d-50b8-46fb-86f2-08a9126f1a39"},{"cell_type":"code","source":"from concurrent.futures import ThreadPoolExecutor","metadata":{"trusted":true},"execution_count":1,"outputs":[],"id":"584cd590-acaf-48d7-a5b5-e4049a9626b7"},{"cell_type":"code","source":"with ThreadPoolExecutor(\n max_workers=1,\n) as exe:\n future = exe.submit(sum, [1, 1])\n print(future.result())","metadata":{"trusted":true},"execution_count":2,"outputs":[{"name":"stdout","text":"2\n","output_type":"stream"}],"id":"73673e42-2c68-4b91-b6ff-db1ecb2c0587"},{"cell_type":"markdown","source":"In this case `max_workers=1` limits the number of threads uses by the `ThreadPoolExecutor` to one. Then the `sum()` \nfunction is submitted to the executor with a list with two ones `[1, 1]` as input. A [`concurrent.futures.Future`](https://docs.python.org/3/library/concurrent.futures.html#module-concurrent.futures)\nobject is returned. The `Future` object allows to check the status of the execution with the `done()` method which \nreturns `True` or `False` depending on the state of the execution. Or the main process can wait until the execution is \ncompleted by calling `result()`. ","metadata":{},"id":"32156314-02a9-4210-8a8c-94afe09b64f6"},{"cell_type":"markdown","source":"The result of the calculation is `1+1=2`. ","metadata":{},"id":"b750d830-bd0d-4474-9f70-913d0b9d6b8a"},{"cell_type":"markdown","source":"The `pympipool.Executor` class extends the interface of the [`concurrent.futures.Executor`](https://docs.python.org/3/library/concurrent.futures.html#module-concurrent.futures) \nclass by providing more parameters to specify the level of parallelism. In addition, to specifying the maximum number \nof workers `max_workers` the user can also specify the number of cores per worker `cores_per_worker` for MPI based \nparallelism, the number of threads per core `threads_per_core` for thread based parallelism and the number of GPUs per\nworker `gpus_per_worker`. Finally, for those backends which support over-subscribing this can also be enabled using the \n`oversubscribe` parameter. All these parameters are optional, so the `pympipool.Executor` can be used as a drop-in \nreplacement for the [`concurrent.futures.Executor`](https://docs.python.org/3/library/concurrent.futures.html#module-concurrent.futures).","metadata":{},"id":"4fbf72a2-0e0e-43ce-be8f-db3489c4eafe"},{"cell_type":"markdown","source":"The previous example is rewritten for the `pympipool.Executor`:","metadata":{},"id":"9b5a26e2-3d18-4778-ba10-e3e213b70433"},{"cell_type":"code","source":"from pympipool import Executor ","metadata":{"trusted":true},"execution_count":3,"outputs":[],"id":"60373c38-63f8-48dc-be0f-ddb71ebf88f8"},{"cell_type":"code","source":"with Executor(\n max_workers=1, \n cores_per_worker=1, \n threads_per_core=1, \n gpus_per_worker=0, \n oversubscribe=False\n) as exe:\n future = exe.submit(sum, [1,1])\n print(future.result())","metadata":{"trusted":true},"execution_count":4,"outputs":[{"name":"stdout","text":"2\n","output_type":"stream"}],"id":"fd755b28-ff01-4530-9099-001cac151e31"},{"cell_type":"markdown","source":"The result of the calculation is again `1+1=2`.","metadata":{},"id":"44c4bc4b-cf97-461e-98e7-62bcdb8caff2"},{"cell_type":"markdown","source":"Beyond pre-defined functions like the `sum()` function, the same functionality can be used to submit user-defined \nfunctions. In the following example a custom summation function is defined: ","metadata":{},"id":"331aed93-806a-4057-ab9c-19479190f472"},{"cell_type":"code","source":"def calc(*args):\n return sum(*args)","metadata":{"trusted":true},"execution_count":5,"outputs":[],"id":"cdeb8710-b328-463d-a436-82d6756e76b3"},{"cell_type":"markdown","source":"In contrast to the previous example where just a single function was submitted to a single worker, in this case a total\nof four functions is submitted to a group of two workers `max_workers=2`. Consequently, the functions are executed as a\nset of two pairs. ","metadata":{},"id":"d5efa995-d4d4-4f9c-a7e6-38dd66143535"},{"cell_type":"code","source":"with Executor(max_workers=2) as exe:\n fs_1 = exe.submit(calc, [2, 1])\n fs_2 = exe.submit(calc, [2, 2])\n fs_3 = exe.submit(calc, [2, 3])\n fs_4 = exe.submit(calc, [2, 4])\n print([\n fs_1.result(), \n fs_2.result(), \n fs_3.result(), \n fs_4.result(),\n ])","metadata":{"trusted":true},"execution_count":6,"outputs":[{"name":"stdout","text":"[3, 4, 5, 6]\n","output_type":"stream"}],"id":"82033832-7ccd-4c67-a1fb-57f55710b77c"},{"cell_type":"markdown","source":"The snippet can be executed with any python interpreter. It returns the corresponding sums as expected. The same can be achieved with the built-in [`concurrent.futures.Executor`](https://docs.python.org/3/library/concurrent.futures.html#module-concurrent.futures)\nclasses. Still one advantage of using the `pympipool.Executor` rather than the built-in ones, is the ability to execute \nthe same commands in interactive environments like [Jupyter notebooks](https://jupyter.org). This is achieved by using \n[cloudpickle](https://github.com/cloudpipe/cloudpickle) to serialize the python function and its parameters rather than\nthe regular pickle package. ","metadata":{},"id":"86838528-312e-46cc-b022-0c946bf95037"},{"cell_type":"markdown","source":"For backwards compatibility with the [`multiprocessing.Pool`](https://docs.python.org/3/library/multiprocessing.html) \nclass the [`concurrent.futures.Executor`](https://docs.python.org/3/library/concurrent.futures.html#module-concurrent.futures)\nalso implements the `map()` function to map a series of inputs to a function. The same `map()` function is also \navailable in the `pympipool.Executor`: ","metadata":{},"id":"4de690ed-661c-4e6e-a97e-c478393d0dc6"},{"cell_type":"code","source":"with Executor(max_workers=2) as exe:\n print(list(exe.map(calc, [[2, 1], [2, 2], [2, 3], [2, 4]])))","metadata":{"trusted":true},"execution_count":7,"outputs":[{"name":"stdout","text":"[3, 4, 5, 6]\n","output_type":"stream"}],"id":"3f06b0c1-5ee1-40c5-82ab-31d77cfcdb46"},{"cell_type":"markdown","source":"The results remain the same. ","metadata":{},"id":"a5d0f249-23bb-4727-8b09-87320ecb98eb"},{"cell_type":"markdown","source":"## Data Handling\nA limitation of many parallel approaches is the overhead in communication when working with large datasets. Instead of\nreading the same dataset repetitively, the `pympipool.Executor` loads the dataset only once per worker and afterwards \neach function submitted to this worker has access to the dataset, as it is already loaded in memory. To achieve this\nthe user defines an initialization function `init_function` which returns a dictionary with one key per dataset. The \nkeys of the dictionary can then be used as additional input parameters in each function submitted to the `pympipool.Executor`.\nThis functionality is illustrated in the following example: ","metadata":{},"id":"580b00ee-6d5b-4ca9-ba36-ff70128c0b6b"},{"cell_type":"code","source":"def calc(i, j, k):\n return i + j + k","metadata":{"trusted":true},"execution_count":8,"outputs":[],"id":"8fe8c750-4dc5-4b26-ad8d-9f755bff3494"},{"cell_type":"code","source":"def init_function():\n return {\"j\": 4, \"k\": 3, \"l\": 2}","metadata":{"trusted":true},"execution_count":9,"outputs":[],"id":"5f943266-1bee-421e-a1b4-583d222b1c99"},{"cell_type":"code","source":"with Executor(max_workers=1, init_function=init_function) as exe:\n fs = exe.submit(calc, 2, j=5)\n print(fs.result())","metadata":{"trusted":true},"execution_count":10,"outputs":[{"name":"stdout","text":"10\n","output_type":"stream"}],"id":"0debe907-b646-4fd5-bae7-46b16645d2f3"},{"cell_type":"markdown","source":"The function `calc()` requires three inputs `i`, `j` and `k`. But when the function is submitted to the executor only \ntwo inputs are provided `fs = exe.submit(calc, 2, j=5)`. In this case the first input parameter is mapped to `i=2`, the\nsecond input parameter is specified explicitly `j=5` but the third input parameter `k` is not provided. So the \n`pympipool.Executor` automatically checks the keys set in the `init_function()` function. In this case the returned \ndictionary `{\"j\": 4, \"k\": 3, \"l\": 2}` defines `j=4`, `k=3` and `l=2`. For this specific call of the `calc()` function,\n`i` and `j` are already provided so `j` is not required, but `k=3` is used from the `init_function()` and as the `calc()`\nfunction does not define the `l` parameter this one is also ignored. ","metadata":{},"id":"72fa803a-ace0-41ea-8090-d64dfd0797cc"},{"cell_type":"markdown","source":"The result is `2+5+3=10` as `i=2` and `j=5` are provided during the submission and `k=3` is defined in the `init_function()`\nfunction.","metadata":{},"id":"1443d216-1add-445a-a662-5b16af6c1443"},{"cell_type":"markdown","source":"## Up-Scaling \nThe availability of certain features depends on the backend `pympipool` is installed with. In particular the thread \nbased parallelism and the GPU assignment is only available with the `pympipool.slurm.PySlurmExecutor` or the \n`pympipool.flux.PyFluxExecutor` backend. The latter is recommended based on the easy installation, the faster allocation \nof resources as the resources are managed within the allocation and no central databases is used and the superior level \nof fine-grained resource assignment which is typically not available on other HPC resource schedulers including the\n[SLURM workload manager](https://www.schedmd.com). The `pympipool.flux.PyFluxExecutor` requires \n[flux framework](https://flux-framework.org) to be installed in addition to the `pympipool` package. The features are \nsummarized in the table below: \n\n| Feature \\ Backend | `PyMpiExecutor` | `PySlurmExecutor` | `PyFluxExecutor` |\n|:--------------------------:|:---------------:|:-----------------:|:----------------:|\n| Thread based parallelism | no | yes | yes | \n| MPI based parallelism | yes | yes | yes |\n| GPU assignment | no | yes | yes |\n| Resource over-subscription | yes | yes | no |\n| Scalability | 1 node | ~100 nodes | no limit |","metadata":{},"id":"8d1e21ec-0b8d-45bf-bfb1-62b3df8e242a"},{"cell_type":"markdown","source":"### Thread-based Parallelism\nThe number of threads per core can be controlled with the `threads_per_core` parameter during the initialization of the \n`pympipool.Executor`. Unfortunately, there is no uniform way to control the number of cores a given underlying library \nuses for thread based parallelism, so it might be necessary to set certain environment variables manually: \n\n* `OMP_NUM_THREADS`: for openmp\n* `OPENBLAS_NUM_THREADS`: for openblas\n* `MKL_NUM_THREADS`: for mkl\n* `VECLIB_MAXIMUM_THREADS`: for accelerate on Mac Os X\n* `NUMEXPR_NUM_THREADS`: for numexpr\n\nAt the current stage `pympipool.Executor` does not set these parameters itself, so you have to add them in the function\nyou submit before importing the corresponding library: ","metadata":{},"id":"f3b9cc80-70ed-4bc8-abf9-62ecbd70b960"},{"cell_type":"code","source":"def calc(i):\n import os\n os.environ[\"OMP_NUM_THREADS\"] = \"2\"\n os.environ[\"OPENBLAS_NUM_THREADS\"] = \"2\"\n os.environ[\"MKL_NUM_THREADS\"] = \"2\"\n os.environ[\"VECLIB_MAXIMUM_THREADS\"] = \"2\"\n os.environ[\"NUMEXPR_NUM_THREADS\"] = \"2\"\n import numpy as np\n return i","metadata":{"trusted":true},"execution_count":11,"outputs":[],"id":"fbf5f7b2-eb3e-4a81-bae8-e429747300a0"},{"cell_type":"markdown","source":"Most modern CPUs use hyper-threading to present the operating system with double the number of virtual cores compared to\nthe number of physical cores available. So unless this functionality is disabled `threads_per_core=2` is a reasonable \ndefault. Just be careful if the number of threads is not specified it is possible that all workers try to access all \ncores at the same time which can lead to poor performance. So it is typically a good idea to monitor the CPU utilization\nwith increasing number of workers. ","metadata":{},"id":"334619d0-8d95-419e-885c-e5bc05747584"},{"cell_type":"markdown","source":"Specific manycore CPU models like the Intel Xeon Phi processors provide a much higher hyper-threading ration and require\na higher number of threads per core for optimal performance. ","metadata":{},"id":"7c3146c1-8722-4b67-ab21-c250b8e7c9dd"},{"cell_type":"markdown","source":"### MPI Parallel Python Functions\nBeyond thread based parallelism, the message passing interface (MPI) is the de facto standard parallel execution in \nscientific computing and the [`mpi4py`](https://mpi4py.readthedocs.io) bindings to the MPI libraries are commonly used\nto parallelize existing workflows. The limitation of this approach is that it requires the whole code to adopt the MPI\ncommunication standards to coordinate the way how information is distributed. Just like the `pympipool.Executor` the \n[`mpi4py.futures.MPIPoolExecutor`](https://mpi4py.readthedocs.io/en/stable/mpi4py.futures.html#mpipoolexecutor) \nimplements the [`concurrent.futures.Executor`](https://docs.python.org/3/library/concurrent.futures.html#module-concurrent.futures)\ninterface. Still in this case eah python function submitted to the executor is still limited to serial execution. The\nnovel approach of the `pympipool.Executor` is mixing these two types of parallelism. Individual functions can use\nthe [`mpi4py`](https://mpi4py.readthedocs.io) library to handle the parallel execution within the context of this \nfunction while these functions can still me submitted to the `pympipool.Executor` just like any other function. The\nadvantage of this approach is that the users can parallelize their workflows one function at the time. \n\nThe following example illustrates the submission of a simple MPI parallel python function: ","metadata":{},"id":"b4976d45-0f4e-496c-8173-9631f512135b"},{"cell_type":"code","source":"def calc(i):\n from mpi4py import MPI\n size = MPI.COMM_WORLD.Get_size()\n rank = MPI.COMM_WORLD.Get_rank()\n return i, size, rank","metadata":{"trusted":true},"execution_count":12,"outputs":[],"id":"cfa072a4-f88f-45b0-be94-a78f0edad513"},{"cell_type":"code","source":"with Executor(cores_per_worker=2) as exe:\n fs = exe.submit(calc, 3)\n print(fs.result())","metadata":{"trusted":true},"execution_count":13,"outputs":[{"name":"stdout","text":"[(3, 2, 0), (3, 2, 1)]\n","output_type":"stream"}],"id":"fd036b03-085d-4850-b11e-537c8fd476d5"},{"cell_type":"markdown","source":"The `calc()` function initializes the [`mpi4py`](https://mpi4py.readthedocs.io) library and gathers the size of the \nallocation and the rank of the current process within the MPI allocation. This function is then submitted to an \n`pympipool.Executor` which is initialized with a single worker with two cores `cores_per_worker=2`. So each function\ncall is going to have access to two cores. \n\nJust like before the script can be called with any python interpreter even though it is using the [`mpi4py`](https://mpi4py.readthedocs.io)\nlibrary in the background it is not necessary to execute the script with `mpiexec` or `mpirun`.","metadata":{},"id":"b4f97426-d8fb-42ef-98ca-135054bd39a7"},{"cell_type":"markdown","source":"The response consists of a list of two tuples, one for each MPI parallel process, with the first entry of the tuple \nbeing the parameter `i=3`, followed by the number of MPI parallel processes assigned to the function call `cores_per_worker=2`\nand finally the index of the specific process `0` or `1`. ","metadata":{},"id":"69dcfdcb-41db-4c3b-a1c5-07ff3be0c9a0"},{"cell_type":"markdown","source":"### GPU Assignment\nWith the rise of machine learning applications, the use of GPUs for scientific application becomes more and more popular.\nConsequently, it is essential to have full control over the assignment of GPUs to specific python functions. In the \nfollowing example the `tensorflow` library is used to identify the GPUs and return their configuration: ","metadata":{},"id":"dc41f241-663c-474e-ae1e-b2365389bc90"},{"cell_type":"raw","source":"import socket\nfrom tensorflow.python.client import device_lib","metadata":{},"id":"6ac9630b-4ab5-4f7f-bf55-812e8189da4f"},{"cell_type":"code","source":"def get_available_gpus():\n local_device_protos = device_lib.list_local_devices()\n return [\n (x.name, x.physical_device_desc, socket.gethostname()) \n for x in local_device_protos if x.device_type == 'GPU'\n ]","metadata":{"trusted":true},"execution_count":14,"outputs":[],"id":"998138f5-f0cb-47a7-ba36-7594b8ec41fc"},{"cell_type":"raw","source":"with Executor(\n max_workers=2, \n gpus_per_worker=1, \n) as exe:\n fs_1 = exe.submit(get_available_gpus)\n fs_2 = exe.submit(get_available_gpus)\n print(fs_1.result(), fs_2.result())","metadata":{},"id":"9d33af22-7b90-4ff7-a434-9c4cd9a930d5"},{"cell_type":"markdown","source":"The additional parameter `gpus_per_worker=1` specifies that one GPU is assigned to each worker. This functionality \nrequires `pympipool` to be connected to a resource manager like the [SLURM workload manager](https://www.schedmd.com)\nor preferably the [flux framework](https://flux-framework.org). The rest of the script follows the previous examples, \nas two functions are submitted and the results are printed. ","metadata":{},"id":"8dc7a989-908a-48a6-8d06-ac1e24173f5c"},{"cell_type":"markdown","source":"To clarify the execution of such an example on a high performance computing (HPC) cluster using the [SLURM workload manager](https://www.schedmd.com)\nthe submission script is given below: ","metadata":{},"id":"d1a17c6c-41ee-4595-913e-4af7272010a5"},{"cell_type":"raw","source":"#!/bin/bash\n#SBATCH --nodes=2\n#SBATCH --gpus-per-node=1\n#SBATCH --get-user-env=L\n\npython test_gpu.py","metadata":{},"id":"11ce332b-d2c1-4434-84c4-1e523e430848"},{"cell_type":"markdown","source":"The important part is that for using the `pympipool.slurm.PySlurmExecutor` backend the script `test_gpu.py` does not \nneed to be executed with `srun` but rather it is sufficient to just execute it with the python interpreter. `pympipool`\ninternally calls `srun` to assign the individual resources to a given worker. ","metadata":{},"id":"14bf6228-db64-406b-b04f-4d23daaa836d"},{"cell_type":"markdown","source":"For the more complex setup of running the [flux framework](https://flux-framework.org) as a secondary resource scheduler\nwithin the [SLURM workload manager](https://www.schedmd.com) it is essential that the resources are passed from the \n[SLURM workload manager](https://www.schedmd.com) to the [flux framework](https://flux-framework.org). This is achieved\nby calling `srun flux start` in the submission script: ","metadata":{},"id":"66e3be02-c11c-4053-9600-6bcfefefb127"},{"cell_type":"raw","source":"#!/bin/bash\n#SBATCH --nodes=2\n#SBATCH --gpus-per-node=1\n#SBATCH --get-user-env=L\n\nsrun flux start python test_gpu.py","metadata":{},"id":"aa0e2abf-7ab2-464f-a341-b93f91fbdd99"},{"cell_type":"markdown","source":"As a result the GPUs available on the two compute nodes are reported: ","metadata":{},"id":"6c84fb7d-4285-4d73-8f1e-7cb88050eb85"},{"cell_type":"raw","source":">>> [('/device:GPU:0', 'device: 0, name: Tesla V100S-PCIE-32GB, pci bus id: 0000:84:00.0, compute capability: 7.0', 'cn138'),\n>>> ('/device:GPU:0', 'device: 0, name: Tesla V100S-PCIE-32GB, pci bus id: 0000:84:00.0, compute capability: 7.0', 'cn139')]","metadata":{},"id":"a431e015-a309-49ac-9f10-756bda0177fc"},{"cell_type":"markdown","source":"In this case each compute node `cn138` and `cn139` is equipped with one `Tesla V100S-PCIE-32GB`.","metadata":{},"id":"70eb9a19-325e-4179-a196-4417e3f30e19"}]} \ No newline at end of file