From d98e9245b1c2dfe2a856973af3bdc313e03c4af2 Mon Sep 17 00:00:00 2001 From: James Corbett Date: Mon, 27 Nov 2023 17:30:33 -0800 Subject: [PATCH] docs: grammar tweaks Problem: some of the grammar in the docs could be improved. Move some commas around and add some dashes. --- docs/source/examples.md | 2 +- docs/source/index.rst | 21 ++++++++++----------- docs/source/installation.md | 6 +++--- 3 files changed, 14 insertions(+), 15 deletions(-) diff --git a/docs/source/examples.md b/docs/source/examples.md index 3e69e749..978cd4d8 100644 --- a/docs/source/examples.md +++ b/docs/source/examples.md @@ -14,7 +14,7 @@ with ThreadPoolExecutor( future = exe.submit(sum, [1, 1]) print(future.result()) ``` -In this case `max_workers=1` limits the number of threads uses by the `ThreadPoolExecutor` to one. Then the `sum()` +In this case `max_workers=1` limits the number of threads used by the `ThreadPoolExecutor` to one. Then the `sum()` function is submitted to the executor with a list with two ones `[1, 1]` as input. A [`concurrent.futures.Future`](https://docs.python.org/3/library/concurrent.futures.html#module-concurrent.futures) object is returned. The `Future` object allows to check the status of the execution with the `done()` method which returns `True` or `False` depending on the state of the execution. Or the main process can wait until the execution is diff --git a/docs/source/index.rst b/docs/source/index.rst index d11a4f5a..d166a2f5 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -8,29 +8,27 @@ pympipool - up-scale python functions for high performance computing Up-scaling python functions for high performance computing (HPC) can be challenging. While the python standard library provides interfaces for multiprocessing and asynchronous task execution, namely `multiprocessing `_ and -`concurrent.futures `_ both are +`concurrent.futures `_, both are limited to the execution on a single compute node. So a series of python libraries have been developed to address the up-scaling of python functions for HPC. Starting in the datascience and machine learning community with solutions -like `dask `_ over more HPC focused solutions like -`fireworks `_ and `parsl `_ up to Python +like `dask `_, over to more HPC-focused solutions like +`fireworks `_ and `parsl `_, up to Python bindings for the message passing interface (MPI) named `mpi4py `_. Each of these -solutions has their advantages and disadvantages, in particular scaling beyond serial python functions, including thread -based parallelism, MPI parallel python application or assignment of GPUs to individual python function remains -challenging. +solutions has its advantages and disadvantages. However, one disadvantage common to all these libraries is the relative difficulty of scaling from serial functions to functions that make use of thread-based, MPI-based, or GPU-based parallelism. To address these challenges :code:`pympipool` is developed with three goals in mind: * Extend the standard python library `concurrent.futures.Executor `_ interface, to minimize the barrier of up-scaling an existing workflow to be used on HPC resources. -* Integrate thread based parallelism, MPI parallel python functions based on `mpi4py `_ and GPU assignment. This allows the users to accelerate their workflows one function at a time. +* Integrate thread-based parallelism, MPI-parallel python functions based on `mpi4py `_, and GPU assignment. This allows users to accelerate their workflows one function at a time. * Embrace `Jupyter `_ notebooks for the interactive development of HPC workflows, as they allow the users to document their though process right next to the python code and their results all within one document. HPC Context ----------- -In contrast to frameworks like `dask `_, `fireworks `_ -and `parsl `_ which can be used to submit a number of worker processes directly the the HPC +Frameworks like `dask `_, `fireworks `_ +and `parsl `_ can be used to submit a number of worker processes directly to the HPC queuing system and then transfer tasks from either the login node or an interactive allocation to these worker processes -to accelerate the execution, `mpi4py `_ and :code:`pympipool` follow a different -approach. Here the user creates their HPC allocation first and then `mpi4py `_ or +to accelerate the execution. By contrast, `mpi4py `_ and :code:`pympipool` follow a different +approach, in which the user creates their HPC allocation first and then `mpi4py `_ or :code:`pympipool` can be used to distribute the tasks within this allocation. The advantage of this approach is that no central data storage is required as the workers and the scheduling task can communicate directly. @@ -69,6 +67,7 @@ The same code can also be executed inside a jupyter notebook directly which enab The standard `concurrent.futures.Executor `_ interface is extended by adding the option :code:`cores_per_worker=2` to assign multiple MPI ranks to each function call. To create two workers :code:`max_workers=2` each with two cores each requires a total of four CPU cores to be available. + After submitting the function :code:`calc()` with the corresponding parameter to the executor :code:`exe.submit(calc, 0)` a python `concurrent.futures.Future `_ is returned. Consequently, the :code:`pympipool.Executor` can be used as a drop-in replacement for the diff --git a/docs/source/installation.md b/docs/source/installation.md index 83abc2f0..6a4c7680 100644 --- a/docs/source/installation.md +++ b/docs/source/installation.md @@ -38,10 +38,10 @@ pip install pympipool ## High Performance Computing `pympipool` currently provides interfaces to the [SLURM workload manager](https://www.schedmd.com) and the [flux framework](https://flux-framework.org). With the [flux framework](https://flux-framework.org) being the -recommended solution as it can be installed without root user rights and it can be integrated in existing resource +recommended solution as it can be installed without root permissions and it can be integrated in existing resource managers like the [SLURM workload manager](https://www.schedmd.com). The advantages of using `pympipool` in combination -with these resource schedulers is the fine-grained resource allocation. In addition, to scaling beyond a single compute -node they add the ability to assign GPUs and thread based parallelism. The two resource manager are internally linked to +with these resource schedulers is the fine-grained resource allocation. In addition to scaling beyond a single compute +node, they add the ability to assign GPUs and thread based parallelism. The two resource manager are internally linked to two interfaces: * `pympipool.slurm.PySlurmExecutor`: The interface for the [SLURM workload manager](https://www.schedmd.com).