Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Support cpu and mem specification via env (#1207)
* Support cpu and mem specification via env This commit adds support for respecting specification of how much CPU and memory should be used by workers via the PARSL_CORES and PARSL_GB_MEMORY environmental variables (as opposed to inferring it to be the full node, once a job has started on the node.) After this commit, the config below will result in 4 workers being started, using a total of 4 cores and 12 GB memory. This is a pre-requisite to a full solution to #942. ``` from parsl.config import Config from parsl.providers import SlurmProvider from parsl.addresses import address_by_hostname from parsl.executors import HighThroughputExecutor config = Config( executors=[ HighThroughputExecutor( cores_per_worker=1, mem_per_worker=3, address=address_by_hostname(), provider=SlurmProvider( 'broadwl', cmd_timeout=60, nodes_per_block=1, init_blocks=1, min_blocks=1, max_blocks=1, scheduler_options='#SBATCH --cpus-per-task=4 --mem-per-cpu=3g, worker_init='worker_init='export PARSL_MEMORY_GB=$(expr $SLURM_CPUS_ON_NODE \* $SLURM_MEM_PER_CPU); export PARSL_CORES=$SLURM_CPUS_ON_NODE', exclusive=False ), ) ], ) ``` * Change PARSL_GB_MEMORY -> PARSL_MEMORY_GB * Revert changes to launchers This reversts the launcher changes introduced in 18061d2.
- Loading branch information