Meta-data extraction

[mgt16-LANL]

Hi - is there any way to get information such as the number of cores, number of threads, and number of gpus from the executor class? I'm imagining this would be useful for automated meta-data extraction/storage when you might have a few executors going at once.

[jan-janssen]

Hi @mgt16-LANL , there is no official storage of the meta-data, still you can access it via:

from pympipool import Executor
exe = Executor(max_workers=2, cores_per_worker=2)
exe._process._kwargs
>>> {'future_queue': <queue.Queue at 0x10c1f2f50>,
>>>  'max_workers': 2,
>>>  'executor_class': pympipool.mpi.executor.PyMPISingleTaskExecutor,
>>>  'hostname_localhost': False,
>>>  'cores': 2,
>>>  'oversubscribe': False,
>>>  'init_function': None,
>>>  'cwd': None}

[mgt16-LANL]

Thanks @jan-janssen , unfortunately this doesn't work on all other Executors:

from pympipool.flux import PyFluxExecutor
exe = PyFluxExecutor(max_workers=2)
exe._process._kwargs

Gives:

AttributeError: 'list' object has no attribute '_kwargs'

However, I found a workaround with just referencing the first index in the exe._process list:

exe._process.[0]_kwargs

This is similar on both the Slurm and Flux executors from my tests. Is this a reliable way to access the meta-data? or is there a chance the relevant meta-data for an executor might be in a another _process from the list?

[jan-janssen]

This is similar on both the Slurm and Flux executors from my tests. Is this a reliable way to access the meta-data? or is there a chance the relevant meta-data for an executor might be in a another _process from the list?

Sorry, I tested on a slightly older version. The meta data is the same on all processes in the list. Basically each process is connected to one worker but still they all get a copy of the whole meta data communicated. The only thing that is missing is the max_workers variable. This can be extracted based on the list of processes: len(exe._process)

Updated example:

from pympipool import Executor
exe = Executor(max_workers=2, cores_per_worker=2)
exe._process[0]._kwargs
>>> {'future_queue': <queue.Queue at 0x10ed8c0d0>,
>>>  'cores': 2,
>>>  'interface_class': pympipool.shared.interface.MpiExecInterface,
>>>  'hostname_localhost': False,
>>>  'init_function': None,
>>>  'cwd': None,
>>>  'oversubscribe': False}

[mgt16-LANL]

Thanks! That's super clear - and very helpful!