Some progress with future.clustermq

[michaelmayer2]

I have been playing around with future.clustermq lately and am getting into things. I now can use it to launch more than one worker on Slurm which is great but now I am stuck at https://github.com/michaelmayer2/future.clustermq/blob/master/R/ClusterMQFuture-class.R#L233

workers$receive_data() reports token: "not set" after which it runs $workers$send_common_data(). This eventually leads to success=NULL and the code stops.

I would be curious if there is any pointers on how to transfer the token to the workers. This at least would get me to a state where the workers are up and running to take some work.

Thanks in advance,

Michael.

[michaelmayer2]

In a debugging session I can see Slurm jobs running

[email protected]$ squeue 
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
            5512_1       all  cmq6336       mm  R       0:04      1 all-st-rstudio-1
            5512_2       all  cmq6336       mm  R       0:04      1 all-st-rstudio-1

Also the workers report everything correctly except the token

Browse[1]> workers$receive_data()
$id
[1] "WORKER_READY"

$auth
[1] "tlfzv"

$pkgver
[1] ‘0.8.95.3’

$token
[1] "not set"

[mschubert]

Token not set simply means there is no common_data transferred to the worker yet.

Your answer (for the worker API) to id="WORKER_READY", token="not set" should be w$send_common_data() (where w is your worker object; common data needs to be set first with w$set_common_data())

Does that help?

(Note that this will likely change with clustermq=0.9, the worker API is needlessly complicated)

[michaelmayer2]

Thanks for that input which was very helpful.

Not so much for the w$send_common_data() part (which is already in the future.clustermq code) but more for the hint about API changes. Seemingly the most recent version of clustermq is not compatible with the most recent version of future.clustermq. If I downgrade clustermq to 0.8.8 and add pkgs=character(0) to the data list() in L#222 of R/ClusterMQFuture-class.R, the futures are starting as expected without any further changes.

Scalability is still an issue but this is the same with PSOCK based cluster futures. I hope that there will be a way to have nested parallelization where each clustermq future will run an array task that internally would use zeromq for efficient communication so that the future overhead is not as significant any more. I have not figured out yet if this is even possible right now.

At the moment I have not figured out if there is a way to specify any resources like n_jobs, cores and mem while using the templates as in https://github.com/mschubert/clustermq/blob/master/inst/SLURM.tmpl

[kkmann]

Very interesting! Nested futures with clustermq backend would be a very interesting alternative to futures.batchtools for scaling targets piplelines.