DEoptim - Future Package

[johncancian]

Hello, my group uses DEoptim with a fitness function that is computationally intense and requires a significant amount of RAM. DEoptim cannot be run in parallel because the server it executes on runs out of RAM. We have been running it sequentially but it unfortunately takes quite a long time to find a solution. DEOptim is part of a larger calculation which we are transitioning to using future and doFuture packages with a Kubernetes cluster. Can DEoptim using parallelType=2 use doFuture to parallelize the population across multiple nodes?

[joshuaulrich]

Hi @johncancian, DEoptim uses foreach to run in parallel. So you can use any foreach backend.

[johncancian]

Hi Joshua,
Thank you for your assistance. Have you had any experience with running DEoptim using parallelType=2 where the fitness function is also executing parallel tasks? This would be a nested parallelization case.

[joshuaulrich]

Nested parallelism on the same machine usually makes performance worse because you have multiple processes competing for the same resources. I suggest you only use parallel tasks for one or the other. An exception would be if the DEoptim backend is sending jobs to a cluster of computers.

[johncancian]

Our intention is to setup a Kubernetes cluster that follows the work of Chris Paciorek (https://www.jottr.org/2021/04/08/future-and-kubernetes/). We are current running DEoptim with parallelType = 0 and using foreach parallelization within the fitness function. Being able to use DEoptim with parallelType = 2 and foreach parallelization in our fitness function, we believe could significantly increase our process performance.

[joshuaulrich]

That should work. I don't have any experience doing something like that though.

[braverock]

foreach loops can be nested, as long as you have different engines, as you've suggested here. So the parent foreach will run single threaded, but your fitness function will use parallelType=2. This should be fine.

[johncancian]

Hello Joshua and Brian, We have continued working on our process of migrating to a horizontal parallelization infrastructure using Kubernetes. We are running DEoptim sequentially using a fitness function that contains a foreach that has a %dopar%. Our parallel backend parallelizes the processing in the fitness function.

Would it be possible to have a nested construct in DEoptim? I tried setting up a parallelType = 3 in the attached code and replaced foreach::"%dopar%" with foreach::"%:%". It's not working: "Error in apply(i, 1, fn, ...) : object 'i' not found". Would you have any advice in this area?

  else if(ctrl$parallelType == 3) { ## use nested foreach 
    if(!foreach::getDoParRegistered()) {
      foreach::registerDoSEQ()
    }
    args <-  ctrl$foreachArgs
    fnPop <- function(params, ...) {
      my_chunksize <- ceiling(NROW(params)/foreach::getDoParWorkers())
      my_iter <- iterators::iter(params,by="row",chunksize=my_chunksize)
      args$i <- my_iter
      args$.combine <- c
      if (!is.null(args$.export))
        args$.export = c(args$.export, "fn")
      else
        args$.export = "fn"
      if (is.null(args$.errorhandling))
        args$.errorhandling = c('stop', 'remove', 'pass')
      if (is.null(args$.verbose))
        args$.verbose = FALSE
      if (is.null(args$.inorder))
        args$.inorder = TRUE
      if (is.null(args$.multicombine))
        args$.multicombine = FALSE
      foreach::"%:%"(do.call(foreach::foreach, args), apply(i,1,fn,...))
      
    }
  }

DEoptim.zip