package dependency problem with mlr and mbo using devtools: load-_all()

[smilesun]

my code is

detachAllPackages()
rm(list = ls(all = TRUE))
load_all("~/mlr4Curve/")
load_all("~/mlrMBO/")
#some code for tuning using mbo

But if I run this twice,

 Error in .C("maximinLHS_C", as.integer(n), as.integer(k), as.integer(dup),  : 
  "maximinLHS_C" not resolved from current namespace (lhs) In addition: Warning message:
character(0) 
11.
(function (n, k, dup = 1) 
{
    if (length(n) != 1 | length(k) != 1 | length(dup) != 1) 
        stop("n, k, and dup may not be vectors") ... 
10.
do.call(fun, insert(list(n = nmissing, k = k), fun.args)) 
9.
generateDesign(n.params * 4L, par.set, fun = lhs::maximinLHS) at mbo.R#49
8.
mbofun(tff, design = control$mbo.design, learner = control$learner, 
    control = mbo.control, show.info = FALSE) at tuneMBO.R#25
7.
sel.func(learner, task, resampling, measures, par.set, control, 
    opt.path, show.info) at tuneParams.R#111

[ja-thomas]

I can not reproduce the problem.
Can you try to post a complete and reproducible example?

If the problem is already solved you can close this issue

[jakob-r]

This example obviously can't work. I assume you use (the function mentioned here)[http://stackoverflow.com/questions/7505547/detach-all-packages-while-working-in-r] for detachAllPackages() and that mlr4Curve is some mlr fork. But then load_all should not work, because it needs devtools, which you just unloaded. Apart from that I might have seen this issues related to using load_all twice. But with the newest versions of mlr and mlrMBO I can not reproduce.
Reopen if you have re reproducible example with an included sessionInfo().

[berndbischl]

what xudong reported is a valid problem.

but thats a "bug" in lhs, you should report it there.

here is the solution for exactly the same problem, read it and know about it
robjhyndman/forecast#378

[mb706]

For those in the future having similar problems:

This seems to be a problem that occurs when R unloads shared libraries ("DLLs"). R has a lookup table of DLLs (LoadedDLL in the R source), and, for each package, keeps a pointer to this lookup table (which can be queried in R as getCallingDLLe(getNamespace(packagename))$info; R source does this here). When R unloads a DLL (devtools does this by calling dyn.unload), it shifts all DLL lookup table entries that come after the removed DLL backwards by one entry (to keep the lookup table compact), but it does not update the cached pointer to this table (I.e. the getCallingDLLe() result stays the same). Therefore, native (.C) calls by some unlucky packages fail after this happens: R looks into the wrong DLL and finds no maximinLHS_C.

I suspect either smilesun's call of detachAllPackages, or one of the subsequent load_all calls unloads (or re-loads) some DLLs.

Note that, even though the lhs people are in a good position to implement a workaround (the PACKAGE argument mentioned above), this is actually a bug in R (or in devtools, depending on your view). In fact, giving the PACKAGE argument to the .C call skips the look up table cache (which might be there for performance reasons).

A minimal example I constructed: (Start R with the --vanilla option for this, since an .RData file might interfere)

> devtools::load_all("ParamHelpers")
Loading ParamHelpers
> lhs::maximinLHS(n=2, k=2)  # implicitly loads lhs dll
          [,1]      [,2]
[1,] 0.6008410 0.9491948
[2,] 0.4723376 0.3303989
> devtools::load_all("ParamHelpers")
Loading ParamHelpers
> lhs::maximinLHS(n=2, k=2)
Error in .C("maximinLHS_C", as.integer(n), as.integer(k), as.integer(dup),  : 
  "maximinLHS_C" not resolved from current namespace (lhs)

Sometimes DLLs get loaded at surprising points, and then a simple devtools::load_all can break things. This happens for example when the R save file contains objects that reference namespaces:

(Example includes shell code) (Also note that for this I have "ParamHelpers" installed as regular package, additionally to the "ParamHelpers" git clone)

$ mkdir newdir && cd newdir
$ R
> references.lhs.namespace = lhs::maximinLHS  # options(error=dump.frames) might do this
> references.ph.namespace = ParamHelpers::makeParamSet
> q("yes")  # end this R session
$ R  # start R again
> # R loads both namespaces and corresponding libraries
> .dynLibs()  # get an (imperfect) view on the lookup table
                                              Filename Dynamic.Lookup
1         /usr/lib64/R/library/methods/libs/methods.so          FALSE
2             /usr/lib64/R/library/utils/libs/utils.so          FALSE
3       /home/user/library/checkmate/libs/checkmate.so           TRUE
4     /usr/lib64/R/library/grDevices/libs/grDevices.so          FALSE
5       /usr/lib64/R/library/graphics/libs/graphics.so          FALSE
6             /usr/lib64/R/library/stats/libs/stats.so          FALSE
7             /home/user/library/BBmisc/libs/BBmisc.so           TRUE
8 /home/user/library/ParamHelpers/libs/ParamHelpers.so           TRUE
9                   /home/user/library/lhs/libs/lhs.so           TRUE
> lhs::maximinLHS(n=2, k=2)  # works
          [,1]      [,2]
[1,] 0.6008410 0.9491948
[2,] 0.4723376 0.3303989
> devtools::load_all("../ParamHelpers")
Loading ParamHelpers
> .dynLibs()  # note the lhs.so entry has shifted by one
                                           Filename Dynamic.Lookup
1      /usr/lib64/R/library/methods/libs/methods.so          FALSE
2          /usr/lib64/R/library/utils/libs/utils.so          FALSE
3    /home/user/library/checkmate/libs/checkmate.so           TRUE
4  /usr/lib64/R/library/grDevices/libs/grDevices.so          FALSE
5   / usr/lib64/R/library/graphics/libs/graphics.so          FALSE
6          /usr/lib64/R/library/stats/libs/stats.so          FALSE
7          /home/user/library/BBmisc/libs/BBmisc.so           TRUE
8                /home/user/library/lhs/libs/lhs.so           TRUE
9          /home/user/library/digest/libs/digest.so           TRUE
10     /home/user/library/devtools/libs/devtools.so           TRUE
11         /usr/lib64/R/library/tools/libs/tools.so          FALSE
12       /home/user/library/stringi/libs/stringi.so           TRUE
13             /home/user/library/Rcpp/libs/Rcpp.so           TRUE
14     /home/user/library/roxygen2/libs/roxygen2.so           TRUE
15      /home/user/ParamHelpers/src/ParamHelpers.so           TRUE
16     /home/user/library/testthat/libs/testthat.so           TRUE
> lhs::maximinLHS(n=2, k=2)
Error in .C("maximinLHS_C", as.integer(n), as.integer(k), as.integer(dup),  : 
  "maximinLHS_C" not resolved from current namespace (lhs)

(This was run on a x86_64 linux system)