Bug in predictions due to recent change in data.table

[danielhorn]

data.table changed its behaviour wrt. rbind by name/position. Recent version issue a message about this, e.g. if you run this code:

set.seed(123)
lrn = makeLearner("classif.svm", predict.type = "prob")
task = iris.task
ctrl = makeTuneControlRandom(maxit = 2)
ps = makeParamSet(makeNumericParam("cost", lower = 0.01, upper = 0.1))
lrn = makeTuneWrapper(lrn, cv3, par.set = ps, control = ctrl)
resample(lrn, task, cv3)

This leads to errors down the analysis, where probabilites are not joint corretly together in prediction objects. I.e., probabiltites for the first class where rbinded to the column for the 2nd class, and vice versa. See

mlr/R/ResamplePrediction.R

Line 26 in 3b52b12

rbindlist(lapply(seq_along(pr.te), function(X) cbind(pr.te[[X]]$data, iter = X, set = "test"))),

.

To turn the message into an error, you can set the option options(datatable.rbindlist.check="error").

[mllg]

Here is a snippet which demonstrates that the probabilities are incorrectly aggregated:

set.seed(123)
lrn = makeLearner("classif.svm", predict.type = "prob")
task = sonar.task

rr = resample(lrn, task, cv3)

pred = rr$pred
calculateConfusionMatrix(pred)
pred = setThreshold(pred, 0.5)
calculateConfusionMatrix(pred)