h20 learner : "Error in checkPredictLearnerOutput(.learner, .model, p)"

[caprone]

HI
os: win7
Rstudio: 1.0.143
R : 3.4
issue:
all h2o learner fails in prediction task: seems this is class levels names conflit:

"Error in checkPredictLearnerOutput(.learner, .model, p) :
predictLearner for classif.h2o.gbm has returned not the class levels as column names: p0, p1"

checking current names in learner , those lacks 'p' prefix; in fact they are '0', '1' and not 'p0, p1' as needed

[jakob-r]

Can you supply a minimum (not) working example?

[caprone]

HI, ok, thanks
`

task --> class levels are '0', '1'

classif.task = makeClassifTask(id = "example",
data = train_scaled,
target = "is_duplicate",
positive = "1")

learner

gbm.lrn = makeLearner("classif.h2o.gbm",
predict.type = "prob",
fix.factors.prediction = TRUE)

cv

rdesc = makeResampleDesc("CV", iters = 3, stratify = TRUE)

r = resample(gbm.lrn, classif.task, rdesc,
measures = list(auc, logloss)
)
`
this works well with all learners but with h2o

[larskotthoff]

You haven't shown us your data.

[caprone]

ok, sorry, this is a perfectly reproducible example:

library(mlr)

df <- data.frame(matrix(runif(100, 0, 1), 100, 9))
classx <- factor(sample(c(0, 1), 100, replace = TRUE))
df <- cbind(classx, df)

classif.task = makeClassifTask(id = "example", 
                               data = df,
                               target = "classx", 
                               positive = "1")

gb.lrn  = makeLearner("classif.h2o.gbm", 
                      predict.type = "prob", 
                      fix.factors.prediction = TRUE)

rdesc = makeResampleDesc("CV", iters = 3, stratify = TRUE)
rin = makeResampleInstance(rdesc, task = classif.task)

r = resample(gb.lrn, classif.task, rin, 
             measures = list(auc, logloss)
)

# error:
Error in checkPredictLearnerOutput(.learner, .model, p) : 
  predictLearner for classif.h2o.gbm has returned not the class levels as column names: p0, p1

thanks!!

[larskotthoff]

Thanks, this is indeed a bug (caused by completely broken code). I've fixed this in #1790.

[andrewcparnell]

Hi,

A similar possible bug I've just found still exists when spaces occur in the class names.

Here's an example based on the above that still breaks for me using mlr v2.13:

library(mlr)

set.seed(123)
df <- data.frame(matrix(runif(100, 0, 1), 100, 9))
classx <- sample(paste(letters[1:4],letters[1:4]), 100, replace = TRUE)
df <- cbind(classx, df)

classif.task = makeClassifTask(id = "example", 
                               data = df,
                               target = "classx")

gb.lrn  = makeLearner("classif.h2o.randomForest", 
                      predict.type = "prob")

rdesc = makeResampleDesc("CV", iters = 3, stratify = TRUE)
rin = makeResampleInstance(rdesc, task = classif.task)

r = resample(gb.lrn, classif.task, rin, 
             measures = list(mmce))

I get:

Error in checkPredictLearnerOutput(.learner, .model, p) : 
  predictLearner for classif.h2o.randomForest has returned not the class levels as column names: a.a,b.b,c.c,d.d

This will run when replacing e.g. classif.h2o.randomForest with classif.randomForestSRC. It seems to fail with other non-standard characters in the class names but I haven't done an exhaustive search.

Many thanks for the wonderful package.

Andrew

[larskotthoff]

Ah, looks like the names are sometimes sanitized and sometimes not. The workaround for now is to sanitize everything yourself, e.g. using the make.names() function. Could you please open a new issue for this?