fix bug in TuneWrapper for param in used in predict function, issue 2472 #2479

berndbischl · 2018-11-09T12:09:42Z

fixes issue #2472

unit tests are missing. please help and merge

mb706 · 2018-11-09T15:14:32Z

R/TuneWrapper.R

-  predictLearner(lrn, .model$learner.model$next.model, .newdata, ...)
+  arglist = list(.learner = lrn, .model = .model$learner.model$next.model, .newdata = .newdata)
+  arglist = insert(arglist, list(...))
+  arglist = insert(arglist, .model$learner.model$opt.result$x)


Should probably only insert the parameters that have when %in% c("predict", "both").

Maybe one could just copy the lines from predict.R, if we are sure there are no other wrappers that introduce a discrepancy between the par.vals of a learner and the ... parameters on purpose.

jakob-r · 2018-11-12T13:41:57Z

Some notes regarding my observation:

setHyperPars within predictLearner is without effect because they will never be requested. The hyperparams got extracted in predict.(WrappedModel) and are put into ....
We don't know if the par.vals got changed after the training of the TuneWrapped Learner.
We will always overwrite the existing par.vals with the ones from the learner.
We assume that the user does not change par.vals and between training and predict. This holds especially as in most cases both is done internally in one step.

mb706 · 2018-11-12T13:53:45Z

setHyperPars within predictLearner is without effect because they will never be requested. The hyperparams got extracted in predict.(WrappedModel) and are put into ....

we should probably not rely on this; right now the parameter values are given both inside the ... and as part of the .learner object; its probably good to make sure they agree.

We don't know if the par.vals got changed after the training of the TuneWrapped Learner.

How would that happen (assuming the user does not directly modify the model$learner object, in which case, too bad)?

We will always overwrite the existing par.vals with the ones from the learner.

You mean the ones being tuned over? It might be worth considering to remove the parameters found in the tuning param set from the wrapped learner's param set, so the user gets an error message when he tries to set a param that will be overwritten later.

We assume that the user does not change par.vals and between training and predict. This holds especially as in most cases both is done internally in one step.

I'm confused again, how would that happen? Does any part of mlr support that?

jakob-r · 2018-11-12T14:13:27Z

setHyperPars within predictLearner is without effect because they will never be requested. The hyperparams got extracted in predict.(WrappedModel) and are put into ....

we should probably not rely on this; right now the parameter values are given both inside the ... and as part of the .learner object; its probably good to make sure they agree.

That is the architecture of mlr at the moment and we are relying on it everywhere. In train and predict the parameters are read and put into the arguments.
This is probably not the best design but keeping the arguments and the par.vals "in sync" would only be a lot of effort.

We don't know if the par.vals got changed after the training of the TuneWrapped Learner.

How would that happen (assuming the user does not directly modify the model$learner object, in which case, too bad)?

Like that, yes., ? mod = train(...); setHyperPars(mod$learner, ...); predict(mod, ...)

We will always overwrite the existing par.vals with the ones from the learner.

You mean the ones being tuned over? It might be worth considering to remove the parameters found in the tuning param set from the wrapped learner's param set, so the user gets an error message when he tries to set a param that will be overwritten later.

A warning somewhere might be useful. But I am afraid we would generate warnings for cases where we have set the par.vals in the definition of the learner. This can entail a lot of work.

We assume that the user does not change par.vals and between training and predict. This holds especially as in most cases both is done internally in one step.

I'm confused again, how would that happen? Does any part of mlr support that?
See above.

My suggestion: We should merge this PR asap and can add convenience later.

mb706 · 2018-11-12T14:40:43Z

That is the architecture of mlr at the moment and we are relying on it everywhere. In train and predict the parameters are read and put into the arguments.

Do you consider this to be a bug?

mod = train(...); setHyperPars(mod$learner, ...); predict(mod, ...)

has "model$learner <- [something]" ever been documented as supported behaviour? If the user modifies internals of objects then of course he can break things without bounds, I thought of mod$learner as such an internal.

berndbischl · 2018-11-12T14:41:18Z

i do not understand the unit test. can you please add at least some comments?
how the test is setup and why you expect certain things?

berndbischl · 2018-11-12T14:42:51Z

also please note: a PR for fixing a certain bug is probably not the best place to dicuss general architecture questions. please move this to a clean issue if you want to discuss this...

jakob-r · 2018-11-12T15:29:14Z

Do you consider this to be a bug?

I would not call it a bug as it's just affecting some error message. But the whole code is strange. How do we know that it's really the wrong family that leads to NAs? But this is another issue and should not be discussed here.

i do not understand the unit test.

Better now?

jakob-r · 2018-11-13T13:48:04Z

@berndbischl ping

mb706 · 2018-11-13T13:55:54Z

R/TuneWrapper.R

@@ -74,8 +74,16 @@ trainLearner.TuneWrapper = function(.learner, .task, .subset = NULL,  ...) {

 #' @export
 predictLearner.TuneWrapper = function(.learner, .model, .newdata, ...) {
-  lrn = setHyperPars(.learner$next.learner, par.vals = .model$learner.model$opt.result$x)


FYI I am relatively sure that removing this line (and not putting .learner = lrn further down etc) will break things.

you mean .learner$next.learner? That moved to another line. But I added setHyperPars now again.

2a0d9d0 is exactly what I meant, with .learner = lrn I meant the list entry ".learner" 👍

pat-s · 2019-04-11T19:18:21Z

@jakob-r If the build passes, can this be merged? I would add an entry to NEWS.md

…ce level of a CV (mlr-org#2479)

fix bug in TuneWrapper for param in used in predict function, issuu 2472

0967696

berndbischl mentioned this pull request Nov 9, 2018

TuneWrapper with clssif.glmnet does not seem to apply tuned hyperparameter in outer cross-validation #2472

Closed

mb706 reviewed Nov 9, 2018

View reviewed changes

Merge branch 'master' into fix_2472

c12b545

jakob-r self-assigned this Nov 12, 2018

fix appraoch, add test

e5d8106

add comments to test

875db8e

Merge branch 'master' into fix_2472

8ee08b5

mb706 reviewed Nov 13, 2018

View reviewed changes

jakob-r and others added 3 commits November 13, 2018 15:17

setHyperPars

2a0d9d0

setHyperPars

a42c084

Merge branch 'master' into fix_2472

68d6285

pat-s added prio-high type-bug labels Apr 11, 2019

pat-s added 3 commits April 14, 2019 23:53

Merge branch 'master' into fix_2472

ea1bf2b

Merge branch 'master' into fix_2472

37233c4

NEWS

1252fc0

pat-s approved these changes Apr 15, 2019

View reviewed changes

pat-s merged commit 7ea4a57 into master Apr 15, 2019

pat-s deleted the fix_2472 branch April 15, 2019 18:27

vrodriguezf pushed a commit to vrodriguezf/mlr that referenced this pull request Jan 16, 2021

Make sure that optimized hyperparameters are applied in the performan…

6408b80

…ce level of a CV (mlr-org#2479)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix bug in TuneWrapper for param in used in predict function, issue 2472 #2479

fix bug in TuneWrapper for param in used in predict function, issue 2472 #2479

berndbischl commented Nov 9, 2018 •

edited

Loading

mb706 Nov 9, 2018

jakob-r commented Nov 12, 2018

mb706 commented Nov 12, 2018

jakob-r commented Nov 12, 2018

mb706 commented Nov 12, 2018

berndbischl commented Nov 12, 2018

berndbischl commented Nov 12, 2018

jakob-r commented Nov 12, 2018

jakob-r commented Nov 13, 2018

mb706 Nov 13, 2018

jakob-r Nov 13, 2018

mb706 Nov 13, 2018

pat-s commented Apr 11, 2019

fix bug in TuneWrapper for param in used in predict function, issue 2472 #2479

fix bug in TuneWrapper for param in used in predict function, issue 2472 #2479

Conversation

berndbischl commented Nov 9, 2018 • edited Loading

mb706 Nov 9, 2018

Choose a reason for hiding this comment

jakob-r commented Nov 12, 2018

mb706 commented Nov 12, 2018

jakob-r commented Nov 12, 2018

mb706 commented Nov 12, 2018

berndbischl commented Nov 12, 2018

berndbischl commented Nov 12, 2018

jakob-r commented Nov 12, 2018

jakob-r commented Nov 13, 2018

mb706 Nov 13, 2018

Choose a reason for hiding this comment

jakob-r Nov 13, 2018

Choose a reason for hiding this comment

mb706 Nov 13, 2018

Choose a reason for hiding this comment

pat-s commented Apr 11, 2019

berndbischl commented Nov 9, 2018 •

edited

Loading