Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

simulated annealing restarting at the wrong iteration #16

Open
topepo opened this issue Apr 25, 2021 · 0 comments
Open

simulated annealing restarting at the wrong iteration #16

topepo opened this issue Apr 25, 2021 · 0 comments
Labels
bug an unexpected problem or unintended behavior

Comments

@topepo
Copy link
Member

topepo commented Apr 25, 2021

For this example, the restart was set to 10 iterations but it restarts at iteration 8:

ctrl_sa <- control_sim_anneal(verbose = TRUE, no_improve = 10L)

set.seed(1234)

svm_sa <-
  svm_wflow %>%
  tune_sim_anneal(
    resamples = penguins_folds,
    metrics = roc_res,
    initial = svm_initial,
    param_info = svm_param,
    iter = 50,
    control = ctrl_sa
  )
## Optimizing roc_auc

## Initial best: 0.84948
##  1 ◯ accept suboptimal  roc_auc=0.57004	(+/-0.172)
##  2 ◯ accept suboptimal  roc_auc=0.57004	(+/-0.172)
##  3 ◯ accept suboptimal  roc_auc=0.56876	(+/-0.1715)
##  4 ◯ accept suboptimal  roc_auc=0.56876	(+/-0.1715)
##  5 ◯ accept suboptimal  roc_auc=0.56876	(+/-0.1715)
##  6 + better suboptimal  roc_auc=0.56942	(+/-0.1715)
##  7 ◯ accept suboptimal  roc_auc=0.56848	(+/-0.1718)
##  8 ✖ restart from best  roc_auc=0.56876	(+/-0.1715)
##  9 ◯ accept suboptimal  roc_auc=0.84948	(+/-0.01685)
## 10 ◯ accept suboptimal  roc_auc=0.84948	(+/-0.01685)

Also, for some other reason, the initial grid gives different ROC results each run. This might be due to the Platt scaling used by kernlab; its CV doesn't use a controllable seed. However, we don't see this in the book example.

@topepo topepo added the bug an unexpected problem or unintended behavior label Apr 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug an unexpected problem or unintended behavior
Projects
None yet
Development

No branches or pull requests

1 participant