Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

name the evaluation time for survival models #81

Closed
topepo opened this issue Nov 13, 2023 · 0 comments · Fixed by #84
Closed

name the evaluation time for survival models #81

topepo opened this issue Nov 13, 2023 · 0 comments · Fixed by #84
Assignees

Comments

@topepo
Copy link
Member

topepo commented Nov 13, 2023

We pick the first metric and, if it is a dynamic metric, the first eval time. Using verbose_iter = TRUE will say which metric was chosen but not the time.

library(tidymodels)
library(finetune)
library(censored)
#> Loading required package: survival

set.seed(1)
sim_dat <- prodlim::SimSurv(500) %>%
  mutate(event_time = Surv(time, event)) %>%
  select(event_time, X1, X2)

set.seed(2)
split <- initial_split(sim_dat)
sim_tr <- training(split)
sim_te <- testing(split)
sim_rs <- bootstraps(sim_tr, times = 4)

time_points <- c(10, 1, 5, 15)

mod_spec <-
  decision_tree(cost_complexity = tune()) %>%
  set_mode("censored regression")

grid <- tibble(cost_complexity = 10^c(-10, -2, -1))

gctrl <- control_grid(save_pred = TRUE)
rctrl <- control_race(save_pred = TRUE, verbose_elim = TRUE, verbose = FALSE)

dyn_mtrc  <- metric_set(brier_survival)

set.seed(2193)
aov_dyn_res <-
  mod_spec %>%
  tune_race_anova(
    event_time ~ X1 + X2,
    resamples = sim_rs,
    grid = grid,
    metrics = dyn_mtrc,
    eval_time = time_points,
    control = rctrl
  )
#> ℹ Racing will minimize the brier_survival metric.
#> ℹ Resamples are analyzed in a random order.
#> ℹ Bootstrap4: 0 eliminated; 3 candidates remain.

Created on 2023-11-13 with reprex v2.0.2

@EmilHvitfeldt EmilHvitfeldt self-assigned this Nov 13, 2023
topepo pushed a commit to tidymodels/extratests that referenced this issue Jan 11, 2024
topepo added a commit to tidymodels/extratests that referenced this issue Jan 12, 2024
* initial tests

* updates for tidymodels/tune#798

* SA test updates

* update tune version

* update bayesian opt tests

* update racing unit tests

* update tune version

* temporarily point to PR

* improve readability of reference object

* racing updates; also covers changes in tidymodels/finetune#90

* updated for changes in #804

* test for #138 and tidymodels/finetune#81

* PR tune#798 has been merged, new one to check with is tune#804

* updated for new tune version

* version bumps

* update with new tune warning wording

* changes for tidymodels/tune#806

* more updated snapshots

* mroe snapshot updates

* update censored bagging snapshot

* irreproducible results; see #160

---------

Co-authored-by: ‘topepo’ <‘[email protected]’>
Co-authored-by: Hannah Frick <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants