Debugging SSVM: Error : Tables have different number of rows (x: 54, y: 108)
#9
Comments
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as outdated.
This comment was marked as outdated.
jemus42
changed the title
Error : Tables have different number of rows (x: 54, y: 108)
|
I can reproduce using CRAN/release versions but the example works fine with all Github versions. Don't know which package update fixed it (and whether it is really fixed), though. |
|
For some reason I can no longer reproduce the error on any of the two machines I have worked on this and that.. scares me? I did however stumble upon this, based on the reprex above: > extract_inner_tuning_archives(bmr)
Error: Tables have different number of rows (x: 4, y: 8)Which suggests the issue is in After some more debugging... the issue is in the Because the When I'm in the bowels of the relevant function via Here's a slightly updated reprex: library("mlr3")
library("mlr3proba")
library("mlr3pipelines")
library("mlr3tuning")
#> Loading required package: paradox
library("batchtools", warn.conflicts = FALSE)
library("mlr3batchmark")
requireNamespace("mlr3extralearners")
#> Loading required namespace: mlr3extralearners
# Create Registry ---------------------------------------------------------
reg = makeExperimentRegistry(NA, work.dir = here::here(), seed = 1, packages = c("mlr3", "mlr3proba"))
#> Sourcing configuration file '~/.config/batchtools/config.R' ...
#> Created registry in '/tmp/RtmpHeOycF/registry3f57211c6ed87c' using cluster functions 'SSH'
set.seed(123)
tasks = list(tsk("veteran"))
resamplings = list(rsmp("holdout")$instantiate(tasks[[1]]))
lrn_base = lrn("surv.svm", type = "hybrid", gamma.mu = c(.1, .1), diff.meth = "makediff3")
lrn_graph = po("fixfactors") %>>%
po("imputesample", affect_columns = selector_type("factor")) %>>%
po("scale") %>>%
po("encode", method = "treatment") %>>%
po("removeconstants") %>>%
lrn_base |>
as_learner()
# To catch the ever present "constraints are inconsistent, no solution!" error
lrn_graph$fallback = lrn("surv.kaplan")
lrn_auto = auto_tuner(
learner = lrn_graph,
search_space = ps(
surv.svm.kernel = p_fct(c("lin_kernel", "rbf_kernel", "add_kernel")),
surv.svm.gamma = p_dbl(-10, 10, trafo = function(x) 10^x),
surv.svm.mu = p_dbl(-10, 10, trafo = function(x) 10^x),
surv.svm.kernel.pars = p_dbl(-5, 5, trafo = function(x) 2^x),
.extra_trafo = function(x, param_set) {
# learner has tuple param gamma.mu = c(x, y)
# we tune separately and reassemble via trafo
x$surv.svm.gamma.mu = c(x$surv.svm.gamma, x$surv.svm.mu)
x$surv.svm.gamma = x$surv.svm.mu = NULL
x
}
),
resampling = rsmp("holdout"),
measure = msr("surv.cindex", id = "harrell_c"),
terminator = trm("evals", n_evals = 4),
tuner = tnr("random_search"),
store_tuning_instance = TRUE,
store_benchmark_result = TRUE,
store_models = TRUE
)
learners = list(
KM = lrn("surv.kaplan"),
SSVMtune = lrn_auto,
SSVMbase = lrn_base
)
mlr3misc::imap(learners, function(l, id) l$id = id)
#> $KM
#> [1] "KM"
#>
#> $SSVMtune
#> [1] "SSVMtune"
#>
#> $SSVMbase
#> [1] "SSVMbase"
grid = benchmark_grid(tasks = tasks, learners = learners, resamplings = resamplings)
ids = batchmark(
design = grid,
store_models = TRUE
)
#> Adding algorithm 'run_learner'
#> Adding problem '82bbee777d929c08'
#> Exporting new objects: 'e3d59f5931e77b8c' ...
#> Exporting new objects: 'feddf499a45e24b5' ...
#> Exporting new objects: '85be83d871b2282b' ...
#> Exporting new objects: '8184694659885028' ...
#> Exporting new objects: 'ecf8ee265ec56766' ...
#> Overwriting previously exported object: 'ecf8ee265ec56766'
#> Overwriting previously exported object: 'ecf8ee265ec56766'
#> Adding 3 experiments ('82bbee777d929c08'[1] x 'run_learner'[3] x repls[1]) ...
# Just running them all
submitJobs()
#> Submitting 3 jobs in 3 chunks using cluster functions 'SSH' ...
waitForJobs()
#> [1] TRUE
# No error?
getStatus()
#> Status for 3 jobs at 2024-09-17 17:28:04:
#> Submitted : 3 (100.0%)
#> -- Queued : 0 ( 0.0%)
#> -- Started : 3 (100.0%)
#> ---- Running : 0 ( 0.0%)
#> ---- Done : 3 (100.0%)
#> ---- Error : 0 ( 0.0%)
#> ---- Expired : 0 ( 0.0%)
bmr = reduceResultsBatchmark()
#bmr$score()
#res <- loadResult(2)
#res$learner_state$log
# This now reproduces the error at least
extract_inner_tuning_archives(bmr)
#> Error: Tables have different number of rows (x: 4, y: 8)Created on 2024-09-17 with reprex v2.1.1 Session infosessionInfo()
#> R version 4.4.1 (2024-06-14)
#> Platform: x86_64-pc-linux-gnu
#> Running under: Ubuntu 22.04.4 LTS
#>
#> Matrix products: default
#> BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
#> LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so; LAPACK version 3.10.0
#>
#> locale:
#> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
#> [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
#> [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
#> [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
#> [9] LC_ADDRESS=C LC_TELEPHONE=C
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
#>
#> time zone: Europe/Berlin
#> tzcode source: system (glibc)
#>
#> attached base packages:
#> [1] stats graphics grDevices utils datasets methods base
#>
#> other attached packages:
#> [1] mlr3batchmark_0.1.1.9000 batchtools_0.9.17 mlr3tuning_1.0.1
#> [4] paradox_1.0.1 mlr3pipelines_0.6.0 mlr3proba_0.6.7
#> [7] mlr3_0.20.2
#>
#> loaded via a namespace (and not attached):
#> [1] gtable_0.3.4 xfun_0.41
#> [3] ggplot2_3.5.1 lattice_0.22-5
#> [5] vctrs_0.6.4 tools_4.4.1
#> [7] generics_0.1.3 base64url_1.4
#> [9] parallel_4.4.1 tibble_3.2.1
#> [11] fansi_1.0.5 pkgconfig_2.0.3
#> [13] Matrix_1.6-3 data.table_1.16.0
#> [15] checkmate_2.3.0 uuid_1.1-1
#> [17] lifecycle_1.0.4 compiler_4.4.1
#> [19] set6_0.2.6 progress_1.2.2
#> [21] munsell_0.5.0 codetools_0.2-19
#> [23] htmltools_0.5.7 bbotk_1.1.0
#> [25] yaml_2.3.7 pillar_1.9.0
#> [27] crayon_1.5.2 mlr3viz_0.6.1
#> [29] parallelly_1.36.0 brew_1.0-8
#> [31] tidyselect_1.2.0 digest_0.6.33
#> [33] stringi_1.8.1 future_1.33.0
#> [35] dplyr_1.1.3 listenv_0.9.0
#> [37] splines_4.4.1 rprojroot_2.0.4
#> [39] fastmap_1.1.1 grid_4.4.1
#> [41] here_1.0.1 colorspace_2.1-0
#> [43] cli_3.6.1 magrittr_2.0.3
#> [45] survival_3.7-0 utf8_1.2.4
#> [47] withr_2.5.2 prettyunits_1.2.0
#> [49] scales_1.3.0 backports_1.4.1
#> [51] rappdirs_0.3.3 ooplah_0.2.0
#> [53] rmarkdown_2.25 globals_0.16.2
#> [55] distr6_1.8.4 hms_1.1.3
#> [57] evaluate_0.23 knitr_1.45
#> [59] mlr3extralearners_0.9.0-9000 dictionar6_0.1.3
#> [61] mlr3misc_0.15.1 rlang_1.1.2
#> [63] Rcpp_1.0.11 glue_1.6.2
#> [65] param6_0.2.4 palmerpenguins_0.1.1
#> [67] reprex_2.1.1 rstudioapi_0.15.0
#> [69] lgr_0.4.4 R6_2.5.1
#> [71] fs_1.6.3@mnwright can you still not reproduce this with whatever github versions of packages? |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
|
hi, i think this example can be boiled down A ALOT. but I can take a look at it. this also should really be an issue at mlr3 somewhere... |
|
So we basically think this is because the gamma.mu is assembled as a vector with 2 value in the extra trafo? and then the unnest fails? |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
|
Updated reprex to use custom debug learner for a proba-free experience: library(mlr3)
library(paradox)
library(mlr3misc)
library(mlr3tuning)
LearnerRegrDebugMulti = R6::R6Class("LearnerRegrDebugMulti", inherit = LearnerRegr,
public = list(
#' @description
#' Creates a new instance of this [R6][R6::R6Class] class.
initialize = function() {
super$initialize(
id = "regr.debugmulti",
feature_types = c("logical", "integer", "numeric", "character", "factor", "ordered"),
predict_types = c("response"),
param_set = ps(
x = p_dbl(0, 1, tags = "train"),
# same as in surv.svm
gamma.mu = p_uty(tags = c("train", "required"))
),
properties = "missings",
man = "mlr3::mlr_learners_regr.debugmulti",
label = "Debug Learner for Regression"
)
}
),
private = list(
.train = function(task) {
pv = self$param_set$get_values(tags = "train")
truth = task$truth()
model = list(
response = mean(truth),
se = sd(truth),
pid = Sys.getpid()
)
set_class(model, "regr.debug_model")
},
.predict = function(task) {
n = task$nrow
pv = self$param_set$get_values(tags = "predict")
predict_types = "response"
prediction = named_list(mlr_reflections$learner_predict_types[["regr"]][[predict_types]])
for (pt in names(prediction)) {
value = rep.int(self$model[[pt]], n)
prediction[[pt]] = value
}
return(prediction)
}
)
)
mlr_learners$add("regr.debugmulti", function() LearnerRegrDebugMulti$new())
lrn_base = lrn("regr.debugmulti", gamma.mu = c(0, 0))
instance = ti(
task = tsk("mtcars"),
learner = lrn_base,
search_space = ps(
gamma = p_dbl(0, 1),
mu = p_dbl(0, 1),
.extra_trafo = function(x, param_set) {
# learner has tuple param gamma.mu = c(x, y)
# we tune separately and reassemble via trafo
x$gamma.mu = c(x$gamma, x$mu)
x$gamma = x$mu = NULL
x
}
),
resampling = rsmp("holdout"),
terminator = trm("evals", n_evals = 3)
)
archive = tnr("grid_search")$optimize(instance)
#> INFO [13:23:15.137] [bbotk] Starting to optimize 2 parameter(s) with '<OptimizerBatchGridSearch>' and '<TerminatorEvals> [n_evals=3, k=0]'
#> INFO [13:23:15.160] [bbotk] Evaluating 1 configuration(s)
#> INFO [13:23:15.168] [mlr3] Running benchmark with 1 resampling iterations
#> INFO [13:23:15.205] [mlr3] Applying learner 'regr.debugmulti' on task 'mtcars' (iter 1/1)
#> INFO [13:23:15.230] [mlr3] Finished benchmark
#> INFO [13:23:15.244] [bbotk] Result of batch 1:
#> INFO [13:23:15.246] [bbotk] gamma mu regr.mse warnings errors runtime_learners
#> INFO [13:23:15.246] [bbotk] 1 0.4444444 32.78585 0 0 0.001
#> INFO [13:23:15.246] [bbotk] uhash
#> INFO [13:23:15.246] [bbotk] 1b48f81d-463f-4148-b141-0a4d63ff3e4c
#> INFO [13:23:15.250] [bbotk] Evaluating 1 configuration(s)
#> INFO [13:23:15.256] [mlr3] Running benchmark with 1 resampling iterations
#> INFO [13:23:15.259] [mlr3] Applying learner 'regr.debugmulti' on task 'mtcars' (iter 1/1)
#> INFO [13:23:15.263] [mlr3] Finished benchmark
#> INFO [13:23:15.281] [bbotk] Result of batch 2:
#> INFO [13:23:15.282] [bbotk] gamma mu regr.mse warnings errors runtime_learners
#> INFO [13:23:15.282] [bbotk] 0.4444444 0.4444444 32.78585 0 0 0
#> INFO [13:23:15.282] [bbotk] uhash
#> INFO [13:23:15.282] [bbotk] 8f5b5b32-e7b0-4d19-a586-ddb221465016
#> INFO [13:23:15.283] [bbotk] Evaluating 1 configuration(s)
#> INFO [13:23:15.287] [mlr3] Running benchmark with 1 resampling iterations
#> INFO [13:23:15.290] [mlr3] Applying learner 'regr.debugmulti' on task 'mtcars' (iter 1/1)
#> INFO [13:23:15.334] [mlr3] Finished benchmark
#> INFO [13:23:15.346] [bbotk] Result of batch 3:
#> INFO [13:23:15.347] [bbotk] gamma mu regr.mse warnings errors runtime_learners
#> INFO [13:23:15.347] [bbotk] 0.8888889 0 32.78585 0 0 0.002
#> INFO [13:23:15.347] [bbotk] uhash
#> INFO [13:23:15.347] [bbotk] ed4494bf-f86b-4e0f-a452-2a348c7e1c64
#> INFO [13:23:15.355] [bbotk] Finished optimizing after 3 evaluation(s)
#> INFO [13:23:15.356] [bbotk] Result:
#> INFO [13:23:15.357] [bbotk] gamma mu learner_param_vals x_domain regr.mse
#> INFO [13:23:15.357] [bbotk] <num> <num> <list> <list> <num>
#> INFO [13:23:15.357] [bbotk] 1 0.4444444 <list[1]> <list[1]> 32.78585
as.data.table(instance$archive)
#> Error: Tables have different number of rows (x: 3, y: 6)
# Because of this step
mlr3misc::unnest(archive, "x_domain")
#> Error: Tables have different number of rows (x: 1, y: 2)Created on 2024-09-19 with reprex v2.1.1 |
|
Opened an issue on mlr3misc mlr-org/mlr3misc#119 |
|
This is fixed for the case of the benchmark thanks to John's PR in mlr3extralearners mlr-org/mlr3extralearners#385 The underlying |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I keep encoutnering the following error wen running SSVM jobs in the batchtools/batchmark setting, specifically after the tuning process appears to have completed without major issues:
I have so far failed to reproduce this outside of batchtools, even trying to use the same task, resampling folds, and random seed.
Here's a reprex for the general setup, including the preprocessing timeline, where tuning and scoring afterwards work fine:
Created on 2024-09-11 with reprex v2.1.1
The text was updated successfully, but these errors were encountered: