Survival measures

[mllg]

No description provided.

[mllg]

Now each survival model has the the training survival times attached (required for cindex.uno and more measures I'll implement).

The time-dependent measures sometimes return NaN/NA and I think this is related to the max.time stuff you were discussing. If we cannot set a good default here, we probably should force the user to set one himself (i.e., initialize with NA). Thoughts?

[mllg]

FWIW, here is a nice overview: https://rstudio-pubs-static.s3.amazonaws.com/3506_36a9509e9d544386bd3e69de30bca608.html

[PhilippPro]

Ok, there is also my wished integrated AUC.

[mllg]

Added iauc.uno. AUC.sh does not work for me, I get segfaults :/

[studerus]

Good idea to make a constructor for time dependent AUC measures!

[mllg]

I've cleaned it up a little. I'm still unsure whether I like the constructor or not...

[berndbischl]

I've cleaned it up a little. I'm still unsure whether I like the constructor or not...

what do you dislike? (i suggested that to Phillip)
that seems to be the best option to make the measure, an object, parameter dependent

[PhilippPro]

"Philipp" please. And if you look here you could also look in the PR for getOOB. ;)

[berndbischl]

we need a spitzname for you

[mllg]

what do you dislike? (i suggested that to Phillip)

I don't see an advantage over a simple and generic setMeasurePars which then can be reused for future measures which come with parameters. Furthermore, measures which come with a special constructor are not documented in ?measures, right?

[berndbischl]

I don't see an advantage over a simple and generic setMeasurePars which then can be reused for future measures which come with parameters. Furthermore, measures which come with a special constructor are not documented in ?measures, right?

ok thats a good point. i do have the feeling that we are invading this PR a bit though. new issue?
also we have one other old measure that this relates to, makeCostMeasure.

[PhilippPro]

What's the status here? Until we have setMeasurePars, we use the constructor?

[PhilippPro]

@studerus: We decided to delete three of the measures for time dependent auc, because they are numerically unstable. They provided results smaller than 0 and many times where NA, so they did not pass our tests. They are now in the todo-files folder, where they can potentially be fixed.

Do you maybe know which of these four time dependent auc measures is the best one?

[PhilippPro]

By the way, we are thinking of writing a small paper about it (maybe after integrating also the ibs, would you like to join?

[mllg]

Might be ready for merge. We will probably add more measures after this PR (we moved some to todo-files). Still no working solution for brier score.

[studerus]

@PhilippPro
It's unfortunate that you had to take out these measures. According to this simulation study the kernel weighting method as implemented in the tdROC package would be the best time dependent ROC measure. What exactly was the problem with the measures?

Writing a paper is good idea and I would be happy to join.

[studerus]

@mllg
What is the problem with the brier score? I thought we had solved the problem after our email discussion with Gerds.

[PhilippPro]

The problem was that in our tests the measures provided errors many times. See my message above:

They provided results smaller than 0 and many times where NA, so they did not pass our tests.

One problem might be, that you cannot set time higher than the highest time point, but in resampling this can happen quite easily.

[PhilippPro]

With the brier score we were unhappy that in the implementations this special kind of object is needed. We also wanted to do that in a separate new commit and either use pec or (my preferential but labourious option) code it by ourselves.

[studerus]

@PhilippPro

Did you also inform the developers of tdROC package about this problem?

[PhilippPro]

No. They probably also did not plan to use it for resampling. ;) But we could maybe write them.

[lintr-bot]

R/measures.R:1405:35: style: Commas should always have a space after.

​      timeROC::timeROC(T = truth[,1L], delta = truth[,2L], marker = response, times = max.time,
                                  ^

R/measures.R:1405:55: style: Commas should always have a space after.

​      timeROC::timeROC(T = truth[,1L], delta = truth[,2L], marker = response, times = max.time,
                                                      ^

tests/testthat/test_learners_all_surv.R:12:100: style: Commas should always have a space after.

​  sub.task = subsetTask(surv.task, subset = c(1:70), features = getTaskFeatureNames(surv.task)[c(1,2)])
                                                                                                   ^

[PhilippPro]

We are somehow stuck here. How is the actual status, do we need #1742?
I also do not understand the failed travis...

My supervisor showed me this paper (https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-017-0336-2), where they recommend the Uno-Index.

[mllg]

I rewrote stuff of this PR in the survival branch in a clean way. I've discovered multiple issues, I'll get back to them in the new PR.