Cannot reproduce the same results

[Hasindri]

Hi,

Thank you very much for publishing this interesting work. I have reproduced the results on TCGA COADREAD cohort, and my results do not replicate your reported values. I downloaded the Case IDs as in your split files, patched using CLAM and encoded using CTransPath. I trained the model for 2 epochs as mentioned in your codebase. However validation c-index results that I obtained, averaged across splits for each LR,DECAY configuration is as follows.

LR | DECAY | AVG VAL_CINDEX

1E-03 | 0.0001 | 0.5904815138908460
5E-05 | 0.0001 | 0.5669782626940630
5E-04 | 1E-05 | 0.6072460042608510
1E-04 | 0.0001 | 0.5792166181879860
5E-04 | 0.001 | 0.6072460042608510
5E-04 | 0.01 | 0.63487368234452
1E-04 | 1E-05 | 0.5792166181879860
1E-03 | 0.01 | 0.6090229612498970
1E-04 | 0.01 | 0.5792166181879860
5E-04 | 0.0001 | 0.6072460042608510
1E-04 | 0.001 | 0.5792166181879860
5E-05 | 0.01 | 0.566692110764221
5E-05 | 0.001 | 0.5669782626940630
1E-03 | 1E-05 | 0.5904815138908460
1E-03 | 0.001 | 0.602858857885369
5E-05 | 1E-05 | 0.5669782626940630
  

When averaged across all configs, I get 0.589621869602649  ± 0.020049446852174125 as the val c-index which is significantly different to 0.673 reported in the paper.

Could you kindly help me understand why I am unable to reproduce the given numbers? I need to understand what I might be doing wrong. Are you reporting an averaged or the highest val c-index? And are there any other implementation details that is not mentioned in the paper?

Thank you.

[AFg6K7h4fhy2]

Tagging @ajv012 because this seems important. Good luck with your results-reproduction @Hasindri! Hope this hasn't gotten buried in time.