Evaluation, Reproducibility, Benchmarks Working Group Meeting Notes

Every year, hundreds of new algorithms are published in the field of biomedical image analysis. While validation of new methods has long been based on private data, publically available data sets and international competitions ('challenges') meanwhile allow for benchmarking algorithms in a transparent and comparative manner. Recent research, however, has revealed several flaws related to common practice in validation. A core goal of the program is, therefore, to provide the infrastructure and tools for quality-controlled validation and benchmarking of medical image analysis methods. In collaboration with the international biomedical image analysis challenges (BIAS) initiative, open technical challenges and research questions related to a variety of topics will be addressed, ranging from best practices (e.g. How to make a trained model public? How to enable reproducibility of a training process? Which metric to use for which application?) and political aspects (e.g. How to create incentives for sharing code and data?) to performance aspects (e.g. How to report memory/compute requirements? How to support identification of performance bottlenecks in pipelines?) and implementation efficiency (e.g. How to provide baseline methods for comparative assessment?).

2024

• Meeting 30 - 2024/11/27
• Meeting 29 - 2024/10/23
• Meeting 28 - 2024/09/25
• Meeting 27 - 2024/04/24
• Meeting 26 - 2024/02/28
• Meeting 25 - 2024/01/24

2023

• Meeting 24 - 2023/11/22
• Meeting 23 - 2023/09/27
• Meeting 22 - 2023/07/26
• Meeting 21 - 2023/06/27
• Meeting 20 - 2023/01/25

2022

• Meeting 19 - 2022/09/28
• Meeting 18 - 2022/07/27
• Meeting 17 - 2022/06/22
• Meeting 16 - 2022/05/25
• Meeting 15 - 2022/02/23

2021

• Meeting 14 - 2021/11/24
• Meeting 13 - 2021/07/28
• Meeting 12 - 2021/06/23
• Meeting 11 - 2021/05/26
• Meeting 10 - 2021/04/28
• Meeting 9 - 2021/03/24
• Meeting 8 - 2021/02/24
• Meeting 7 - 2021/01/27

2020

• Meeting 6 - 2020/11/25
• Meeting 5 - 2020/10/28
• Meeting 4 - 2020/09/23
• Meeting 3 - 2020/07/22
• Meeting 2 - 2020/07/01
• Meeting 1 - 2020/05/25

---

Chairs

• Annika Reinke
• Carole Sudre

Secretary

• Nicholas Heller

Group members

• Lena Maier-Hein
• Nicola Rieke
• Michela Antonelli
• M. Jorge Cardoso
• Keyvan Farahani
• Olivier Colliot
• Anne Mickan

Previous Roles

• Chair: Lena Maier-Hein
• Secretary: Annika Reinke

---

Task forces

Metrics task force

Lead: Carole Sudre

Quick data access task force

Lead: Michela Antonelli

Benchmarking task force

Lead: Annika Reinke