Some questions about the Text-based MLC

[Eldo-rado]

Hi 👋, thanks for your great job! And I have some questions about the Text-based MLC to confirm.

1. When using MIMIC-CXR for pre-training, is the label of multi-label classification extracted by CheXpert labeler?
2. In the code(class pretrain_dataset), I found that all the data of MIMIC-CXR was used in the pre-training of multi-label classification. Will this not cause information leakage in the subsequent downstream tasks?
   Thank you in advance. I am looking forward to hearing from you!

[chenzcv7]

Hi, thanks for your interest in our work and sorry for the late reply!

1. the label of MIMIC-CXR is accompanied in the official dataset file (can be downloaded from https://physionet.org/content/mimic-cxr/2.0.0/).
2. the text-based MLC will not cause the information leakage, since 1) we only use the training split of MIMIC-CXR for pretraining, and 2) the pretraining text-based MLC task enforces the alignment of images and paired textual formed labels, where the labels play the similar role with the report. The downstream diagnosis classification task, however, performs multi-label classification based on one-hot formed label.

[Eldo-rado]

Thanks for your reply!
I would also like to confirm that: Is text-based MLC trained with the whole network in Fig. 8? (The multi-label classification performance based on the updated feature $f_g^{kv}$ can also be effective enhanced.) But why is it called a pretraining task?