How to reproduce the FID indicator?

[TechSheng]

Thank you for your good job! I have a question about the FID. I use the code in the picture to calculate the FID score. However, I failed to reproduce the FID score. Would you share your experimental settings?

[TechSheng]

I use the provided model in the README.md file to calculate the FID score.

[VIROBO-15]

Hi @zuimeiyujianni
What is the FID score you are getting for babies dataset, I am also having the same problem in the evaluation

[TechSheng]

Hi @zuimeiyujianni What is the FID score you are getting for babies dataset, I am also having the same problem in the evaluation

In 'Self-Test', I calculate the FID score using 1000 generated images(using the pretrained model provided in README.md). FID score are 86.62, 51.02, 48.12. The result of my calculation is much worse than the paper. What is the FID score you are getting? 👀

[TechSheng]

Hi @zuimeiyujianni What is the FID score you are getting for babies dataset, I am also having the same problem in the evaluation

Are you also using the pretrained model provided in README.md ?

[VIROBO-15]

I have only trained babies dataset and got FID score as 81.95. According to my thought we should have actual evaluation setup for getting the desire number. @utkarshojha Can you please provide experimental setup for getting the desired output. I have trained the model for babies dataset from sketch taking ffhq as a source and not used the pretrained one given in the Readme.md

[TechSheng]

I have only trained babies dataset and got FID score as 81.95. According to my thought we should have actual evaluation setup for getting the desire number. @utkarshojha Can you please provide experimental setup for getting the desired output. I have trained the model for babies dataset from sketch taking ffhq as a source and not used the pretrained one given in the Readme.md

I also want to calculate the LPIPS score. However, the link to the dataset is invalid. Do you have the dataset used to calculate lpips score? 😄 @utkarshojha Can you update the dataset link for evaluating intra-cluster distance? 😄

[JJuOn]

I have only trained babies dataset and got FID score as 81.95. According to my thought we should have actual evaluation setup for getting the desire number. @utkarshojha Can you please provide experimental setup for getting the desired output. I have trained the model for babies dataset from sketch taking ffhq as a source and not used the pretrained one given in the Readme.md

@VIROBO-15 how many samples you used to calculate FID score? I used FFHQ->babies pretrained model from README.md and sampled 2492 images the same number of given real babies dataset.
I got FID score of 77.75.
I think the number of sampled image does matter.

[TechSheng]

I have only trained babies dataset and got FID score as 81.95. According to my thought we should have actual evaluation setup for getting the desire number. @utkarshojha Can you please provide experimental setup for getting the desired output. I have trained the model for babies dataset from sketch taking ffhq as a source and not used the pretrained one given in the Readme.md

@VIROBO-15 how many samples you used to calculate FID score? I used FFHQ->babies pretrained model from README.md and sampled 2492 images the same number of given real babies dataset. I got FID score of 77.75. I think the number of sampled image does matter.

I use 1000 samples. The lpips indicator is calculated by 1000 images in paper. So I use the 1000 samples to calculate the FID.