Questions about testing results

[weleen]

Thank you for your great job! I have tried to reproduce the results and encountered some issues.

Following instructions, I evaluate the provided checkpoint downloaded from huggingface.

I run the following commands:

python -m test --cfg configs/config_h3d_stage3.yaml --task t2m
python -m test --cfg configs/config_h3d_stage3.yaml --task m2t

The evaluation results are not consistent with the results reported in the paper. The attachments are the log and metrics.

t2m results:

log_2023-10-04-19-56-23_test.log

Would you happen to have any idea about what's wrong with the configuration?

[weleen]

About m2t task, the testing process is stuck at the 4th replication since the SIGTERM signal.

Similar to t2m, the testing result is behind the results reported in the paper. Especially Bleu@4 and CIDEr, only around 6 and 7.

I would appreciate it if you have time to help fix my issue.😄

[LinghaoChan]

@weleen hi! Has this issue been resolved? We met the same issue.

[weleen]

@weleen hi! Has this issue been resolved? We met the same issue.

@LinghaoChan I think there are some mistakes in get_motion_embeddings.

In m2t.py

MotionGPT/mGPT/metrics/m2t.py

Lines 325 to 329 in 0499f16

	m_lens = torch.div(m_lens,
	self.cfg.DATASET.HUMANML3D.UNIT_LEN,
	rounding_mode="floor")
	ref_mov = self.t2m_moveencoder(feats_ref[..., :-4]).detach()
	m_lens = m_lens // self.unit_length

In t2m.py

MotionGPT/mGPT/metrics/t2m.py

Lines 251 to 254 in 0499f16

	m_lens = torch.div(m_lens,
	self.cfg.DATASET.HUMANML3D.UNIT_LEN,
	rounding_mode="floor")
	m_lens = m_lens // self.cfg.DATASET.HUMANML3D.UNIT_LEN

m_lens are divided two times.

However, even I fix these errors, the results are still different. Have you solved this issus?

[Spark001]

same issue

[GuangtaoLyu]

@weleen hi! Has this issue been resolved? We met the same issue.

hi, me too.

[shin-wn]

me too.
Furthermore, I cant reproduce the real score of m2t on this MotionGPT paper and MotionGPT-2 paper.
In these papers, R-Precision, MM Dist. are following:

However, when I run test.py in this repository, the R-precision and MM dist deviate by 0.2 points and 0.07 points, respectively.

I consider this to be a very problematic deviation.