make bvh datasets

[birdflies]

Do anybody know how to make bvh datasets as same as the format of GENEA Challenge 2020~

[ghenter]

There are two aspects to creating the data used in the GENEA Challenge 2020:

1. Recording simultaneous speech audio and 3D motion capture of gesture motion.
2. Converting recorded materials to the format used in the GENEA Challenge (same skeleton, BVH format, etc.).

Is this question about step 1 or step 2?

[Svito-zar]

I belive it is about the step 2

[birdflies]

@ghenter 2.Converting recorded materials to the format used in the GENEA Challenge (same skeleton, BVH format, etc.).
I do not know how to transform video to the bvh format used in the GENEA Challenge~
So, how to solve it

[ghenter]

I do not know how to transform video to the bvh format used in the GENEA Challenge

I interpret this to mean that you want to convert a single-camera RGB video, like an mp4 file, into a BVH file. I'm afraid it doesn't quite work that way.

Extracting 3D motion from regular, single-view video files is extremely difficult. A lot of work is being to improve these technologies using deep learning (here's a long article on the topic), but the results are often disappointing when applied to real videos. Software such as OpenPose does a good job at identifying major keypoints on the body in 2D video, but I don't think it does a good job at giving you reliable 3D data unless you have multiple views (and, even then, it might give you coordinate-based data that is tricky to convert to a rotation-based format like BVH).

The secret to convert videos to 3D human body poses is to capture many videos from different angles of the same person at the same time. This is called (multi-view) motion capture. In other words, to do the conversion in step 2 earlier you have to start from step 1, and record your data in a very particular way. The data used for the GENEA Challenge was recorded with a system of 20 special, synchronised cameras running at 60 frames per second, as detailed in this paper.

I hope this answers your question!

[birdflies]

@ghenter Thank you~ It means that making bvh datasets as same as the format of GENEA Challenge is very difficult~
"The secret to convert videos to 3D human body poses is to capture many videos from different angles of the same person at the same time. This is called (multi-view) motion capture. In other words, to do the conversion in step 2 earlier you have to start from step 1, and record your data in a very particular way. The data used for the GENEA Challenge was recorded with a system of 20 special, synchronised cameras running at 60 frames per second, as detailed in this paper." It seems a bit difficult~

[Svito-zar]

yes, @birdflies is is fairly difficult

[ghenter]

It seems a bit difficult

This is why people write entire research publications about new datasets that they create, and why datasets like the GENEA Challenge data hold substantial value to the research community. Getting good data is expensive and hard.

[Svito-zar]

I would like to note that it was not us, but Ylva Ferstl who collected the dataset used for the GENEA Challenge