Using BFM in Python with expressions

[tobyclh]

Hello,
I've recently switched to using the Python binding of this repo like #140, not exactly getting what I want, but it's rather a minor issue. The real problem comes when am trying to use BFM instead.

Also, the estimated pose is also completely inaccurate.
It could be the fact that I'm using the blend shape of 3448 (because I cannot go without any blendshape), but I would really like to see if it is possible to achieve such thing.

[patrikhuber]

Hi!

Just to clarify. You're using the BFM with the Surrey Face Model 3448-resolution blend shapes? This can and will never work. If you're really running like that, I'm surprised you don't get any errors. Please clarify that point and then I might be able to say more with regards to the rest.

[tobyclh]

Thanks for the response and everything else 🍻
Yes, I first tried to simply call the method with none, but it throw some errors, so I replace it with the SFM 3448 blendshape.

[tobyclh]

Any thought on this?

[patrikhuber]

@tobyclh Sorry for the late reply! It's right before a paper deadline.

It seems like two things are at hand here:

1. Needing expression blendshapes (or expression basis) for the BFM. The SFM blendshapes will obviously not work as the mesh topology is completely different. eos should give an error - I may check at some point and try to add an error message if it's used in this way.
   Some people created expression blendshapes for the BFM from the FaceWarehouse database, you could check the literature/web a bit.

2. The main fitting function, in Python and C++, currently expects a vector of Blendshapes, and cannot be run without it. This is probably a problem, we should have a fitting function that works without fitting blendshapes. I don't have time currently to do this change unfortunately. If you want this, you could have a look at fit-model-simple.cpp? It fits shape and camera, without expressions. You should be able to easily expose this functionality to Python with the existing bindings. Unfortunately, fit-model-simple also fits without contour landmarks, which means the fitting results will not be too good probably. You could always also take the current fitting function and just remove the few lines that fit blendshapes, that might be easiest actually.

Let me know your thoughts.

[tobyclh]

No problem, we have all been there ;) Good luck!
Yes I went through some of the papers using the FaceWarehouse but if I remember correctly they never explicitly produce a blendshape for the purpose (or I just read the wrong papers), if you can point me to the right paper/ keywords, I can try to implement it during the summer. Else, I think I will simply expose a function that deal with BFM (and a PR if wanted

[patrikhuber]

Thank you! :-)

Hmm I think there are quite a few papers but they normally only mention it on the very side in a sentence that they have a 3DMM (e.g. in that case the BFM) and for expressions, they used FaceWarehouse to generate an expression basis (more often they create a PCA expression basis, and not blendshapes - eos could work with that as well but would need a few modifications). I'm sorry I don't have any concrete links/papers to hand (if I come across them again or in the future, I'll try to remember and link them here). I'm not sure I remember correctly but I think FaceWarehouse already provide registered scans, in the topology of the BFM? If so, then creating blendshapes for the BFM (same topology) would be super-easy.

Otherwise, as a starter, yes, just copy the current fit_shape_and_pose() function (you can make a function overload without the blendshapes parameters), remove the expression bits in the code, and expose the new one in the Python bindings.

I'll be quite busy until end of August but I'm always happy to collaborate, particularly expressions is something I wanted to probably look at anyway as one of possible work items for this autumn. Feel free to email me.

[oopming]

hi patrikhuber:
I'm using BFM instead of SFM,and need BFM expressions data,Do you have any latest developments or suggestions?I'm looking forward to your help,thank you!

[oopming]

I found similar implementations in High-Fidelity Pose and Expression Normalization for Face Recognition in the Wild

[patrikhuber]

Hi @oopming, yes, I will be working on things related to this over the next weeks (i.e. check that the BFM works properly and generating expression shapes from FaceWarehouse (but not for the BFM I think).
You can create expression blendshapes for the BFM easily by yourself though using the FaceWarehouse dataset.

[patrikhuber]

BFM2009 and BFM2017 support (with expressions) has been in the devel branch now for a while. Everything is working and quite well tested. So I think this can be closed.

[zengxianyu]

Thank you! :-)

Hmm I think there are quite a few papers but they normally only mention it on the very side in a sentence that they have a 3DMM (e.g. in that case the BFM) and for expressions, they used FaceWarehouse to generate an expression basis (more often they create a PCA expression basis, and not blendshapes - eos could work with that as well but would need a few modifications). I'm sorry I don't have any concrete links/papers to hand (if I come across them again or in the future, I'll try to remember and link them here). I'm not sure I remember correctly but I think FaceWarehouse already provide registered scans, in the topology of the BFM? If so, then creating blendshapes for the BFM (same topology) would be super-easy.

Otherwise, as a starter, yes, just copy the current fit_shape_and_pose() function (you can make a function overload without the blendshapes parameters), remove the expression bits in the code, and expose the new one in the Python bindings.

I'll be quite busy until end of August but I'm always happy to collaborate, particularly expressions is something I wanted to probably look at anyway as one of possible work items for this autumn. Feel free to email me.

Hello, I am looking for FaceWarehouse registered scans in the topology of the BFM but haven't got any clew. Do you know where can we find them?

[patrikhuber]

@zengxianyu Hi, sorry no idea from the top of my head. Maybe the 3DDFA or MOFA papers / repos are a good start to look? If you find them, I am sure people would appreciate it if you posted here what you found.

[zengxianyu]

@zengxianyu Hi, sorry no idea from the top of my head. Maybe the 3DDFA or MOFA papers / repos are a good start to look? If you find them, I am sure people would appreciate it if you posted here what you found.

I found an issue in 3DDFA repo looking for Facewarehouse meshes in BFM format but unfortunately there is no answer either.

[megahard25]

Hello, @patrikhuber ! Thank you for your great work! I have a question: can I create the same to expression_blendshapes_3448.bin file, but for BFM 2017 model? I understood that BFM 2017 contain expression model, but how I can extract it and covert to list of blendshapes?
Thank you!

[patrikhuber]

Hi @megahard25, you can't just create blendshapes from a PCA basis, but they're both a linear combination of vectors. I would recommend that you get familiar with the literature on 3DMMs and blendshapes. Also, think about whether you really need blendshapes, perhaps you can also just fit using the PCA expression model, which is the more common approach.
If you search through the past issues in this repo, I think you might find a few more pointers, people have asked similar questions before.

[zengxianyu]

Thank you! :-)
Hmm I think there are quite a few papers but they normally only mention it on the very side in a sentence that they have a 3DMM (e.g. in that case the BFM) and for expressions, they used FaceWarehouse to generate an expression basis (more often they create a PCA expression basis, and not blendshapes - eos could work with that as well but would need a few modifications). I'm sorry I don't have any concrete links/papers to hand (if I come across them again or in the future, I'll try to remember and link them here). I'm not sure I remember correctly but I think FaceWarehouse already provide registered scans, in the topology of the BFM? If so, then creating blendshapes for the BFM (same topology) would be super-easy.
Otherwise, as a starter, yes, just copy the current fit_shape_and_pose() function (you can make a function overload without the blendshapes parameters), remove the expression bits in the code, and expose the new one in the Python bindings.
I'll be quite busy until end of August but I'm always happy to collaborate, particularly expressions is something I wanted to probably look at anyway as one of possible work items for this autumn. Feel free to email me.

Hello, I am looking for FaceWarehouse registered scans in the topology of the BFM but haven't got any clew. Do you know where can we find them?

I found the procedure described here works well in converting mesh into BFM topology. First apply nicp to register the BFM mean face and FWH mean face without expression; then assign the closest triangle on FWH to each vertice of BFM and save the triangle index and the barycentric weight; then for meshes with expressions just apply these weight and corresponded triangle