@avaer @skaljac
Can someone else pick this up and modify the PR, fix the conflicts, make it ready for integration ? or is this waiting on a heavy update before it can be merged?

I'm back on it, it needs a bit more work.

I actually think the pose resolution calculations in kalidokit leave a lot to be desired.

That code uses mostly vector math hacks with tons of corner case detection, which has strange singularities, like the pose having a strong tendency to lean to the left, which feels very slightly wrong.

I think we can perhaps do basic world quaternion pointing with the mediapipe data (which is presented in world XYZ based on what the model thinks) to get a much more accurate avatar pose out of it.

I see no reason the math couldn't be perfect without any corner cases, as long as the model is giving us correct data.

This page really helped me: https://google.github.io/mediapipe/solutions/pose_classification.html

Especially this diagram:

With basic 3d maths you can easily compute the corresponding avatar bones quats.

For posterity, here is the face geometry map for mediapipe:

Examples of kalidokit hax/code problems:

https://github.com/yeemachine/kalidokit/blob/505e21e5bcb86771edf96acaa72a0493bd358200/src/utils/vector.js#L243
https://github.com/yeemachine/kalidokit/blob/505e21e5bcb86771edf96acaa72a0493bd358200/src/PoseSolver/calcHips.js#L29

This is almost done.

I rewrote the core to use my own math rather than kalidokit and indeed, it turned out cleaner code and higher quality output. We should probably get rid of kalidokit entirely, porting over the necessary parts.

I also want to optimize this by running the AI pose model in a worker.

Somewhat blocked on #1848, which screws up the main model's posing.

I ended up implementing the isolated origin iframe version of this to take advantage of the strict process separation in that mode.

Note that to make this work, fake RAF is need to make the iframe seem active in 3d. Apparently rendering is not enough to trigger that heuristic in Chrome. If you do not include it you will mysteriously have the iframe tick rate throttled behind time tickets.

This is done enough that someone other than me could take it to the finish line.

• needs to take in inputs from the actual hands, rather than the fakeQuaternion which was being used to test rotation orders (it's an easy switch, the necessary code is already there and commented out)
• needs major cleanup/optimization, mostly for allocations which are everywhere in the face tracking code path. the cleanup is mostly mechanical as the code should be well behaved and abstracted already, the only problem being dead code and cloning.

Device: Dell precision 7710
Nvidia Quadro m5000m 8gb
ram: 16 gb
Ci7 6th gen 2.70 GHz

The PR has issue on my device as it doesn't shows the avatar down the bottom right, instead, it shows ''StandBy''. Meanwhile, the camera is being used. I have attached the log ss.

This has not reached step 1 yet (rebase).