Provide bidirectional access to avatar humanoid bone rotations

[knah]

The main idea here is to allow external software to read and write humanoid bone rotations on the local avatar.
Humanoid bone rotations are already (mostly) synchronized via NetIK, so this shouldn't affect networking performance.

Possible use cases:
Reading:

• Corrective blendshapes which correct for imperfect weight paint when bending joints to the extreme
• Adding some custom logic based on pose that couldn't be provided by the upcoming Avatar Dynamics
  • Activating a parameter if the avatar is upside-down
  • Activating a parameter if the avatar has both hands raised
• Recording, motion capture, extra information for stream overlays

Writing:

• Better support for custom FBT implementations that use IMUs instead of trackers (SlimeVR, HaritoraX, and the rest of them)
  • These solutions produce bone angles first and foremost
  • Shipping that information directly into VRChat without the intermediate steps of estimating tracker positions and then reconstructing bone rotations via IK will likely offer better results
• Possible hardware integration, including desktop players
  • Leap Motion hand tracking
  • Gaze trackers converted to head direction
  • Manual hand pose control via external software
• Alternative IK solvers for less-than-humanoid rigs
  • Some furry avatars have different leg structure

A side note regarding reading bone rotations - to make sense of received bone rotations, they would either need to be normalized to a specific reference rig, or, alternatively (and better), avatar-specific T-pose bone positions and rotations could be provided to the external software in some fashion.

[NarryG]

When it comes to an alternative IK solver implementation, how would you propose compatability with functionality such as stations as well as animators that override tracking state in specific ways? Without full access to the internal state of the player, they'd need to take priority. This would also all have to function without introducing incompatbility that breaks worlds for other players (again, stations are the big example there.) If a world is placing you in a station and an alternative IK solver was in use, how would it be guaranteed to operate the same as the native solver such that the world stays operational?

[knah]

Last time I saw a player using FBT care about stations was approximately... Never.

But aside from that, I don't necessarily see a huge conflict here - there's already animator state behaviors that control IK/animations behaviors, and this can easily be extended to reject OSC writes to bone rotations. Parenting the player to the station is largely orthogonal to their IK too.
Additionally, the game can provide that extra information to the external solver too - player being in a station, preferred animation/tracking state for limbs, station surface and front edge positions, or any other necessary information.

Conversely, some players would prefer to forcefully override any and all animations forced on them (be it a public avatar messing with those state behaviors or a world) to avoid breaking immersion when in FBT. This is a fairly popular feature among users of a certain client mod that can't be named here.

[NoLoPhe]

Hello! I estimated the pose from 6 IMUs based on the results of the TransPose paper. I want it to work with VRChat, how do I do that. Thank you!
Link: https://twitter.com/Tn23008624/status/1494618444821839877

[EIA485]

hey, you can use lox9973/OSCMotion