Add OpenXR hand tracking demo

[BastiaanOlij]

This demo shows how to use the hand tracking functionality available in Godot XR.

This demo does expect certain OpenXR extensions to be implemented so this is with a view to the future. This is further explained in the readme.

I've attempted to keep things simple and not cloud the demo with all forms of fallback logic.

Todos:

• Replace Valve hands with our own/add logic to generate hands/figure out platform issues
• Add in functionality around the hand interaction profile (Add support for OpenXR hand interaction extension godot#81533)
• Add in some primitive objects that we can interact with.
• As soon as we support it, add in support for XR_EXT_hand_tracking_data_source (OpenXR: Add support for hand tracking source extension godot#85204)

[BastiaanOlij]

Turns out we don't need to change anything for XR_EXT_hand_tracking_data_source, implemented it so that this is automatically used when available. Hand/finger tracking now works both in inferred and optical hand tracking modes on the Quest.

[BastiaanOlij]

I've removed the hand tracking interaction changes for now. Planning on completing this just for hand tracking visualisation first. Then will add the interaction bits once hand interaction profiles are better supported.

[BastiaanOlij]

I did a serious rewrite of this demo so that it now uses the new nodes in Godot 4.3, makes use of hand meshes that work across platforms, and makes use of the hand interaction profile extension (if available) when hand tracking is used.

Sadly we are still awaiting adoption of this extension, see the readme.md for further information on this.

[BastiaanOlij]

OK, a bit of good news. It turns out that Meta Quest does support the Simple Controller Interaction Profile as part of optical hand tracking and this works with this demo. Sadly this profile does not recognise a grab gesture, but picking up the blocks is possible through the pinch gesture (which is exposed as the select input) and this works fully with this demo.

I have added info related to this in the readme.

Further more, there is an HTC hand interaction extension and a Microsoft hand interaction extension. The Microsoft extension is also supported by Meta.

Both extensions however are variations of the simple controller interaction profile and only serve to more easily identify whether optical hand tracking is used, or some unknown controller is used.

I've also been able to confirm that UltraLeap and by extension the Lynx R1 also support the full OpenXR hand interaction profile extension and thus work fully as intended.

[BastiaanOlij]

Also added some info that shows what we're actually tracking.

[BastiaanOlij]

Now that #93075 has been merged, I've added the MSFT hand interaction profile to this demo project and making a grabbing motion while hand tracking on a Quest works perfectly.

[Malcolmnixon]

This is working fine for me using:

• Godot 4.3-beta2
• Godot Openxr Vendors 3.0.0 beta 2

[skyace65]

Tested on Beta 2. I'm getting this with my Samsung Odyssey+(WMR Headset) in SteamVR, is this expected?

I'm unable to pick anything up.

EDIT: Also, geometry isn't rendering in my headsets right eye, I'm going to assume it's an engine but and I'll open an issue

[dsnopek]

I tested on the Meta Quest 3 with Godot 4.3-beta2 and it worked great, both with real hand-tracking and and the inferred hand-tracking when using controllers. I skimmed the code, and that looks good to me too!

[dsnopek]

EDIT: Also, geometry isn't rendering in my headsets right eye, I'm going to assume it's an engine but and I'll open an issue

I wonder if this is an issue with multiview support?

@skyace65 What GPU do you have?

[skyace65]

EDIT: Also, geometry isn't rendering in my headsets right eye, I'm going to assume it's an engine but and I'll open an issue

I wonder if this is an issue with multiview support?

@skyace65 What GPU do you have?

AMD 6950XT (16GB VRAM)

[BastiaanOlij]

EDIT: Also, geometry isn't rendering in my headsets right eye, I'm going to assume it's an engine but and I'll open an issue

I wonder if this is an issue with multiview support?
@skyace65 What GPU do you have?

AMD 6950XT (16GB VRAM)

Make sure you're on the latest drivers, there were issues with AMD drivers in the past, though I'm sure you've checked that already. I would also open the headset preview in SteamVR itself, see if it's Godot that's messing up or if it's further down the pipeline.

The controllers not working is likely because I didn't setup additional profiles. Officially according to the OpenXR spec, every runtime should fall back on other profiles, but that is a promise very few runtimes actually fulfil but only soso.
You can see in your screenshot that SteamVR is falling back on Oculus Touch controllers, but it might do some weird mapping that fails. You could add the binding for Microsoft controllers and see if that fixes it (don't forget to press save :)).

(note that it should show the hands, that goes through SteamVRs own inferred hand tracking system and completely bypasses the action set system)

[dsnopek]

@skyace65:

AMD 6950XT (16GB VRAM)

Thanks!

I had been hoping I could maybe Google your GPU and get more info, but no such luck. :-) Could you try using GLview and get a list of the OpenGL extensions supported by your GPU? If you've got GL_OVR_multiview or GL_OVR_multiview2, then your GPU should support multiview - if not, then that's probably the reason for only rendering the left eye.

[skyace65]

@skyace65:

AMD 6950XT (16GB VRAM)

Thanks!

I had been hoping I could maybe Google your GPU and get more info, but no such luck. :-) Could you try using GLview and get a list of the OpenGL extensions supported by your GPU? If you've got GL_OVR_multiview or GL_OVR_multiview2, then your GPU should support multiview - if not, then that's probably the reason for only rendering the left eye.

I downloaded the program and it said I have both. I've opened an issue for this in the Godot Repo. godotengine/godot#93921 Full details are in there but I figured out more after doing more testing, This is only happening in the compatibility renderer, it's also happening... In a weird way in one of the XR Tools demo scenes.

[skyace65]

(note that it should show the hands, that goes through SteamVRs own inferred hand tracking system and completely bypasses the action set system)

The hands are visible in the left eye, they're just not in that screenshot since I had the controllers out of view

[BastiaanOlij]

@skyace65 can you test if this has the same result?

teststereovrs.zip

[skyace65]

@skyace65 can you test if this has the same result?

teststereovrs.zip

I took your demo project and changed it to the compatability renderer since that's the one with issues. Still having the problem

Also I'm assuming this isn't supposed to output to the headset, because it's not

EDIT: Do you want to keep discussion about this in the engine repo issue?

[BastiaanOlij]

EDIT: Do you want to keep discussion about this in the engine repo issue?

We should move it into the engine issue, this is obviously not related to this demo :)

[BastiaanOlij]

Note that @skyace65 issue #93921 was resolved with #94095

[skyace65]

Everything is working for me now in 4.3 Beta 3. The issue with the right eye is gone and I can pick up the cubes.

[skyace65]

Thanks!