Add Gaussian Splats support on the Looking Glass through WebXR

[ncastaldo]

I would like to submit a feature request for Gaussian Splats support on the Looking Glass through WebXR, which seems currently not available.

I have tested the library locally using the React Three Fiber library and its extension called Drei.
Simple meshes are currently supported, but splats are not.
This are the docs for the Splats objects in Drei: https://github.com/pmndrs/drei?tab=readme-ov-file#splat

This is the snippet to reproduce the error:

<VRButton />
<Canvas>
  <XR>
    <Splat src="https://huggingface.co/cakewalk/splat-data/resolve/main/nike.splat" />
  </XR>
</Canvas>

These are the errors from I get in the console:

WebGL: INVALID_VALUE: uniform1fv: no array
[.WebGL-0x1040678c000] GL_INVALID_OPERATION: Mismatch between texture format and sampler type (signed/unsigned/float/shadow).

I think this might be cause by the ShaderMaterial that is used inside the <Splat> component , which might not be supported by the Looking Glass WebXR library.

Here’s a link to the Drei implementation: https://github.com/pmndrs/drei/blob/master/src/core/Splat.tsx

[BryanChrisBrown]

interesting, I've tested this locally and it's working ok, can you let me know what version of Drei, Three.JS and the Looking Glass WebXR Library you're using?

[ncastaldo]

Hey Bryan, thanks for checking this out!

Yes of course, here's the version of the packages involved:

{
  "react": "18.2.0",
  "@lookingglass/webxr": "^0.6.0",  
  "@react-three/drei": "^9.108.1",
  "@react-three/fiber": "^8.16.8",
  "@react-three/xr": "^5.7.1",
  "three": "^0.164.1",
}

[ncastaldo]

Hey @BryanChrisBrown , can you please tell me what is your setup so that I can try to replicate and see if it works?
In particular, how are you instantiating the LookingGlassWebXRPolyfill class?

[BryanChrisBrown]

@ncastaldo one thing you'll want to be careful of with react-three-fiber is re-rendering the canvas element, I'd be cautious to put any useState hooks that'd cause the canvas itself to get re-rendered.

Are you able to share a code sandbox or other link to your project?