feat(WebXR): Support rendering to Looking Glass holographic displays

[tbirdso]

Adds support for rendering to Looking Glass holographic display via Looking Glass WebXR polyfill.

View transforms provided by the Looking Glass WebXR polyfill are retrieved each frame and used to render into a quilt to be copied onto the Looking Glass external display.

Includes minor breaking changes to the OpenGL RenderWindow API where the "isAR" boolean is replaced with an enum for the XR session type to request.

Context

See Looking Glass support under (4) in the vtk.js WebXR roadmap: #2571

xref https://lookingglassfactory.com/webxr , https://docs.lookingglassfactory.com/keyconcepts/quilts

Results

Scene views are appropriately generated and composited to render a hologram on a Looking Glass Portrait display.

Inline browser view (swizzled):

Inline browser view (quilt):

Changes

• Defines XrSessionTypes enum in RenderWindow/Constants.js and updates RenderWindow.startXR throughout the project and examples to use the new enum definitions
• Adds support for Looking Glass XR sessions
• Adds basic cone example and documentation for Looking Glass XR support

PR and Code Checklist

• semantic-release commit messages
• Run npm run reformat

Tested On

Windows 10
Chrome 110.0.0.0
vtk.js v26.6.1
Looking Glass WebXR Polyfill v0.3.0 release

Notes

Originally tested with [email protected] pre-release, verified compatibility with 0.3.0 release.

Includes a fix where the WebGL scissor box is reset in RenderWindow.xrRender. Rendering otherwise does not reset the scissor box at the end of an animation frame but also the Looking Glass polyfill does not reset the scissor box when settings its viewport. Will follow up with investigation into whether a more sustainable fix is available in the Looking Glass polyfill or in vtk.js.

[tbirdso]

Hi @floryst , would you please take a look and let me know what I'm doing wrong in this branch? Attempting to use vtkResourceLoader to fetch the LookingGlassWebXRPolyfill script, but the LookingGlassWebXRPolyfill class is apparently not made available to the vtk.js example script after load.

Once I can get the polyfill loaded for testing I can move forward with adjusting OpenGL WebXR rendering to create the LookingGlass quilt for display.

[floryst]

It looks like that exports a module and does not register a global LookingGlassWebXRPolyfill variable. Try using a dynamic import: import(URL).then(obj => ....) for now (in a bid to avoid having to add that as a dependency).

[tbirdso]

@floryst Thanks for the feedback. When I try to use a dynamic import

import('https://unpkg.com/@lookingglass/[email protected]/dist/@lookingglass/bundle/webxr.js').then((obj) => {
  console.log('lookingglasspolyfillloaded!');
  console.log(obj);
  new obj.LookingGlassWebXRPolyfill();
});

then compile

npm run example -- LookingGlass

I get an error

ERROR in external "https://unpkg.com/@lookingglass/[email protected]/dist/@lookingglass/bundle/webxr.js"
The target environment doesn't support dynamic import() syntax so it's not possible to use external type 'module' within a script
Error: The target environment doesn't support dynamic import() syntax so it's not possible to use external type 'module' within a script
    at ExternalModule._getSourceData (C:\repos\vtk-js\node_modules\webpack\lib\ExternalModule.js:580:13)
    at ExternalModule.codeGeneration (C:\repos\vtk-js\node_modules\webpack\lib\ExternalModule.js:650:29)
    at C:\repos\vtk-js\node_modules\webpack\lib\Compilation.js:3326:22
    at C:\repos\vtk-js\node_modules\webpack\lib\Cache.js:91:34
    at Array.<anonymous> (C:\repos\vtk-js\node_modules\webpack\lib\cache\MemoryCachePlugin.js:45:13)
    at C:\repos\vtk-js\node_modules\webpack\lib\Cache.js:91:19
    at Hook.eval [as callAsync] (eval at create (C:\repos\vtk-js\node_modules\webpack\node_modules\tapable\lib\HookCodeFactory.js:33:10), <anonymous>:19:1)
    at Cache.get (C:\repos\vtk-js\node_modules\webpack\lib\Cache.js:75:18)
    at ItemCacheFacade.get (C:\repos\vtk-js\node_modules\webpack\lib\CacheFacade.js:111:15)
    at Compilation._codeGenerationModule (C:\repos\vtk-js\node_modules\webpack\lib\Compilation.js:3319:9)
 @ ./Geometry/LookingGlass/index.js 21:0-88

From what I can find online the @babel/preset-env plugin should support dynamic imports, but I can't find any other examples in vtk.js where they are used.

https://babeljs.io/blog/2019/07/03/7.5.0

Any thoughts on how to proceed?

[floryst]

Try the following: import(/* webpackIgnore: true */ 'https://unpkg.com/@lookingglass/[email protected]/dist/webxr.js')

[tbirdso]

It looks like that worked, excellent. I am able to load and instantiate the polyfill from the module. Thank you @floryst !

[aylward]

Looks good to me.!

[BryanChrisBrown]

Will follow up with investigation into whether a more sustainable fix is available in the Looking Glass polyfill or in vtk.js.

I'm definitely interested in seeing what we can do to improve this on the Looking Glass side, in general our goal is to make it as easy and straightforward as possible to integrate our WebXR Library with any WebGL framework. The connection between the rendered canvas dimensions and the framebufferwidth has definitely hit us before and there are a few approaches we've been looking at as possible alternatives, like offScreenCanvas. The most interesting one to me currently is using Virtual WebGL Contexts to improve the rendering between the two different webGL contexts we're using with the library.

On the babel side of things, we have an open issue to investigate a non-module based build option. This was primarily for adding support for global namespaces as frameworks like PlayCanvas and Babylon.JS still commonly use it.

I am glad to see you were able to find a solution for this in your build tool but would love to make it more streamlined for you. Would providing a version of the library with global namespaces improve the integration on your end?

[tbirdso]

@BryanChrisBrown Yes definitely, thanks for your work on the WebXR polyfill so far.

I've opened an issue at Looking-Glass/looking-glass-webxr#35 to track the scissor test issue we've addressed here, that would be a small change with major impact on integration.

The connection between the rendered canvas dimensions and the framebufferwidth has definitely hit us before and there are a few approaches we've been looking at as possible alternatives, like offScreenCanvas. The most interesting one to me currently is using Virtual WebGL Contexts to improve the rendering between the two different webGL contexts we're using with the library.

The variable framebuffer size may have contributed to debugging difficulties in 0.3.0 but does not seem to have been the root cause of issues. In general it is nice to control the number of views for performance, but the fixed quilt size in 0.4.0-alpha.3 is also valuable for debugging. Is it possible to support both options?

I am glad to see you were able to find a solution for this in your build tool but would love to make it more streamlined for you. Would providing a version of the library with global namespaces improve the integration on your end?

A version with global namespaces would be an integration improvement, but not strictly necessary. The solution @floryst described is sufficient for using the module-based option in our case.

[BryanChrisBrown]

The variable framebuffer size may have contributed to debugging difficulties in 0.3.0 but does not seem to have been the root cause of issues. In general it is nice to control the number of views for performance, but the fixed quilt size in 0.4.0-alpha.3 is also valuable for debugging. Is it possible to support both options?

I think we could figure something out with the API to support both options, it sounds like the main thing you want to do is reduce the view count correct? I think we can use the quilt presets to set the default view count per device, but then provide an override using the numViews input, to allow the user to specify a lower number of views. I do understand why this is useful, but I will note that going below 45 views will degrade the holographic quality you see on Looking Glass displays.

A version with global namespaces would be an integration improvement, but not strictly necessary. The solution @floryst described is sufficient for using the module-based option in our case.

Glad to hear the current solution is sufficient, It's likely the global namespace work won't occur until v0.5 which will start work after v0.4 is released.

[tbirdso]

Updated WebXR volume rendering examples with Looking Glass support and rebased on master.

[tbirdso]

@BryanChrisBrown

I think we could figure something out with the API to support both options, it sounds like the main thing you want to do is reduce the view count correct? I think we can use the quilt presets to set the default view count per device, but then provide an override using the numViews input, to allow the user to specify a lower number of views. I do understand why this is useful, but I will note that going below 45 views will degrade the holographic quality you see on Looking Glass displays.

Yes, having a default number of views that can be overridden would address the issue. This PR will adapt several of our existing examples for vtk.js WebXR support to run on Looking Glass, including several that are severely processing intensive. As you've said, diminishing the number of views improves scene responsiveness at the cost of the holographic quality.

Once this PR is merged you could try out cinematic volume rendering on the Looking Glass here: https://kitware.github.io/vtk-js/examples/WebXRHeadFullVolumeCVR.html

Glad to hear the current solution is sufficient, It's likely the global namespace work won't occur until v0.5 which will start work after v0.4 is released.

Sure, that sounds good.

[tbirdso]

@floryst @aylward I've included updates to add Looking Glass support to existing WebXR examples, ready for re-review and merge.

[aylward]

I'm still playing with the head and neck data to reduce its size (between meetings). Give me a few more minutes, and perhaps we can include it as an option? s

…

On Tue, Feb 28, 2023 at 12:04 PM Tom Birdsong ***@***.***> wrote: @floryst <https://github.com/floryst> @aylward <https://github.com/aylward> I've included updates to add Looking Glass support to existing WebXR examples, ready for re-review and merge. — Reply to this email directly, view it on GitHub <#2669 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACEJL6WE323VLUSD5B3RC3WZYV27ANCNFSM6AAAAAATZNSUAQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Stephen R. Aylward, Ph.D. Senior Director of Strategic Initiatives Chair of MONAI Advisory Board --- Kitware: *Delivering innovation.*

[tbirdso]

Added a WebXRVolume link to load ultrasound fetus volume data from VolView sample data set, in .vti format

https://data.kitware.com/#collection/586fef9f8d777f05f44a5c86/folder/634713cf11dab81428208e1e

[tbirdso]

Added @aylward 's Head_And_Chest volume with "CT-AAA" presets, please take a look.

[tbirdso]

@floryst I've addressed comments and squashed WebXRVolume commits. Ready for me to merge?

[floryst]

LGTM!

[github-actions]

🎉 This PR is included in version 26.8.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀