Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a developer-facing API for interacting with scene anchors #107

Merged
merged 1 commit into from
Apr 8, 2024

Conversation

dsnopek
Copy link
Collaborator

@dsnopek dsnopek commented Mar 11, 2024

This builds on the work from PR #66 which added extension wrappers for XR_FB_scene and all the related extensions, but didn't expose a nice developer-facing API.

This PR adds that API!

It's centered around a new OpenXRFbSceneManager node, where you configure scenes to be instantiated for each scene anchor depending on it's semantic labels:

Selection_124

For each scene anchor that it discovers, it creates an XRAnchor3D node (which gets its position from an XRPositionTracker) <-- this is the generic support for anchors already built into Godot, which is also used by the ARKit plugin

And then depending on the semantic labels of the scene anchor, it instantiates the given scene and makes it a child of the XRAnchor3D, and calls a method on it with an OpenXRFbSpatialEntity object, which the developer can use to query any information they want about the anchor, for example, including easily creating a MeshInstance3D or CollisionShape3D.

The demo project uses this to place some boxes and labels in the room based on the captured scene data:

e56d0a2a027db5dfb386cf3db8561067
Press the Y button to toggle on/off passthrough (the scene anchors will only be shown when in passthrough) and the Menu button to do scene capture.

I also took the opportunity to make a friendlier way to access the scene capture API. Rather than calling OpenXRFbSceneCaptureExtensionWrapper.request_scene_capture(), this PR adds a similar method on the OpenXRFbSceneManager node, so developers should be able to do everything related to scene anchors via that node (never having to interact with the singletons).

This PR was already getting huge, so there's a couple things I've left out for now, which I plan to add in follow-ups:

  • Getting the triangle mesh for an anchor - this requires integrating with the XR_FB_triangle_mesh extension
  • Developer-created spatial anchors (this PR only covers scene anchors)

@dsnopek dsnopek added the enhancement New feature or request label Mar 11, 2024
@dsnopek dsnopek force-pushed the fb-spatial-entities-api branch from c8d0ea3 to 4a30c6d Compare March 11, 2024 21:08
mesh_instance->set_position(bounding_box.get_center());

} else if (is_component_enabled(COMPONENT_TYPE_BOUNDED_2D)) {
Ref<PlaneMesh> plane_mesh;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, wouldn't a quad mesh be more appropriate here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I don't think so. My understanding is that a QuadMesh is just like a PlaneMesh, except it's oriented towards Z+. The anchors are oriented towards Z-, which we get by rotating the mesh instance 90 degrees around the X-axis. I guess we could use a QuadMesh and then rotate it 180 degrees around the Y-axis, but I don't think we really gain anything by doing it that way? And the the way it is currently in the PR makes it consistent with how we handle the collision shape, which I think is a good thing.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah Quadmesh is a subclass of PlaneMesh with just some different default settings, it's mostly a convenience as this is often used for billboarding. So I would use PlaneMesh in most scenarios.

@dsnopek dsnopek force-pushed the fb-spatial-entities-api branch 2 times, most recently from 97589a1 to 5d092d2 Compare March 12, 2024 14:37
@dsnopek dsnopek force-pushed the fb-spatial-entities-api branch 2 times, most recently from 1d9206b to 4e00c04 Compare March 12, 2024 20:01
Copy link
Member

@BastiaanOlij BastiaanOlij left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, didn't see the PR till now. I need to play around with this some day but purely looking at this from a code perspective, this looks really nice!

It needs documentation, both from a user POV and from someone who wants to understand the underlying approach, but we need to decide on a place to add this. Maybe we should add a help system similar to what we did with XR tools?

@m4gr3d m4gr3d added this to the 3.0.0 milestone Mar 28, 2024
@dsnopek dsnopek force-pushed the fb-spatial-entities-api branch from 4e00c04 to 2309a62 Compare April 8, 2024 18:34
@m4gr3d m4gr3d merged commit e64452c into GodotVR:master Apr 8, 2024
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants