Add reading mode support to the Unity camera for HoloLens 2

[BrandonBray]

Describe the problem

HoloLens 2 draws at a higher resolution than apps render at. In the display pipeline, the image is resampled to enlarge it to the drawing resolution. Apps render at 1140x936 pixels, and it's possible to override the projection matrix so that the image drawn on the screen is not resampled. This results in a smaller field of view, but the resulting image seen on the display has more detail. This capability is called Reading Mode.

Describe the solution you'd like

A code sample showing how to override projection matrix is attached that shows how to do this. Unity's rendering mode must be MultiPass for this to work. It does not work with instanced rendering.

ProjectionOverride.cs

Additional context

Reading Mode can be enabled or disabled per frame. This is useful for parts of the experience that require extra detail, like reading smaller text or inspecting details in diagrams or models. When not reading, the app experience can then use the full field of view.

The developer can adjust the resolution scale.

• The maximum ResolutionScale is 45.0f/33.0f. At 45.0f/33.0f, the display pipeline will not resample the image and the image will be drawn at the same resolution that the image was rendered. This will also result in the smallest field of view on HoloLens 2. A number larger than 45.0f/33.0f will produce undefined results.
• The minimum ResolutionScale is 33.0f/33.0f -- naturally. This results in using the full field of view on HoloLens 2.
• Any ResoutionScale between the maximum and minimum will interpolate accordingly.

Overriding the projection matrix does not currently work when remote rendering.

[qian256]

If I understand correctly, Unity limits the drawing buffer size to 1140x936? If so, is this limitation going to be lifted in the future?

[wiwei]

Copy-pasting the attached code for easier reading (no need to open attachments)

using System;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;

#if WINDOWS_UWP
using Windows.Perception.Spatial;
using Windows.Graphics.Holographic;
#endif


public class ProjectionOverride : MonoBehaviour
{
    public bool EnableOverride = false;

#if WINDOWS_UWP
    private struct InternalHoloLensFrameStructure
    {
        public uint VersionNumber;
        public uint MaxNumberOfCameras;
        public IntPtr ISpatialCoordinateSystemPtr;
        public IntPtr IHolographicFramePtr;
        public IntPtr IHolographicCameraPtr;
    }
#endif

    IEnumerator ResetViewMatricesOnFrameEnd()
    {
        yield return new WaitForEndOfFrame();
        Camera cam = GetComponent<Camera>();
        cam.ResetStereoViewMatrices();
        cam.ResetStereoProjectionMatrices();
    }

    void OnPreCull()
    {
        if (!EnableOverride)
            return;

        StartCoroutine(ResetViewMatricesOnFrameEnd());

#if WINDOWS_UWP
        IntPtr nativeStruct = UnityEngine.XR.XRDevice.GetNativePtr();
        if (nativeStruct != IntPtr.Zero)
        {
            InternalHoloLensFrameStructure s = System.Runtime.InteropServices.Marshal.PtrToStructure<InternalHoloLensFrameStructure>(nativeStruct);

            if (s.IHolographicFramePtr != IntPtr.Zero)
            {
                var holographicCamera = (HolographicCamera)System.Runtime.InteropServices.Marshal.GetObjectForIUnknown(s.IHolographicCameraPtr);
                var holographicFrame = (HolographicFrame)System.Runtime.InteropServices.Marshal.GetObjectForIUnknown(s.IHolographicFramePtr);
                HolographicFramePrediction prediction = holographicFrame.CurrentPrediction;
                for (int i = 0; i < prediction.CameraPoses.Count; ++i)
                {
                    HolographicCameraPose cameraPose = prediction.CameraPoses[i];

                    // Default spatial locator is the HMD
                    SpatialLocator defaultSpatialLocator = Windows.Perception.Spatial.SpatialLocator.GetDefault();

                    SpatialLocatorAttachedFrameOfReference attachedFrameOfReference = defaultSpatialLocator.CreateAttachedFrameOfReferenceAtCurrentHeading();
                    SpatialCoordinateSystem attachedCoordinateFrame = attachedFrameOfReference.GetStationaryCoordinateSystemAtTimestamp(prediction.Timestamp);

                    HolographicStereoTransform stereoProjection = cameraPose.ProjectionTransform;
                    if (holographicCamera.CanOverrideViewport)
                    {
                        float ResolutionScale = 45.0f / 33.0f;


                        Matrix4x4 leftProj = Camera.main.GetStereoProjectionMatrix(Camera.StereoscopicEye.Left);
                        Matrix4x4 rightProj = Camera.main.GetStereoProjectionMatrix(Camera.StereoscopicEye.Right);
                        leftProj.m00 *= ResolutionScale;
                        leftProj.m11 *= ResolutionScale;
                        rightProj.m00 *= ResolutionScale;
                        rightProj.m11 *= ResolutionScale;
                        Camera.main.SetStereoProjectionMatrix(Camera.StereoscopicEye.Left, leftProj);
                        Camera.main.SetStereoProjectionMatrix(Camera.StereoscopicEye.Right, rightProj);


                        stereoProjection.Left.M11 *= ResolutionScale;
                        stereoProjection.Left.M22 *= ResolutionScale;
                        stereoProjection.Right.M11 *= ResolutionScale;
                        stereoProjection.Right.M22 *= ResolutionScale;

                        cameraPose.OverrideProjectionTransform(stereoProjection);

                    }
                }
            }
        }
#endif
    }
}