Add a getTexParameter enum for the last video frame uploaded to a WebGLTexture

[kenrussell]

Colleagues working on 360 degree videos in WebVR have expressed the need for more metadata associated with each video frame uploaded from an HTMLVideoElement into a WebGLTexture. The form of this metadata has already been proposed in the WEBGL_video_texture extension:
https://www.khronos.org/registry/webgl/extensions/proposals/WEBGL_video_texture/

The same metadata would need to be returned in some way from the existing core WebGL APIs (texImage2D, texSubImage2D) which take HTMLVideoElement as argument.

To avoid modifying the signatures for these methods, @jdashg has proposed adding a new WebGL-specific enum for getTexParameter which would return a "WebGLVideoFrameMetadata" object that's associated with the WebGLTexture object bound to the given target. This would be set when uploading a video frame to the texture, and left as null otherwise. It could be specified to always return the same object for a given WebGLTexture object, to avoid creating garbage in WebAssembly applications. The exact metadata needed will be decided as WEBGL_video_texture is prototyped per #2508. This way, that extension and the core APIs will return the same metadata.

We can start small, and add properties to the object as they're discovered to be necessary. Currently known to be needed are the frame's width, height, and some sort of timestamp, as already documented in WEBGL_video_texture.

CC'ing @grorg @RafaelCintron @zhenyao @kainino0x in particular for further comment.

[RafaelCintron]

What is the difference between the frame width/height metadata and the videoWidth and videoheight attributes of the video element? Why are the videowidth and videoheight attributes insufficient for 360 degree videos?

[kenrussell]

Feedback from the YouTube 360 team and some external customers (see https://bugs.chromium.org/p/chromium/issues/detail?id=639174#c28 ) has been that the video element's attributes (videoWidth, videoHeight, currentTime) are not precise enough.

[kdashg]

Width/height queries for tex images, I think would be nice to have for all DOM elements, and so probably tex images in general.

[kenrussell]

Understood. GetTexLevelParameter{if}v were removed from the OpenGL ES 2.0 and 3.0 specs (though they were reintroduced in 3.1), so it's a little difficult to reintroduce them, though the browser should track the metadata for all texture levels already.

I'd be more inclined to support this for uploads from all DOM elements if you prefer, and have some of the fields have default values if the last upload wasn't from a video. In particular I'd like to avoid requiring apps to make multiple calls to fetch this information.

Perhaps:

interface WebGLDOMMetadata {
  readonly attribute long width;
  readonly attribute long height;
  readonly attribute double videoTimestamp; // 0 if last upload didn't come from a video
  // Other fields as determined necessary
}

and queried via something like:

gl.getTexParameter(gl.TEXTURE_2D, gl.DOM_METADATA_WEBGL)

returning an instance of this object. Sound OK?

[RafaelCintron]

I would prefer that we have these metadata be individual attributes instead of returning an object with a bunch of fields that do not make sense for the object. Having individual attributes allows us to return null for ones that do not apply.

This proposal seems messy for textures with multiple subresources: mipmaps, cubemap faces, volume slices, etc. Say I upload a video element to the third 3rd mipmap of the positive X face, then upload an SVG element to the 2nd mipmap of the negative Y face, then draw a gradient on top of the video, then do a texImage2D with an array buffer to a subrect of the video, what value will I get from the "videoTimestamp" field? Does the video count as "the last upload", or does the SVG element, or the texImage2D call with the array buffer?

[kenrussell]

@RafaelCintron to clarify are you saying you want:
gl.getTexParameter(gl.TEXTURE_2D, gl.DOM_METADATA_WIDTH_WEBGL)
gl.getTexParameter(gl.TEXTURE_2D, gl.DOM_METADATA_HEIGHT_WEBGL)
and so forth? Or something different?

Can you be convinced to bundle this up into a single object? This is going to be an exploratory API - we will certainly need to add more fields - and we don't know exactly what they will need to be yet. Adding more and more queries will be more difficult than using JavaScript's open objects for extensibility.

The lack of glGetTexLevelParameter[if]v in ES 2.0 and 3.0 causes the messiness you mention. I would rather not add that API call in this extension due to unknown interactions and uncertainty about emulating it on ES 2.0 and 3.0. The scenario you mention is unlikely in the real world, and I think that with sufficient spec text (i.e., the last upload to any mip level and/or cube map face dictates the returned DOM metadata, if any), it can be resolved.

[kdashg]

I do actually think a D3D11_TEXTURE2D_DESC-style would be cleaner API, though clearly divergent from our GL specs.

glGetTexLevelParameter is easy enough to expose/reimplement, though to be performant, Chrome would likely need to shadow the values to avoid a billion round trips.

[RafaelCintron]

The WebVR community group was strongly discouraged by W3C leadership from adding getXXXData APIs that returned the same instance with changing values. Returning a brand new object every time was preferred. See the getFrameData issue.

If we prefer typed attributes on objects, I think we should add them to the WebGLTexture object itself. In lieu of that, I would prefer having them on getTexParameter as @kenrussell describes. The advantage of that mechanism is we can return null for ones that are not set such as videoTimestamp.

I agree we can spec something reasonable and forward compatible with respect to the metadata attributes. We should have any texture upload clear the existing attributes and set the ones relevant to the incoming object. ArrayBuffer uploads should count towards that.

[kenrussell]

Thanks @RafaelCintron for the pointer to WebVR's getFrameData issue. After reading the thread and given how strong the reaction was to returning the same object repeatedly with new values in it I don't want to fight that battle so let's just return a new object instance from getTexParameter. @jdashg @juj any objections?

[MarkCallow]

On Nov 1, 2017, at 2:37, Rafael Cintron ***@***.***> wrote: This proposal seems messy for textures with multiple subresources: mipmaps, cubemap faces, volume slices, etc. Say I upload a video element to the third 3rd mipmap of the positive X face, then upload an SVG element to the 2nd mipmap of the negative Y face, then draw a gradient on top of the video, then do a texImage2D with an array buffer to a subrect of the video, what value will I get from the "videoTimestamp" field? Does the video count as "the last upload", or does the SVG element, or the texImage2D call with the array buffer?

This kind of activity is not supported by the underlying external_texture extensions. Preventing this was another of the reasons for having a separate TEXTURE_EXTERNAL target in addition to providing a way to easily locate where to inject YUV to RGB shader code when necessary. The specs say that TexImage2D and friends cannot be used on this target. This is because there was no hardware support. Whether that has changed I don’t know but I am unaware of any newer OpenGL ES extensions for video to texture. The bottom line is that operations like the above cannot be supported by a zero-copy video to texture pipeline (unless there have been recent improvements in hardware).