Shared Textures with Native Backend

[jinleili]

On camera, video or AR apps, pixel data updated every frame, currently, we use wgpu_command_buffer_copy_buffer_to_texture to fill a texture, this command will cause pixel copy.

On these scenario, we need more efficient way to fill a wgpu texture, maybe shared memory with app client?

[jinleili]

On iOS and macOS, camera or video decode generated data is called CVPixelBuffer,on Android, camera’s frame data is called byte[] , but video decode data is called ByteBuffer, I understand that it may be hard to provide a general fn wrap a texture.

[kvark]

This is a much needed issue, which also happens to be very hard to bring in. "wgpu-native" attempts to be safe and portable. Having externally managed texture contents means it's not directly in control of our implementation, and thus it's difficult to make it safe. Portability is also thrown out because these video subsystems are all different in different platforms.

First step - we need some way to provide external texture data to gfx-hal backends. Second step would be to make our tracking aware of external resources, investigate the needs for state transitions there, etc. It's a whole ton of work.

[bes]

Sorry for bumping this, but I am looking to start experimenting on converting our video pipeline (ios/web/android/desktop/headless) which has implementation copied and slightly different for each platform into wgpu.

I am trying to get a feel for how far into the future this support would be, if there has been any movement?

As stated each platform is different, and I guess it has overlap with #274 and maybe other issues as well.

Sorry for bumping an old issue :(

[kvark]

There's been a ton of changes since this issue was updated. The last state of things is: we still want this.
One way to get there is via #1609 on Vulkan only. Another way would be to rely on texture sharing (between backends, and backend-users), which we haven't implemented yet.

[bes]

Thanks for the update @kvark, I would be happy with an OpenGL-only solution for now, but it seems I am out of luck. Would be happy to help with testing or other simliar things if needed. I really like this project 👍

[Xaeroxe]

I'm interested in making shared textures happen and might be able to provide some labor towards it. I have a specific focus on Windows, which means I'd like wgpu to be able to interact with shared resources from DirectX 11 and DirectX 12. Is there any work towards this? Do we have any specifications or prior art for how we'd like this feature to work?

[kvark]

https://docs.microsoft.com/en-us/windows/win32/direct3darticles/surface-sharing-between-windows-graphics-apis would be a good starting point

[bes]

Is the implementation of this issue (positively) affected by the new functionality in v0.15.0 around Queue::copy_external_image_to_texture by @cwfitzgerald in #3288 ? Or are "texture sharing" and "low-copy" orthogonal concerns?

[Xaeroxe]

Just as an FYI my employment situation has changed and I'm no longer going to work on this. By the way if your company needs an experienced Rust Engineer, I'm looking for a job.

[cwfitzgerald]

See #4067 for plans that should enable this.