Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stream video to a texture #5221

Open
HackerFoo opened this issue Jul 5, 2022 · 13 comments
Open

Stream video to a texture #5221

HackerFoo opened this issue Jul 5, 2022 · 13 comments
Labels
A-Rendering Drawing game state to the screen C-Feature A new feature, making something new possible S-Needs-Design This issue requires design work to think about how it would best be accomplished

Comments

@HackerFoo
Copy link
Contributor

What problem does this solve or what need does it fill?

It's useful to stream a video from a compressed form, such as MPEG4 or WebM, for:

  1. Cut scenes
  2. In game displays, such as billboards and TVs in the background
  3. Short looping videos in tutorials to explain what the player should do

What solution would you like?

I would like a way to stream compressed video from an asset to a texture that can be applied to a mesh.

What alternative(s) have you considered?

I've considered using libraries provided by the OS to layer video on top, but this isn't ideal.

Additional context

Related wgpu issue: gfx-rs/wgpu#2330
Software decoding could be provided by gstreamer: https://crates.io/crates/gstreamer

@HackerFoo HackerFoo added C-Feature A new feature, making something new possible S-Needs-Triage This issue needs to be labelled labels Jul 5, 2022
@alice-i-cecile alice-i-cecile added A-Rendering Drawing game state to the screen and removed S-Needs-Triage This issue needs to be labelled labels Jul 5, 2022
@alice-i-cecile
Copy link
Member

This is also super useful for making snippets to share on social media for both players and devs!

@drewpotter
Copy link

I planned to build a bevy_gstreamer plugin about 9 months ago, but got distracted by other things. Glad to see this issue here.

@johan-smits
Copy link

Is there progress on this issue?

@bes
Copy link
Contributor

bes commented Jul 11, 2023

There might be a more efficient way to do this for wgpu now (at least for some backends), since this was merged: Add support for importing external buffers

@Kanabenki Kanabenki assigned Kanabenki and unassigned Kanabenki Nov 23, 2023
@Kanabenki
Copy link
Contributor

Kanabenki commented Nov 23, 2023

Sorry for the issue assignment, misclicked 😇

I took a quick look at what's available for hardware video decoding depending on the platform/wgpu backend:

@bes
Copy link
Contributor

bes commented Nov 23, 2023

On the web it is also be possible to get a video frame from a video element, which is how WebGL works now, and how WebGPU shall work. (No need for WebCodecs)

@Kanabenki
Copy link
Contributor

Kanabenki commented Nov 23, 2023

Indeed I just saw that the api actually allows creating an external texture either from a (WebCodecs) VideoFrame or a video element. I remember reading somewhere there were some limitations inherent to HTMLVideoElement regarding video/frame manipulation (like frame-precise seeking) but that may not matter for this use case 🙂

@zflat
Copy link

zflat commented Apr 3, 2024

The GStreamer plugin for QtQuick uses the GStreamer method gst_gl_memory_get_texture_id to get a texture that has been uploaded earlier in the pipeline. So ideally, we could get the underlying GL context and supply it to the pipeline or make it current like what is done in the GstQt6QuickRenderer::initializeGstGL method.

Wrap the texture ID in a GpuImage and update it as new frames are pushed in GStreamer?

@schizobulia
Copy link

schizobulia commented Jun 26, 2024

I try to use gstreamer+image+rodio to play the video.
I am not a professional expert in the field of graphics / audio and video, but this demo may have some reference value.

https://github.com/schizobulia/bevy_gst_video

@tychedelia
Copy link
Contributor

@Be3y4uu-K0T
Copy link

Related wgpu issue gfx-rs/wgpu#2330 was closed as not planned. Does the Bevy community plan to implement independently to support the important, in my opinion, functionality of this for cutscenes, built-in videos, gifs, etc.?

@alice-i-cecile
Copy link
Member

At some point we need a solution for video: cut scene support is not negotiable in games. I don't have strong feelings on the best way to implement that yet though.

@alice-i-cecile alice-i-cecile added the S-Needs-Design This issue requires design work to think about how it would best be accomplished label Dec 12, 2024
@tychedelia
Copy link
Contributor

tychedelia commented Dec 13, 2024

It's blocked on gfx-rs/wgpu#4067 for using vulkan video.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-Rendering Drawing game state to the screen C-Feature A new feature, making something new possible S-Needs-Design This issue requires design work to think about how it would best be accomplished
Projects
None yet
Development

No branches or pull requests

10 participants