Texture api change

[xiaowei-guan]

1.Merge flutter_tizen_texture_registrar.h into common interface

we add GpuBufferTexture in Texture_registrar.h for tbm surface.

2.Delete ref/unref tbm_surface code.

we add destruction_callback in GpuBufferTexture, notify app to destory tbm surface.

3.Support pixel buffer texture.

we can implement pixel buffer texture.

This code only completes the basic architecture,we don't implement the detail,
We need to discuss the feasibility of this code first, and need more suggestions.

[bwikbs]

FIrst of all, Thanks !!! I love this patch! 👍 ❤️

we add GpuBufferTexture in Texture_registrar.h for tbm surface.

But I don't know whether we can handle these difference...
I hope to hear enough opinions from others as we proceed this refactoring.

[bbrto21]

@xiaowei-guan
Could you resolve conflicting files? and you can convert this to draft not only title

[xiaowei-guan]

don't know whether we can handle these difference...
About add GpuBufferTexture,

1. we can't affect the function of pixel buffer texture, so we just need add code.
2. we need make it common, not just for tbm surface.

[xiaowei-guan]

@xiaowei-guan
Could you resolve conflicting files? and you can convert this to draft not only title

ok.

[xuelian-bai]

we add GpuBufferTexture in Texture_registrar.h for tbm surface.

But I don't know whether we can handle these difference...

For this, you may see comments inside Texture_registrar.h,

// The available texture variants.
// Only PixelBufferTexture is currently implemented.
// Other variants are expected to be added in the future.
typedef std::variant<PixelBufferTexture> TextureVariant;

If we make it common, maybe we can contribute it to community

[bwikbs]

If we make it common, maybe we can contribute it to community

If this can be the starting point for contributing to flutter, I'll be absolutely happy. 😄
If that happens, our useless efforts will be reduced a lot... but honestly, I'm not sure what will happen next stage..
So, for now, it is necessary to be careful for painless rebasing.

[bwikbs]

@xiaowei-guan
I know this patch is still in preparation, but...
Could you show me the code for calling `TextureRegistrarImpl::RegisterTexture(TextureVariant* texture)`` as soon as you make it?
Because I tried to do related work, but when you're ready, I'm going to do something else.

[xiaowei-guan]

@xiaowei-guan
I know this patch is still in preparation, but...
Could you show me the code for calling `TextureRegistrarImpl::RegisterTexture(TextureVariant* texture)`` as soon as you make it?
Because I tried to do related work, but when you're ready, I'm going to do something else.

1.Get texture register

flutter::PluginRegistrarManager::GetInstance()
          ->GetRegistrar<flutter::PluginRegistrar>(registrar)->texture_registrar()

2.RegisterTexture

 textureVariant_ =
      std::make_unique<flutter::TextureVariant>(flutter::GpuBufferTexture(
          [this](size_t width,
                 size_t height) -> const FlutterDesktopGpuBuffer * {
            this->CopyGpuBuffer(width, height);
          },
          [this]() -> void { this->Destruction(); }));

  LOG_INFO("[VideoPlayer] register texture");
  textureId_ = textureRegistrar->RegisterTexture(textureVariant_.get());

3.MarkTextureFrameAvailable

  textureRegistrar_->MarkTextureFrameAvailable(player->textureId_);

4.UnregisterTexture

textureRegistrar_->UnregisterTexture(textureId_);

[bwikbs]

Thanks for quick response:exclamation:

Well, please see if I understand it or not:

1. When we get some packet from the decoder, we call GpuBufferTexture 's some api for storing its surface data.
2. Call MarkTextureFrameAvailable
3. FlutterDesktopGpuBufferTextureCallback is called, we make a texture using GL_TEXTURE_EXTERNAL_OES to fill up FlutterOpenGLTexture

Is it correct what I understand?

[xiaowei-guan]

1. packet from the decoder, we call GpuBufferTexture 's some api for storing its surface data.

Thanks for quick response❗

Well, please see if I understand it or not:

1. When we get some packet from the decoder, we call GpuBufferTexture 's some api for storing its surface data.
2. Call MarkTextureFrameAvailable
3. FlutterDesktopGpuBufferTextureCallback is called, we make a texture using GL_TEXTURE_EXTERNAL_OES to fill up FlutterOpenGLTexture

Is it correct what I understand?

1.when we get packet from decoder,store the packet, and call MarkTextureFrameAvailable
player->mediaPacket_ = packet;
player->textureRegistrar_->MarkTextureFrameAvailable(player->textureId_);

2. when we receive copy gpu buffer callback.
   FlutterDesktopGpuBuffer *VideoPlayer::CopyGpuBuffer(size_t width,
   size_t height) {
   std::lock_guardstd::mutex lock(mutex_);
   if (mediaPacket_ == nullptr) {
   LOG_ERROR("[VideoPlayer.CopyGpuBuffer] mediaPacket_ is null");
   return nullptr;
   }
   tbm_surface_h surface;
   int ret = media_packet_get_tbm_surface(mediaPacket_, &surface);
   if (ret != MEDIA_PACKET_ERROR_NONE) {
   LOG_ERROR(
   "[VideoPlayer.onVideoFrameDecoded] media_packet_get_tbm_surface "
   "failed, error: %d",
   ret);
   media_packet_destroy(mediaPacket_);
   mediaPacket_ = nullptr;
   return nullptr;
   }
   FlutterDesktopGpuBuffer gpuBuffer;
   gpuBuffer.buffer = surface;
   gpuBuffer.width = width;
   gpuBuffer.height = height;
   return &gpuBuffer;
   }

3.when Flutter GL render finish , will trigger destruction_gpu_buffer_callback_
void VideoPlayer::Destruction() {
std::lock_guardstd::mutex lock(mutex_);
if (mediaPacket_) {
media_packet_destroy(mediaPacket_);
mediaPacket_ = nullptr;
}
}

[xiaowei-guan]

1. packet from the decoder, we call GpuBufferTexture 's some api for storing its surface data.

Thanks for quick response❗
Well, please see if I understand it or not:

1. When we get some packet from the decoder, we call GpuBufferTexture 's some api for storing its surface data.
2. Call MarkTextureFrameAvailable
3. FlutterDesktopGpuBufferTextureCallback is called, we make a texture using GL_TEXTURE_EXTERNAL_OES to fill up FlutterOpenGLTexture

Is it correct what I understand?

1.when we get packet from decoder,store the packet, and call MarkTextureFrameAvailable
player->mediaPacket_ = packet;
player->textureRegistrar_->MarkTextureFrameAvailable(player->textureId_);

1. when we receive copy gpu buffer callback.
   FlutterDesktopGpuBuffer *VideoPlayer::CopyGpuBuffer(size_t width,
   size_t height) {
   std::lock_guardstd::mutex lock(mutex_);
   if (mediaPacket_ == nullptr) {
   LOG_ERROR("[VideoPlayer.CopyGpuBuffer] mediaPacket_ is null");
   return nullptr;
   }
   tbm_surface_h surface;
   int ret = media_packet_get_tbm_surface(mediaPacket_, &surface);
   if (ret != MEDIA_PACKET_ERROR_NONE) {
   LOG_ERROR(
   "[VideoPlayer.onVideoFrameDecoded] media_packet_get_tbm_surface "
   "failed, error: %d",
   ret);
   media_packet_destroy(mediaPacket_);
   mediaPacket_ = nullptr;
   return nullptr;
   }
   FlutterDesktopGpuBuffer gpuBuffer;
   gpuBuffer.buffer = surface;
   gpuBuffer.width = width;
   gpuBuffer.height = height;
   return &gpuBuffer;
   }

3.when Flutter GL render finish , will trigger destruction_gpu_buffer_callback_
void VideoPlayer::Destruction() {
std::lock_guardstd::mutex lock(mutex_);
if (mediaPacket_) {
media_packet_destroy(mediaPacket_);
mediaPacket_ = nullptr;
}
}

I have upload some code, it can work.

[bwikbs]

I have upload some code, it can work.

Thanks! I'll check that code first!

[bwikbs]

Thank you.
Now, I understand what you're trying to do at this work.

I'll ask you one more thing

player->mediaPacket_ = packet;

1. At this time, if there is an existing packet, how about replacing it with a new packet? I've been wanting to suggest it before.

2. ExternalTexturePixelGL will replace ExternalTextureGL later? or Is there any special plan for ExternalTexturePixelGL?
   I look forward to complete this patch 😄

[xiaowei-guan]

2. ExternalTexturePixelGL will replace ExternalTextureGL later? or Is there any special plan for ExternalTexturePixelGL?
I look forward to complete this patch 😄

I view the Engine code, I find we can call MarkTextureFrameAvailable multiple, It can be successful only once, during the rendering process of one frame . so :
1.we can store the media_packet in a stack, and call MarkTextureFrameAvailable when receive onVideoFrameDecoded.
2.when receive copy buffer callback, we only need return the media_packet which on the top and clear the stack.
3.handle destruction callback.
In this way, we can reduce the lost frames at plugin side.

[bwikbs]

I think we can see the end of the long journey 👏
Overall, the texture-related code seems to be neat, so I love it.
Thanks for your hard working!

[xiaowei-guan]

he texture-related code seems to be neat, so I love it.

I think we can see the end of the long journey 👏
Overall, the texture-related code seems to be neat, so I love it.
Thanks for your hard working!

To complete this task, you gave a lot of useful suggestions, thank you very much~

[swift-kim]

Are we going to submit the change related to the public API and client_wrapper to the upstream?

[xiaowei-guan]

Are we going to submit the change related to the public API and client_wrapper to the upstream?

Yes, we want to try submit this change to upstream.
before submit, we still have some work to do:
1.we need implement unit test for gpu buffer texture.
2.we want to implement sample code with gpu buffer texure on windows, to verify our interface.

[bbrto21]

LGTM

[xiaowei-guan]

flutter/flutter/issues/83693