Video Frame Processor for Screen sharing

[majo35-6c]

I can modify media stream which i already share.

• for example i merge together media stream form my desktop and from my camera.

Does there any way for screenSharing similar to Custom Video Frame Processor for Video Tile of attendee ?

I would like to use it somehow like this:

const mergedMediaStream = chime.audioVideo.startContentShareFromScreenCapture([myVideoProcesor]);
chime.audioVideo.startContentShare(mergedMediaStream);

I can used it now this way:

const stream = await navigator.mediaDevices.getDisplayMedia({ video: true });
const screenPresenter = new ScreenPresenter();
screenPresenter.setVirtualBackgroundStream(stream);
const mediaStream = screenPresenter.getMediaStream();
chime.audioVideo.startContentShare(mediaStream);

However this way has performance issue.

Any ideas how i can build something like startContentShareFromScreenCapture with the same performance ?

[nainkunal933]

Hi @majo35-6c,

Thank you for opening this issue. If I understand your question correctly you are trying to modify the content share media stream and you achieved that with the following code.

const stream = await navigator.mediaDevices.getDisplayMedia({ video: true });
const screenPresenter = new ScreenPresenter();
screenPresenter.setVirtualBackgroundStream(stream);
const mediaStream = screenPresenter.getMediaStream();
chime.audioVideo.startContentShare(mediaStream);

But after modifying the media stream you are getting some performance issues.

Can you please expand on what type of performance issues are there? For example, is there lag in content share video, are there frame drops. Also, can you please post the console logs from your application? They contain some useful performance metrics.

[majo35-6c]

Hi @nainkunal933

With stream is all ok.

However UI of App is laggy - Animations, click events,...
I know that human detection, create sagmentaion mask and then cut person from one video and add to other stream is not easy operation.

When i used this code for virtual background and put this stream as video Input app Works fine.

I would like to have the same result for content share media stream. But there is not framework for build custom frame procesor for shared média stream

[ltrung]

@majo35-6c Could you clarify on the difference/code snippets of your code between regular video input and content share?

[majo35-6c]

@ltrung
main problem is maybe this in my ui componemt

 

const update = async () => {
        if (this.sharing) {
          await this.screenPresenter.doEffect(960, 540);
          requestAnimationFrame(update);
        } else {
          this.screenPresenter.stopInputMediaStream();
          this.stream.getTracks().forEach((track) => {
            track.stop();
          });
        }
      };
   
      const mediaStream = this.screenPresenter.getMediaStream();
      update();
      DefaultVideoTile.connectVideoStreamToVideoElement(mediaStream, this.$refs.video, false);
      this.$chime.state.audioVideo.startContentShare(mediaStream);

for video Input i use video processing API so i dont need call update for refresh canvas.

When i set fps for canvas stream
getMediaStream = () => this.inputVideoCanvas2.captureStream(25);

UI of App is not laggy

[ltrung]

So we currently do not support any video processing API for content share. However, I think you can still use the video transform pipeline to do it. I tried to apply the demo emoji video processor to a screen capture using the following code and it worked for me:

// Get the screen share
const stream = await navigator.mediaDevices.getDisplayMedia();
// Create a video transform pipeline
this.contentPipe = new DefaultVideoFrameProcessorPipeline(this.meetingLogger, [new EmojifyVideoFrameProcessor('🚀')]);
await this.contentPipe.setInputMediaStream(stream);  //Set the screen capture as input to video pipeline
await this.audioVideo.startContentShare(this.contentPipe.getActiveOutputMediaStream()); //Stream the pipeline output

Would this work for you?

[majo35-6c]

@ltrung Thank you 👍
It works perfectly with Pipeline

[ltrung]

@majo35-6c Could you add more details about your use case? What kinda virtual background or video processing you want to add to your screen capture? Trying to see whether we should add API to support video process for content share so would like to understand the use case a bit more.

[majo35-6c]

@ltrung This is my use case. I hope it is more clear now via my picture.

• in one stream i want to show my content share video and also my body from webcam.
• to one steam i merged result of segmentation and content shared video is like virtual background
• it is picture in picture mode
• also i can add to my stream copyrights mark e.g.

• custom grid with media stream and presenter in the same stream
• presenter share something - similar to Twitch live stream
• attendee - user with turn on webcam

[darek-gc]

Hello,

Is it possible to use Video Frame Processor to send frames by http requests. I would like to process frames in Rekognition as it is implemented here https://github.com/aws-samples/amazon-rekognition-virtual-proctor#architecture.

Finally, my app will work on Android and iOS, so next question whether Chime SDK for Android(https://github.com/aws/amazon-chime-sdk-android) and iOS has equivalent to Video Frame Processor service?

[darek-gc]

Seems I already found an answer for my first question, I managed to send frames by HTTP requests by modifying this example
trackit@a380821#diff-e870111fa9bc964600df41f9dedbd7396bef0bd816432b2333d0e33da18c0b15R244

to something like this

function isHTMLCanvasElement(obj: HTMLCanvasElement | OffscreenCanvas): obj is HTMLCanvasElement {
  return (obj as HTMLCanvasElement).toDataURL  !== undefined;
}
class BlurBackgroundProcessor implements VideoFrameProcessor {
  async process(buffers: VideoFrameBuffer[]): Promise<VideoFrameBuffer[]> {
    for (let i = 0; i < buffers.length; i++) {
      const canvas = await buffers[i].asCanvasElement();
      
      if (isHTMLCanvasElement(canvas)) {
        const base64Canvas = canvas.toDataURL("image/jpeg").split(';base64,')[1];
        
        await fetch('https://q#######5.execute-api.########-1.amazonaws.com/Prod/process', {
          method: 'POST',
          body: JSON.stringify({image: base64Canvas})
        });      
      }
    }
    return buffers;
  }

  async destroy() {}
}

[ltrung]

Finally, my app will work on Android and iOS, so next question whether Chime SDK for Android(https://github.com/aws/amazon-chime-sdk-android) and iOS has equivalent to Video Frame Processor service?

@darek-gc Thanks for the question. Both Android and iOS SDK support video frame processors. Please refer to the corresponding guides: https://github.com/aws/amazon-chime-sdk-ios/blob/master/guides/custom_video.md and https://github.com/aws/amazon-chime-sdk-android/blob/master/guides/custom_video.md

[darek-gc]

Thank you

[namtxhybrid]

So we currently do not support any video processing API for content share. However, I think you can still use the video transform pipeline to do it. I tried to apply the demo emoji video processor to a screen capture using the following code and it worked for me:

// Get the screen share
const stream = await navigator.mediaDevices.getDisplayMedia();
// Create a video transform pipeline
this.contentPipe = new DefaultVideoFrameProcessorPipeline(this.meetingLogger, [new EmojifyVideoFrameProcessor('🚀')]);
await this.contentPipe.setInputMediaStream(stream);  //Set the screen capture as input to video pipeline
await this.audioVideo.startContentShare(this.contentPipe.getActiveOutputMediaStream()); //Stream the pipeline output

Would this work for you?

Hi @ltrung,

According to Video Frame Processor, Can I place a video camera stream into content share? I meant the background is a content share and video camera is on it.