Add thumbnails to the UI

[hiren3897]

Have you read the Tutorials?
YES
Have you read the FAQ and checked for duplicate open issues?
YES

What version of Shaka Player are you using?
3.0.10

Please ask your question

1. Do we have something so far to display thumbnails on the presentation timeline? I have learned the issues DASH thumbnail hack #1606 and Thumbnail track support #559
   but I don't have a clear roadmap on how to do it?

Can you please elaborate here in detail about it if there are any possibilities for this feature?

Thank You in advance :)

[ismena]

Nothing so far!
@michellezhuogg are we planning this for gsoc? (gsoc is an internal program for students to contribute to Google's OpenSource projects).

If not - we'll be happy to accept a PR, as this isn't on the roadmap for the foreseeable future (the next 6 months or so).

[joeyparrish]

Thumbnail track support without the UI (issue #559, PR #3145) landed recently. We don't have it incorporated into our UI yet. If you're building your own UI, the API is:

• player.getImageTracks()
• await player.getThumbnails(trackId, time)
• shaka.extern.Thumbnail

Then you have the image URLs and metadata to position the image where you want in the UI.

I'll convert this question into an enhancement request to add this to our UI. In the meantime, please let us know what you think of the API, and whether you are interested to extend our UI and send a PR. Thanks!

[hiren3897]

Hello @joeyparrish
Thank you for your response.
As I am building my own UI. So, if I want to implement the thumbnails API in my project.
How can I use it. because I am using npm shaka-player v3.0.10 and I think this has not yet officially released!

can you please give me a hint about it so that I can try it in my project and then when I am successful. I can do a PR :)

Thank you :)

[joeyparrish]

You can build Shaka Player from source for now instead of using NPM releases, or you can wait for the v3.1.0 release, which we intend to ship today or tomorrow.

[hiren3897]

woohoo!
Thank You so much @joeyparrish
It's great news :) waiting for release 👍

[francoism90]

Thanks for adding this for custom UI's! :)

How does this work? Is it like loading a <track> with type thumbnail?

[hiren3897]

Hello @joeyparrish,

After loading the player. I did player.getImageTracks() but I got an empty image tracks.
Then I checked my manifest by doing player.getManifest() that holds imageStreams which was also empty.
So I have a question that the imageStreams are generated from the manifest loaded in the player automatically? or the manifest should contain the <AdaptationSet> that should have the thumbnails init?

[avelad]

Example: https://dash.akamaized.net/akamai/bbb_30fps/bbb_with_tiled_thumbnails.mpd

[hiren3897]

Example: https://dash.akamaized.net/akamai/bbb_30fps/bbb_with_tiled_thumbnails.mpd

Thank you @avelad for the example :)

[avelad]

@hiren3897 You can find more examples in: https://nightly-dot-shaka-player-demo.appspot.com/demo/#panel=ALL_CONTENT;panelData=THUMBNAILS;build=uncompiled

[hiren3897]

@hiren3897 You can find more examples in: https://nightly-dot-shaka-player-demo.appspot.com/demo/#panel=ALL_CONTENT;panelData=THUMBNAILS;build=uncompiled

Thanks for this 👍
@avelad I saw your video https://github.com/google/shaka-player/files/6219623/Thumbnails.-.share.mp4.zip

Can you please tell me how you made thumbnails appeared on shaka-player with a sample code?
It will be a great help :)

[avelad]

The code is in my application, but I can share some guidelines ...

As there can be multiple image tracks, you must select one, an example would be:

function getThumbnailImageTrack () {
  const imageTracks = player.getImageTracks();
  if (imageTracks && imageTracks.length) {
    const mimeTypesPreference = [
      'image/jpeg',
      'image/png',
      'image/svg+xml',
    ];
    for (const mimeType of mimeTypesPreference) {
      const estimatedBandwidth = player_.getStats().estimatedBandwidth;
      const bestOptions = _(imageTracks).filter((track) => {
        return track.mimeType.toLowerCase() === mimeType && track.bandwidth < estimatedBandwidth * 0.01;
      }).sortBy('bandwidth').reverse().value();
      if (bestOptions && bestOptions.length) {
        return bestOptions[0];
      }
    }
    return imageTracks[0];
  }
}

Then you have to do the logic to get the image given a certain point, which would be something like this:

async function showThumbnail (position) {
  if (previousThumbnailRequest) {
    previousThumbnailRequest.abort();
  }
  if (!video_ || !player_) {
    return;
  }
  const thumbElement = document.getElementById('thumb');
  if (!thumbElement) {
    return;
  }
  const prevThumbImageElement = document.getElementById('thumb-image');
  if (prevThumbImageElement) {
    thumbElement.removeChild(prevThumbImageElement);
  }
  const width = thumbElement.clientWidth;
  let height = Math.floor(width * 9 / 16); // This is up to you, it depends on the aspect ratio you want the image to have
  const thumbVideoElement = document.getElementById('thumb-video');
  const imageTrack = getThumbnailImageTrack();
  if (thumbnails.image && imageTrack) {
    const thumbImageElement = document.createElement('img');
    thumbImageElement.id = 'thumb-image';
    thumbImageElement.draggable = false;
    const seekRange = player_.seekRange();
    const realPosition = seekRange.end - position;
    const thumbnail = await player_.getThumbnails(imageTrack.id, realPosition);
    if (!thumbnail) {
      return;
    }
    const uri = _.head(thumbnail.uris);
    let url = uri;
    if (_.startsWith(uri, 'offline:')) {
      try {
        const netEngine = player_.getNetworkingEngine();
        const requestType = shaka.net.NetworkingEngine.RequestType.APP;
        previousThumbnailRequest = netEngine.request(requestType, ({uris: [uri]}));
        const response = await previousThumbnailRequest.promise;
        previousThumbnailRequest = undefined;
        const urlCreator = window.URL || window.webkitURL;
        url = urlCreator.createObjectURL(new Blob([response.data]));
        thumbImageElement.onload = () => {
          urlCreator.revokeObjectURL(url);
        };
      } catch (e) {
        if (e && e.code === shaka.util.Error.Code.OPERATION_ABORTED) {
          return;
        }
        console.error('Error getting thumbnail', uri, e);
        return;
      }
    }
    const scale = width / thumbnail.width;
    thumbImageElement.src = url;
    thumbImageElement.style.left = '-' + scale * thumbnail.positionX + 'px';
    thumbImageElement.style.top = '-' + scale * thumbnail.positionY + 'px';
    thumbImageElement.style.transform = 'scale(' + scale + ')';
    thumbImageElement.style.transformOrigin = 'left top';
    thumbElement.insertBefore(thumbImageElement, thumbElement.firstChild);
    // Update container height
    height = Math.floor(width * thumbnail.height / thumbnail.width);
  } else {
    return;
  }
  thumbElement.style.display = 'block';
  thumbElement.style.height = height + 'px';
}

I don't think it's the best or the only implementation possible, but it's one I've been working on.

[hiren3897]

Thanks, @avelad I will take it as a reference and do implement it in my project 💯

[hiren3897]

The thumbnail is supported only for the videos that have manifest and not with the VOD links?

[avelad]

Right now it is only supported on HLS and DASH. I have seen that there are some players that support a special WebVTT track with thumbnails, if you want support for this, please open a new enhancement type issue.

More info in:
https://support.jwplayer.com/articles/how-to-add-preview-thumbnails
https://flowplayer.com/developers/plugins/thumbnails
https://developer.akamai.com/tools/AdaptiveMediaPlayer/docs/web/amp-web-react/tutorial-3-thumbnails.html

[gonzajet]

Hey @hiren3897 @avelad how are you?

I was able to render correctly the thumbnails in a certain position but this according to the manifest returns me a Sprite type image, so I can't position them in the second indicated. Have you found a way to crop the image and position it at the required second?

Example:

[hiren3897]

Hello @gonzajet,

You can use CSS properties to achieve this problem, like background Image, X and Y position, and scale the particular Image to display it on a seek bar "mouseon"
for example.
backgroundImage,
backgroundPositionX
backgroundPositionY
transform = scale(XXpx)

This code is in my application that could help you to have some idea

      const scale = width / thumbnail.width;
      this.thumbnailImage_.style.backgroundImage = 'url(' + url + ')';
      this.thumbnailImage_.style.backgroundPositionX = thumbnail.positionX + 'px';
      this.thumbnailImage_.style.backgroundPositionY = thumbnail.positionY + 'px';
      this.thumbnailImage_.style.transform = 'scale(' + scale + ')';
      this.thumbnailImage_.style.height = thumbnail.height + 'px';
      this.thumbnailImage_.style.width = thumbnail.width + 'px';

Good luck :)

[gonzajet]

Hello @hiren3897 ,

Thank you very much for your quick response!

[hoferben]

Hi all,
thanks for providing a starting point for implementing thumbnails support.
I tried to follow this direction with the UI library but unfortunately I'm stuck right at the beginning:
What is the recommended way to register or reuse an eventListener that handles a mouseover event for the seekbar?
I could not find help in the docs and simply doing

let seekBar = ui.getControls().seekBar_;
seekBar.eventManager.listen(seekBar.bar, "mouseover", (e) => {...})

did not work.

[theodab]

When are you running that code, in the load process? I tried it out myself and it seems to work fine for me. The only reason I could think that it might not work would be if you call it and then the controls are remade afterwards, or something like that.

If you set up the event listener inside the constructor of ui/seek_bar.js rather than external to it, does that work?

this.eventManager.listen(this.bar, 'mouseover', () => {
  console.log('mouseover');
});

this.eventManager.listen(this.bar, 'mouseout', () => {
  console.log('mouseout');
});

[hiren3897]

I am using Native Player(src=) in safari

[avelad]

With src= there is no way to use thumbnails in Safari.

[hiren3897]

Then can you please tell me how I can use MSE on safari? to play HLS live and VOD?

[avelad]

You can set streaming.useNativeHlsOnSafari to false

[hiren3897]

Thanks, it worked and thumbnails appeared.

can you please quickly explain to me the difference between native safari (src=) and MSE?

Specially which one is the best to use MSE or Native?

Does it also reflect other information required like seekRange() and getChaptersTracks?

[avelad]

HLS on MSE still has some limiting features, but it is fully functional. Are you using FairPlay?

Chapters work in both modes.

[hiren3897]

For HLS Live with DRM protected I am using FairPlay.

Else for other HLS live without DRM protection we use a normal HLS playlist
For both, I use Native (src=). if I use native I don't have a good seekRange() information in HLS live and sometimes the timeline is not even displayed.

Will it be better if I use MSE? Will I be able to get the seekRange and other suffs?

Because It is difficult to mark the chapter or get the thumbnails for live HLS because of a lack of seekRange() information

[hiren3897]

after setting streaming.useNativeHlsOnSafari to false on iPhone safari it still use Native (src=).

On iPad we have MSE but when I mousemove the seek-bar the thumbnail doesn't appear!! when I click or seek-bar sometimes the thumbnails appear! do you know the reason why?
Is there any other event we have to bind to have thumbnails in iPad and iPhone level when seek-bar is moved?

touchmove event? touchcancel event?

[avelad]

Yes, iOS does not support MSE yet (see https://bugs.webkit.org/show_bug.cgi?id=225564 ), on iPadOS MSE is supported as long as the web is not added to the home screen.

[hiren3897]

I am facing a problem while scrubbing on the shaka-seek-bar using the mousemove event on iPad Safari and Chrome.

On the web browser level, everything works like a charm

While testing it on iPad OS safari.
when I scrub the seek bar the thumbnails are not displayed (LIKE YOU HOLD THE INPUT RANGE ELEMENT AND TRY TO SEEK THE VIDEO)
but when I click at a certain position on seek-bar then the thumbnails are displayed but it has some inaccurate position.

I did much research on mousemove event and yes it does support on safari browser but why it is not working on iPadOS safari and chrome is a nightmare.

Do we have anything in the shaka-player UI that is affecting the mousemove event on iPad Safari and Chrome?

ON CHROME
there is no such thumbnail display, as long as we have MSE support in iPad OS then Chrome must also work for thumbnails?

@avelad can you please share with me your email so you can test it.

[hiren3897]

Hello, @avelad

we have a DASH live with a DVR of 3 hours with thumbnails.

the player is able to return only the thumbnails for lesser than 30 mins approx
If we go back more than 30 mins it returns the last thumbnail pointed at 30 mins
but not the actual request from the player.

for example, the live seek is from 40 to 200

then it gets the thumbnail between 150 to 200, after < 150 it gives 151 thumbnails only even if I hover at the start of the timeline.

here is the test URL for you: https://test-drm.hexaglobe.net/hrathod/new-plugin/dash-live-thumbs.php

If you need the manifest I can send you personally,
Please help me with this.

[hiren3897]

I guess maybe the player is not able to get the correct reference position.

shaka-player/lib/player.js

Line 3606 in 5818260

const referencePosition = imageStream.segmentIndex.find(time);

[hiren3897]

It only has a 1030 segment index for the 3 hours of LIVE
I made a debug

The duration of each thumbnail is 1
layout: 1x1

when we seek more the 30 mins the referencePosition is 0 and it returns the first segmentIndex reference. But the Dash manifest has more than that because it is 3 hours of DVR and has approx 5000+ thumbnails

LAST SEGMENT returned when referencePositon is 0

You can have a look when we find a reference it gets the correct Thumbnails from the segments

[hiren3897]

@avelad can we also see this bug on thumbnails with 3 hours of DVR?

[avelad]

@hiren3897 please, open a new issue for it. Thank you!

[avelad]

I created a PR for it #5502

[avelad]

New funcionality deployed in the nightly demo: https://nightly-dot-shaka-player-demo.appspot.com/demo/#audiolang=es-ES;textlang=es-ES;uilang=es-ES;panel=ALL_CONTENT;panelData=THUMBNAILS;build=uncompiled