A means to save state during detach/attach cycle for ads

[OrenMe]

Can I attach a video element, load a media, pause playback, detach video element(to be used by other libraries), call attach again and resume playback?
And all of the above is required with keeping the state and buffer of Shaka while it is detached from the video element.

Popular use case is platforms that supports only single video tag and it needs to be shared with playback of main content(by Shaka for example) and by third party ads SDK that wants to play midrolls.
Sometimes these 3rd party ad sdks are VERY opinionated and don't allow controlling the selection of video element, so you can't even tell them to use another video tag.

When such am SDK wants to play the ad it will snatch the video tag and kill the media source binding to the video element.

[OrenMe]

I see that if I call detach at a certain point in time, I can then call attach again when I want to, but the loaded media is unloaded when I call detach, so I need to call load again, and possibly add the time I detached the video element so Shaka will resume from last position.
Wa this the intended usage? can I get the behaviour I mentioned above that retains all of the current loaded media state but just detach the media source from the video element?

[matvp91]

Honestly, I think letting a third party ad SDK wipe the source buffer is not a good idea to begin with (as it does not call player.destroy(), but merely sets the src attribute of your video element I believe). I've always went with embedding ads in the manifests (when I had to work with a single video element) and handle these as a single stream, not sure if that's an option for you.

Would you mind sharing what ad SDK you're implementing? I've worked with both FreeWheel and Google IMA, both can be configured not to use a video element playing content.

[joeyparrish]

The intent is that you can call attach after detach, yes. Detach removes all association to the video element, including MediaSource/SourceBuffer and your current playback position. So after attaching, you need to call load() again if you want to resume where you left off.

The meaning of detach is not "temporarily let somebody else have the video element". Nothing like that exists today.

Current options are either to call attach() and load() again, or to have the ads SDK use a separate video element that hides the main video.

[OrenMe]

Thanks for the prompt replies @matvp91 and @joeyparrish .
I’ll shed some more light on the use case.
The ads SDK in question is IMA SDK and the platforms in question are smart TVs(LG webos and Samsung tizen) chromecast, kaiOS and some more.
These platforms do not permits having more than 1 active video tag at a time, and some have really quirky ways to ensure it(like unloading the source from any video tag if you i it another).
To add to the party, IMA SDK doesn’t ask you if you would like to us another video tag(you can trick it but better not) and it utilizes the same video tag.
For midrolls this means that during playback IMA will ask the app to give it control to play the video ad, and if it does it uses the same video tag.
The only solution currently is to destroy the current shaka instance or do detach, which, if I understand correctly, unloads current media and all its state(current buffer/cache, track state - bitrate, selected audio and text languages).
To regain playback one needs to re create the shaka player (if destroyer) or call attach and load(with the detach call time).
Maybe I’m missing something but I don’t understand why detach even exist? What are the intended workflows for it?
And in general also for attach? Would really appreciate an example, specifically in the context of my use case.

[joeyparrish]

Before attach/detach, Player instances required a video element passed into the constructor. There was a feature request (IIRC) to make this optional and allow the Player to exist without a video element. So attach/detach were added as way to enable that. Now the Player can be constructed without a video element, and the video element can be attached later, before load() is called. Detach was added to mirror that. The FR & discussion was in #1087.

Attach/detach also fits nicely into our future plans for preload(), in which you'll be able to start preloading content from several sources before attaching to a video element and completing the pipeline. This will allow applications to achieve both lower-latency startup and speculative preloading of media.

I can see the advantage of saving state for reattaching after an ad, though. In particular when you mentioned selected tracks. It's easy enough to re-attach() and re-load() at the same point in time, but more burdensome to select the same languages/tracks if the user has changed them.

I can't promise a particular timeline at the moment, but if this would be a useful feature (a way to save state somehow during detach/attach cycle), we can make this issue into a feature request.

[OrenMe]

It would be very useful so yes, please consider this as a feature request.
And I thought about the preload feature as a use case, and when you do add it I would defiantly use it for our playlist feature!

A question about how performant this approach is - if calling detach unloads the media then (with and without the discussed optimizations) calling attach again re-inits the media, it still means it will be loaded from network again, right? And if it’s right then I assume we still have fallback behavior that network calls will be loaded from browser cache(unless they have explicit expiration duration which is very short), right?

[joeyparrish]

If calling detach unloads the media then (with and without the discussed optimizations) calling attach again re-inits the media, it still means it will be loaded from network again, right?

Yes, that's right.

And if it’s right then I assume we still have fallback behavior that network calls will be loaded from browser cache(unless they have explicit expiration duration which is very short), right?

Correct.

[OrenMe]

Thanks @joeyparrish!

[SemihGk]

We had the same issue for both Tizen and LG in order to implement Freewheel ads. For Tizen, we used their AvPlayer element to play ads and at the same time, we detach the video element from DOM by using the removeChild method of DOM element. Then, reattach the video element again to DOM when ad playback is finished.

As for LG, it is similar. To play ads, we were using their video element with some options that are specific to video element in LG. Plus, hiding the element during the switch was enough instead of detaching the video element.

[ismena]

@OrenMe
Slightly OT, but we are considering developing a built-in integration for IMA SDK.
Sounds like you have some experience with it. Would you be open to answering a few questions to help us ensure we adequately understand the use cases?

[OrenMe]

Sure @ismena

[ismena]

Yay! So...
Do you rely on manifest signaling for when an ad should occur? If yes, what type of signaling?
Do you configure an ad tag in advance -- e g, know all the ad request info at load time?

If you have sample content you're able to share (privately) that would be awesome, but the discussion is valuable to us either way.

Thanks!

[OrenMe]

We don’t have clients doing client side dynamic ad insertion so we don’t have a use case for manifest signaling, but we definitely get this asked from time to time.
The common signaling is either scte-35 or some other metadata format(id3 tags).
Yes, we configure ad tag in advance, either vast or vmap, so we know the ad layout in advance.
Sometimes clients want vmap like service but have only vast so need to support such a case as well where multiple vast ad tags are configured with a timestamp.
Another extremely popular request is waterfall - define a set of ad tags for a given slot and try to play one in order of importance, so if first one doesn’t return valid vast response go to the next one and next one until one returns valid response and plays.

Unfortunately I don’t have any examples to share at the moment, but there’s a nice selection of test tags in google IMA vast inspector website.

I’ll add four more points to consider:

1. On devices that support only single video tag at a time things get complicated, and even more complicated when using MSE, as each SDK would need to get signaled that the other one wants to use the video tag and will then need to mitigate the blob and source buffer. Also to note the buffer management, as you would want to keep the buffer ahead while detaching your blob from the video tag.
   Platform to look out from: smart TVs, kaiOS.
2. SSAI - the popular request is that the player, for its UI and also event scheme, will behave exactly the same as in client side ad insertion, and this gets complicated as you are using one video tag only and it is not aware of main content context and ad context, it just plays one video file.
   You might need to add your abstraction later above it(IMA supply utility functions for that though) so UI will behave the same and also events so analytics written once will keep working.
3. UI - IMA SDK is very opinionated on UI on mobile and there are a few elements it puts there and you can’t remove or position.
4. Make it modular - IMA is best kept as modular plugin that you can use or not use in your app. I’m talking of course on the ad binding code you will write, and not IMA SDK which is simply a url you load.

Hope this helps.

[ismena]

Very helpful, @OrenMe thanks for the info!

[ismena]

@OrenMe another question (I feel a bit stupid asking this, but I want to make sure):
When I look at the sample code for SSAI IMA SDK, I see them referring to the assets by IDs as opposed to providing manifest links (the CONTENT_SOURCE_ID and VIDEO_ID references). As an AdManager/SDK user, is this what you do to get an integration with them? Do you host your content on their servers and refer to it by ids when you get to the player level?

Thanks again for your help, I'm almost dome with the initial integration design :)

[OrenMe]

Hi @ismena, sorry, I missed your last question.
Yes, there is a distribution process to the DFP ad server and you get an identifier per asset(it differs a bit for VOD and Live).
You then use this identifier on the IMA DAI ads manager to obtain your playback url and associated metadata.
It is a bit funny cause you will still need to call load source with your original source, then call the DAI SDK with the identifier, if you get a response you need to internally switch the source, and if you get an invalid response then fallback with your original source.
I guess with the request filter this might be easy on Shaka.

[joeyparrish]

I'm going to fold this into #2792.

[joeyparrish]

Duplicate of #2792