Temporarily ban tracks after streaming errors

[sandeep-mirpuri]

Have you read the FAQ and checked for duplicate open issues?:
Yes

What version of Shaka Player are you using?:
2.4.3

Can you reproduce the issue with our latest release version?:
Yes

Can you reproduce the issue with the latest code from master?:
Yes

Are you using the demo app or your own custom app?:
Tried on both

If custom app, can you reproduce the issue using our demo app?:
Yes

What browser and OS are you using?:
Chrome 68

What are the manifest and license server URIs?:
Unable to share, but we can use any on the demo app to test this

What did you do?
Simulated 1001, 1002 and 1003 errors by proxying and overwriting stream responses with 401s

What did you expect to happen?
Similar to the native video players, example Safari's player, I expected Shaka player to attempt to use all available tracks before giving up and erroring out unrecoverably

What actually happened?
I feel like Shaka player errors out too easily. I am aware that the current streaming failure callback retries indefinitely for Live and not at all for VOD, however, it doesn't retry with other tracks.

For example, if there is something wrong with a track with of a particular bitrate, then the player should attempt with the next most appropriate bitrate. This should be the case for both Live and VOD.

Suggestions
I tried overriding the failure callback to manually selectVariantTrack and then retryStreaming. This works to switch and retry but even if the manually selected bitrate works fine, the ABR kicks in and tries again with the original failed bitrate (assuming that was the most appropriate one). Disabling the ABR doesn't make sense since the ABR could still potentially have 6-7 tracks to play with besides the failed one. Instead the ABR should remove tracks that have retried and failed a certain number of times (can be defined in the streaming or abr configuration) and only if it has tried and failed all the different tracks should it then throw some sort of unrecoverable error so that the player can surface up an error message to the user.

Please :)
This is something I need quite urgently, if it is something that needs time, in the meantime can we add something to the abr configuration to ignore tracks with certain bitrates? So that when a bitrate fails consistently, we can update the configuration to tell the abr to ignore it for that session. (Atm we only have the option to set a max and min bitrate)

Thanks again for everything Shaka has helped us with!

[ismena]

@joeyparrish could you please schedule this and/or advise on a workaround if need be

[TheModMaker]

What kinds of errors would exist for a single track? If it is a 401/403 error, it would affect every URL at that domain; if it was a 404, that seems like either bad content or a transient error that might be fixed by waiting; 5XX errors are also likely to affect every stream. Also, if there are 2 audio streams and 5 video streams, that means we will need to exhaust the retry parameters on 10 attempts before we fail to play, that is a lot of network requests to wait for a failure.

For your application, you could implement a custom AbrManager. You could subclass our implementation and filter out any variants you don't want to play.

[chrisfillmore]

@TheModMaker I get requests all the time from other teams in my org to improve player resiliency. Often the org wants to go to extreme (and IMO unreasonable) lengths to ensure the player makes every possible attempt to play something before producing an error. This is ostensibly to improve user experience, but mostly has to do with keeping the error rate down. There is an engineer who sits next to me whose sole job is to investigate error rates and video start time on web clients. I am sympathetic to @sandeep-mirpuri 's plight; demand for these types of features is strong, even if the engineering basis for them is weak.

[ismena]

@sandeep-mirpuri @chrisfillmore
I'll try to talk our TL @joeyparrish in person about this and see if he's open to a conversation in that direction.

[joeyparrish]

In a nutshell, this request is for us to adapt between tracks as a way to avoid playback errors. Is that right?

[sandeep-mirpuri]

@joeyparrish yes that is the gist of it @TheModMaker @chrisfillmore when the number of pieces to get the content becomes many the potential of particular tracks erroring out, where others don’t, gets higher. We ingest VOD content from our customers or ingest and convert live streams directly from broadcasters into VOD content. And our app makes all of these available. There are often server side ads inserted into the VOD content. All our content is on multiple origins which are behind Akamai. So there are so many places where something can go wrong and a particular track might not get written out properly or when inserting ads there may be an issue. While I agree the chances are few and we need to be fixing the conversion or ingestion algorithms that are screwing up, in the meantime we need to make sure that streaming doesn’t fail for a user on VOD or LIVE if it so happens that one track didn’t get written to the origin correctly

@TheModMaker thanks for your advice on extending shakas abr manager. I was able to work that out and hack something together. So I am not in as much of a sticky situation as 3 days ago. But I feel like a more robust solution within shakas abr manager would be more desirable in the long run than my hack.

And it just can’t be said enough, thanks to everyone working on Shaka! It is still by far one of the best players and very usable and customisable.

[KarlGallagher]

@joeyparrish The Google ExoPlayer for Android implements similar behaviour. In the event of a 404 error it will retry on the same fragment for a fixed retry count. Once retries are exhausted it will move down to the next bitrate and so on. ExoPlayer temporarily blacklists any representation that causes a the max retries to be exceeded (excluding the lowest bitrate representation), to avoid the ABR trying to select bitrates that are currently experiencing errors.

[joeyparrish]

Okay, that sounds reasonable. It will go onto the backlog for now, since our current milestones are very full. I will also change the issue name to reflect the plan. Thanks!