A default text track in language X is preferred over a forced text track in language Y when audio language Y is selected

[SaurabhOfficial]

When filing a question:

1. What is the default behavior for forced-subtitle text tracks in exo-player, if no text track is selected then is it expected that the forced-subtitle track, appropriate for that particular audio track will be selected?
2. At present with the demo application, I am able to see repeated text tracks (As mentioned in Issue Filter out forced subtitle tracks from StyledPlayerControlView text track selector #9432), and selecting None doesn't select the appropriate forced-subtitle track but it disables the renderer altogether?

I have shared a sample .mpd file in the mail.

cc @vinodkputta @LuGO0

[icbaker]

The MPD provided over email uses relative URLs, so I can't use it to play back the content (since I don't know where the content is hosted).

Can you please provide a link to a hosted MPD file that will allow me to play back the content so that I can reproduce the issue?

[SaurabhOfficial]

Sure will share some public stream after reproducing this scenario, and notify here.

[SaurabhOfficial]

I was able to get multiple English subtitle texts with this stream https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/bipbop_16x9_variant.m3u8

[icbaker]

I've played with the bipbop stream and the StyledPlayerView subtitle selector. I think everything is working as expected, given that #9432 is not implemented. I observed:

• When starting playback with no explicit track selection preferences, the 'first' english subtitle track is selected. This is because it has the default selection flag (which is derived from the DEFAULT=YES HLS attribute).
• When selecting None, no text is shown. This is tracked/mentioned in this comment:
  • Filter out forced subtitle tracks from StyledPlayerControlView text track selector #9432 (comment).

What is the default behavior for forced-subtitle text tracks in exo-player, if no text track is selected then is it expected that the forced-subtitle track, appropriate for that particular audio track will be selected?

Yes, this is the expectation of ExoPlayer's internal track selection logic (ignoring StyledPlayerView for a second): If no other track selection preferences are expressed, and no track marked default is present, then a track marked forced with a language matching the current audio track language will be selected.

---

Aside: I notice in the MPD provided over email you're using the forced-subtitle value. This is what the spec defines, but due to some confusion from pre-publish drafts ExoPlayer 2.16.1 currently only recognises the forced_subtitle value. forced-subtitle is supported on the dev-v2 branch, and this fix will be included in the next release. More info in #9727.

Does that answer your questions? If not, please describe in more detail what you expect to see and what you actually see.

[SaurabhOfficial]

Aside: I notice in the MPD provided over email you're using the forced-subtitle value. This is what the spec defines, but due to some confusion from pre-publish drafts ExoPlayer 2.16.1 currently only recognises the forced_subtitle value. forced-subtitle is supported on the dev-v2 branch, and this fix will be included in the next release. More info in #9727.

Yes, I saw that fix for #9727 was made in the dev-v2 branch so I used that branch for testing the .mpd file shared over mail. I was able to see multiple entries in the text selection dialog since that fix worked.

Yes, this is the expectation of ExoPlayer's internal track selection logic (ignoring StyledPlayerView for a second): If no other track selection preferences are expressed, and no track marked default is present, then a track marked forced with a language matching the current audio track language will be selected.

It turns out, when I start the playback without specifically selecting any text track, no subtitle is being displayed but I was expecting that forced-subtitle for en language tag be rendered since I was playing the English audio track?
is it not expected ?

[icbaker]

It turns out, when I start the playback without specifically selecting any text track, no subtitle is being displayed but I was expecting that forced-subtitle for en language tag be rendered since I was playing the English audio track?
is it not expected ?

From what you've described that sounds unexpected. Please provide media I can use to reproduce this behaviour.

[SaurabhOfficial]

It turns out, when I start the playback without specifically selecting any text track, no subtitle is being displayed but I was expecting that forced-subtitle for en language tag be rendered since I was playing the English audio track?
is it not expected ?

From what you've described that sounds unexpected. Please provide media I can use to reproduce this behaviour.

Sure I will try and get the same stream for reproduction of this Issue and notify here again.

And one more question which might help me in understanding this better, how do we make sure that no text track is selected while playing a content so that forced subtitle for that particular audio language takes over? since If I select None renderers get disabled, and if I select a particular text track then it is expected that no forced subtitle will be rendered since it is bound to be the subset of the actual subtitle file. How do we emulate this scenario where no text track is selected in the demo application?

[icbaker]

And one more question which might help me in understanding this better, how do we make sure that no text track is selected while playing a content so that forced subtitle for that particular audio language takes over?

Using an ExoPlayer instance with no explicit track selection customisations should have the expected behaviour of selecting the forced track.

since If I select None renderers get disabled, and if I select a particular text track then it is expected that no forced subtitle will be rendered since it is bound to be the subset of the actual subtitle file. How do we emulate this scenario where no text track is selected in the demo application?

In the demo app you should be able to achieve this by starting playback and not interacting with either of the subtitle selection controls.

[SaurabhOfficial]

In the demo app you should be able to achieve this in two ways:

• Start playback and don't interact with either of the subtitle selection controls.
• If playback has already started and you want to restore 'default' behaviour then use the 'SELECT TRACKS' button at the top of the screen, then the 'TEXT' tab and then the 'Auto' option.

The Auto option doesn't render the forced subtitle for me even in the bipbop stream that I shared above. When I select none in the bottom cc image button and auto in the select tracks dialog, I am not sure if It's expected to render the forced-subtitle or track marked with DEFAULT=YES, but I was not able to see any subtitle track while playing it.

Similarly, I am not able to get any forced subtitle in the MPD stream as well( which did not define any such default text track) when I selected None in the bottom cc button and Auto for the select tracks button.
I will try and get the stream available for testing or try and find any similar dash stream where I am able to reproduce this issue.

[icbaker]

I just played around some more, and I think my second bullet was incorrect - I've removed it now. The best way to get 'automatic' behaviour is just to start playback from scratch. The two track selectors in the demo app interact in quite a confusing way.

I observe:

• Selecting a specific track or 'None' (i.e. not 'Auto') in the top (demo-app only) selector will cause the same track to be selected in the bottom (StyledPlayerView) selector (providing the StyledPlayerView selector is not already set to 'None').
  • However in this state it's impossible to select a different specific track from the StyledPlayerView selector (tapping them has no effect, the 'old' track remains selected).
  • Selecting 'None' in the StyledPlayerView selector does work (subtitles disappear) - however the 'top' selector still shows the 'old' selection.
    • In this state (e.g. top=French, StyledPlayerView=None) then selecting any specific track in the StyledPlayerView will re-enable the top-selected track (i.e. French).
• Selecting 'Auto' in the top selector will re-select whatever track was selected by StyledPlayerView before. i.e. the following steps will have the following result:
  • Start playback of the bipbop 16x9 stream. English subtitles are shown. Tops selector says 'Auto', StyledPlayerView says 'English')
  • Select 'French' in StyledPlayerView. French subtitles are shown. Top selector still says 'Auto'.
  • Select 'Spanish' in top selector. Spanish subtitles are shown. StyledPlayerView selector now says 'Spanish'.
  • Select 'Auto' in top selector. French subtitles are shown. StyledPlayerView selector says 'French'.

I think these are all UI problems resulting from having two different text track selectors in the demo app - with one intended to be a 'real' track selector that you might show to end users, while the other is a deliberately 'advanced' one that shows all available tracks, even if an end-user should never specifically select them. I don't think any of this indicates a problem with the underlying track selection logic (which of course is mostly unrelated to the UI components: https://exoplayer.dev/track-selection.html).

[SaurabhOfficial]

True once I uninstall the application and then start the .mpd stream again without fiddling with the controls, forced English subtitles to start playing since I started the playback with English audio, StyledPlayerView at the bottom says English and select tracks at the top says Auto.
One more question before I close the Issue, is there a way to change the audio track and see the corresponding change in text track as well?
what I tried :

1. Started playback without touching controls, starts with English Audio and forced English subtitles.
2. Tried switching the audio track from the bottom selector to Hindi still the forced English subtitles played on
3. Again uninstalled the application and did the 1st step again, but this time used the top track selector to change the audio track still the forced-subtitle of English language played on.

I am trying to get the mpd uri available for testing meanwhile if possible is there any open-source dash sample with forced subtitle with which I can try to simulate this scenario? I tried almost all the uri's in the demo application did not find any with forced subtitle

[icbaker]

I am trying to get the mpd uri available for testing meanwhile if possible is there any open-source dash sample with forced subtitle with which I can try to simulate this scenario? I tried almost all the uri's in the demo application did not find any with forced subtitle

I hacked the demo app to 'manually' add some subtitle tracks with various languages and selection flags to a playback using MediaItem.Builder#setSubtitleConfigurations. I replaced this line with the following code:

ExoPlayer/demos/main/src/main/java/com/google/android/exoplayer2/demo/PlayerActivity.java

Line 255 in 46ab94b

mediaItems = createMediaItems(intent);

MediaItem mediaItem = new MediaItem.Builder()
    .setUri(
        "https://storage.googleapis.com/wvmedia/clear/h264/tears/tears.mpd")
    .setSubtitleConfigurations(
        ImmutableList.of(
            new SubtitleConfiguration.Builder(
                    Uri.parse(
                        "https://storage.googleapis.com/exoplayer-test-media-1/ttml/netflix_ttml_sample.xml"))
                .setMimeType(MimeTypes.APPLICATION_TTML)
                .setLanguage("eng")
                .setSelectionFlags(C.SELECTION_FLAG_FORCED)
                .build(),
            new SubtitleConfiguration.Builder(
                    Uri.parse(
                        "https://storage.googleapis.com/exoplayer-test-media-1/webvtt/numeric-lines.vtt"))
                .setMimeType(MimeTypes.TEXT_VTT)
                .setLanguage("eng")
                .build(),
            new SubtitleConfiguration.Builder(
                    Uri.parse(
                        "https://storage.googleapis.com/exoplayer-test-media-1/ttml/netflix_japanese_ttml.xml"))
                .setMimeType(MimeTypes.APPLICATION_TTML)
                .setLanguage("jpn")
                .setSelectionFlags(C.SELECTION_FLAG_FORCED)
                .build()))
mediaItems = ImmutableList.of(mediaItem);

The result is that whatever sample you pick from the first screen of the demo app it will always play the above media.

When I play this I observe that the TTML English text track is selected (this is expected, because it's marked FORCED and its language matches the selected audio language of the media).

(I couldn't simply add a JSON snippet because we don't have any JSON syntax for the selection flags)

---

One more question before I close the Issue, is there a way to change the audio track and see the corresponding change in text track as well? what I tried :

1. Started playback without touching controls, starts with English Audio and forced English subtitles.
2. Tried switching the audio track from the bottom selector to Hindi still the forced English subtitles played on
3. Again uninstalled the application and did the 1st step again, but this time used the top track selector to change the audio track still the forced-subtitle of English language played on.

I didn't find a sample in the demo app with multiple audio tracks in different languages and no 'default' subtitle tracks, so I wasn't able to easily test this part - but if the Hindi track is also marked 'forced' then it should be selected when you switch to Hindi audio.

[SaurabhOfficial]

Yes it should play the Hindi track I think its because of the UI Issues, The forced text track is not getting changed Thanks for all the help, will try to fiddle with the StyledPlayerControlView to fix the Issue of filtering the available tracks using the flag C.SELECTION_FLAG_FORCED also will try and make the behavior of None in the bottom selector similar to the Auto option in the top select tracks dialog.
Closing this Issue please provide any pointers that I should take care while implementing #9432 if you have any in your mind!

[icbaker]

I added a comment to #9432 about whether the UI should expose that a forced track is selected at all.

While looking into this issue I also spotted what I think is a bug in DefaultTrackSelector when the content has both default and forced tracks in a variety of languages. Something like:

audio:
  english
  german
text:
  english, default
  english, forced
  german, forced

Current behaviour:

• When the english audio track is selected the default english text track is selected.
  • It's unclear to me whether this is expected vs selecting the forced text track.
• When the german audio track is selected the default english text track is still selected.
  • This seems unexpected, because the user has expressed a preference for german audio and no overriding preference for subtitle language, so we should select the 'accompanying' forced german text track.

I have a pending change which changes this behaviour to:

• When the english audio track is selected the forced english text track is selected.
  • As above, it's unclear if this is expected vs selecting the default text track.
• When the german audio track is selected the forced german text track is selected.
  • This seems more expected to me.

I'm going to re-open this issue and use it to track making this change.

[ojw28]

I added a comment to #9432 about whether the UI should expose that a forced track is selected at all.

While looking into this issue I also spotted what I think is a bug in DefaultTrackSelector when the content has both default and forced tracks in a variety of languages. Something like:

audio:
  english
  german
text:
  english, default
  english, forced
  german, forced

Current behaviour:

• When the english audio track is selected the default english text track is selected.
  • It's unclear to me whether this is expected vs selecting the forced text track.

I'm pretty sure this is expected, because the track has the "default" flag specified. I'd expect this track to be a superset of what's in the forced track of the same language. As a concrete example, suppose you're providing a Star Trek streaming service to a primarily non-English speaking region (without dubbed audio). You might provide two text tracks:

1. A default track in the primary language of the region, containing full subtitles.
2. A forced track in the primary language of the region, containing text only when characters are speaking Klingon.

The idea is that you default to (1) because you think most of your users will want the full subtitles. If the user turns it off because they understand english, they then get the forced track so they can still understand the Klingon.

• When the german audio track is selected the default english text track is still selected.
  • This seems unexpected, because the user has expressed a preference for german audio and no overriding preference for subtitle language, so we should select the 'accompanying' forced german text track.

I'm not sure about this. The content has specified a default track. It doesn't seem unreasonable that the user should have to change the text track explicitly if they want something else. So this still feels working as intended to me.

[icbaker]

Current behaviour:

• When the english audio track is selected the default english text track is selected.

  • It's unclear to me whether this is expected vs selecting the forced text track.

I'm pretty sure this is expected [...]

Yes this reasoning makes sense - the commit I've sent to you for review retains this behaviour (when audioLanguage=X then a default text track in language X is preferred over a forced track in language X) - I changed it after writing the comment above (and added a test case to confirm the behaviour).

---

• When the german audio track is selected the default english text track is still selected.

  • This seems unexpected, because the user has expressed a preference for german audio and no overriding preference for subtitle language, so we should select the 'accompanying' forced german text track.

I'm not sure about this. The content has specified a default track. It doesn't seem unreasonable that the user should have to change the text track explicitly if they want something else. So this still feels working as intended to me.

Hmmm, maybe you're right. I started writing this example as a counter-argument, but it's not quite as clear cut as I thought at first. I'll post it anyway so the reasoning is clear.

To expand your Star Trek analogy slightly, let's assume we're streaming it to a mainly german-speaking region but we do provide alternate language audio tracks. Concretely, the following audio and text tracks are available:

audio:
  1. german (default)
  2. english
text:
  1. german (default) # contains both human and klingon dialogue
  2. german (forced) # contains only klingon dialogue
  3. english # contains both human and klingon dialogue
  4. english (forced) # contains only klingon dialogue

If an english-speaker (who doesn't speak german) is visiting this german-speaking region and they start watching this content, assuming the streaming service is unaware of the user's language preferences, it will default to audio=1,text=1. The user will quickly realise they don't understand the german audio - they will open the audio-track switcher and select the english audio track (2).

Under the current logic: The german subtitles will remain visible - the user will open the text-track switcher and select 'none', which should (ignoring #9432 for now) result in the forced english subtitles (track 4) being selected (because they match the audio language).

Under my proposal: When the user selects the english audio track the german subtitles will immediately/automatically disappear, because the forced english text track (4) is selected.

I suspect which is most 'expected' depends on whether a klingon or human is speaking when the user switches from the german to english audio tracks. If a klingon is speaking and the subtitles don't change language then it will be impossible to tell that the audio language change was successful until a human character speaks. The 'fix' of disabling subtitles may not be obvious in this case! A user might end up selecting track 3 just to understand the klingon (because track 4 is hidden from the text-track selector) - and then be 'stuck' with unwanted english subtitles on the english human dialogue. If a human character is speaking then it's immediately obvious that the audio language has changed but the text track hasn't, so it's more obvious that the 'fix' is to disable the german subtitles.

Same content, different viewer

[this assumes the content was originally filmed with an english dialogue track] A german-speaking viewer starts watching. They can understand both the audio and text, but they don't like the german dub - they want to listen to the original english audio but with german subtitles. They do the same operation as our english-speaking viewer: open the audio-track selector and select the english audio (2). They probably would be miffed if the german subtitles immediately disappeared (my proposal)!

So yeah, on balance I think the existing behaviour is probably less surprising.

CC @tonihei because we were discussing this over chat yesterday.

[icbaker]

After further discussion we've agreed that my example star trek content above is slightly 'broken' - because if the content provider is going to mark the german text track (1) as default, then they should also be marking the equivalent english track as default (3) too. And in that case the user experience is less surprising with the existing code: when the user switches to the english audio (2) then the default english text track (3) will be selected too.

I'm going to mark this issue back as a 'question' and close it.