(HLS) Labelling of captions and subtitles in Safari

[dave-nicholas]

We have noticed that Safari will label the kind of the track as 'captions' if CHARACTERISTICS are present.
In shaka we then will assign labels based on the kind of the track.

I am wandering what would be the best approach to solving this issue, as we don't want to be surfacing the wrong information.
Happy to to do the work for the implementation for whatever is agreed.

The following vtt track will be labelled by shaka as CLOSED_CAPTION_MIMETYPE:

#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="English",AUTOSELECT=YES,FORCED=NO,LANGUAGE="en-US",CHARACTERISTICS="public.accessibility.transcribes-spoken-dialog,public.accessibility.describes-music-and-sound",URI="subtitles.5.m3u8"

The following vtt track will be labelled by shaka as WEBVTT_MIMETYPE :

#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="English",AUTOSELECT=YES,FORCED=NO,LANGUAGE="en-US",URI="subtitles.5.m3u8"

[gkatsev]

I took a brief look. It seems like Safari doesn't expose any way of knowing whether something was 608 or WebVTT when it comes from the manifest.

An INSTREAM-ID would come out as a captions kind as would ttml/webvtt that are included via a EXT-X-MEDIA and with the mentioned CHARACTERISTICS.

Ideally, we wouldn't be marking kind captions as 608 unless we know for sure it's 608. i.e.

shaka-player/lib/util/stream_utils.js

Lines 1409 to 1411 in 36b7367

	if (textTrack.kind == 'captions') {
	track.mimeType = CLOSED_CAPTION_MIMETYPE;
	}

[shaka-bot]

Closing due to inactivity. If this is still an issue for you or if you have further questions, the OP can ask shaka-bot to reopen it by including @shaka-bot reopen in a comment.

[tykus160]

@shaka-bot reopen

[avelad]

@dave-nicholas @gkatsev @tykus160 It's not that there are many alternatives, either we do what we have today or we don't add the mimetype (but this is a breaking change, which I don't know if we want to do)... What is your proposal?

[gkatsev]

Unfortunately, I'm not sure if there is a clear-cut answer here; just trade-offs all the way down. Though, I think if I had to choose, I might consider an unknown type here instead of defaulting to 608. Alternatively, defaulting to text/webvtt feels a bit less wrong.

[avelad]

Are you agree with track.mimeType = 'unknown'?