Allow messages to contain a captions or subtitles file

[robby1066]

Area affected

Message Authoring / Message Display

What's the problem?

Messages should be consumable by all intended recipients. In some cases, organizations will have members who can't use audio. These may be deaf members, or members who are viewing the message in a noisy environment. Having an alternative to the audio in the message would allow these people to access the content of the message.

A related problem is multi-lingual members of an organization. If some members of the organization struggle with the message author's language, a subtitles track may help.

Description of feature

At the most basic level, letting message creators who are motivated to create their own caption files add them to messages would be desirable. This would be significant work on the creator's part, but would be technically simple to accommodate. If the file is available, it could be included when the message is viewed. If not available, it would just be ignored.

A more advanced implementation would offer the option for auto-transcribing the messages using a service like AWS Transcribe. This would incur an additional cost and may only be appropriate for a higher price point.

Alternatives and workarounds

There is currently no way to provide captions or subtitles on messages. If a viewer has real-time transcription on their device (example: Android has this) they would be able to get some of this functionality, but the message creator would have no control over it.

Additional context

It's worth considering what the smallest initial implementation would be here. This will affect a very small percent of potential users, but may give them a substantial benefit. A minimally viable implementation would help clarify what future steps were necessary

[robby1066]

I just ran a test with rev.com and it looks like the workflow is reasonable. You can auto-transcribe a video by pasting a url and then have it generate a .vtt file.

If Keep Posted allowed for attaching VTT files to messages, this would close the loop and this would work.

See: https://www.rev.com/blog/resources/how-to-create-a-vtt-file-webvtt-caption-generator for more information.

I also attempted to use Otter.ai and it didn't really work. I had to download the video file, then upload it to otter, then the only transcription option was .srt, which isn't recognized in the player keep posted uses.

Rev seems like the way to go, I'll keep looking for other options.

[robby1066]

Notes from testing AWS Transcribe:

• It's a fairly reasonable approach for closed-loop captions generation
• The quality is good, but not perfect. I haven't directly compared it to other options yet.
• There's not a direct way to have Transcribe save in a .vtt file. In fact, there is only one output option. A JSON file that would need to be parsed and converted to .vtt.
• There doesn't appear to be a ruby library for converting the JSON to .vtt, but .vtt is a fairly simple file format. There are some examples in other languages (Javascript, Python) that look useful as guides.

[robby1066]

This is related to issue #15

[robby1066]

Working example of closed captions in a Keep Posted message