A standalone VR overlay designed with accessibility in mind using speech recognition technology to deliver closed captions in virtual reality.
Note: VRChat application used here only as an example for demonstration purposes
The overlay facilitates better communication in VR for deaf and hard-of-hearing users and enables you to enjoy all kinds of different media content like Twitch or YouTube with real-time closed captions.
Additionally, the overlay will automatically attempt to translate speech from different languages into the default language (e.g., English) or your native language with various success rate making it a convenient tool for real-time translation.
Note: The presence of loud music and background noise may significantly reduce the accuracy of speech recognition, highlighting the need for our further research in this area.
After installation VR Overlay runs locally on PC, meaning no network connection is required. It supports both NVIDIA CUDA GPUs and CPU mode. The latest version for Windows is available below.
- NxLL (nilpkthmsk) Unity - Text Mesh Pro Japanese
- icedterminal FFmpeg installer
- NVIDIA Corporation CUDA Toolkit
- Vincent Burel VB-Audio Virtual Audio Cable Driver
- OpenAI Whisper
- Special thanks to Dianamite and TachDeaf for helping with this project