A chrome extension that allows teachers to record lectures and save them as a text transcript
Features a summary function which summarizes the lecture text into several key points!
Submitted to GarudaHacks2020
This project 🛠️ was made for GarudaHack2020. It is a chrome-extension with a react front-end and a background script which is injected into all teams.microsoft and hangouts.google domains. The code was written by Andre Benedito and Souradeep Saha. The logo was created by Samir Haque. The website was created by Souradeep Saha and Goziem Abuah.
The front-end was designed using the Semantic-UI framework and features a settings menu, which stores user data using the chrome.storage API, and a Notes page 📃, which dynamically renders saved notes. Each note also has its own view and edit button which allows you to see the full note and change its properties respectively.
This is just a technical overview of how the chrome extension works. If you have any questions, take a look at the contact section to reach out!
Once the extension is installed and the user is on Microsoft Teams or Google Hangouts, they can 📞 call 📞someone and begin the recording using the Manual Record -> Start Recoding button. This sends a message to a background script, which is injected into the page 📃 on load, to create a Speech Recognition object from the Web Speech API and start recording through the microphone 🎤. This audio is automatically converted into text by the API.
When the user clicks the button again, which should now be labeled "Recording in progress ..." to stop 🛑 the recording, it sends another request to the background script to stop recording and send back the text 💬 created from the speech-to-text engine.
One issue with the Web Speech API speech-to-text is that it does not ❌ have any punctuation marks ❗ since it does not listen for intonation or pauses, only individual words. To solve this issue, the unpuncuated text is sent to the DeepAffects API which features a Smart Punctuation API which re-puncuates blobs of text using a machine learning algorithm. The resulting puncuated text is then sent to a PipeDream WebHook.
Once the data is processed and the user navigates back to the Notes page 📃 or clicks the refresh 🔃 button at the top of the Notes page 📃, a request to collect new data from the PipeDream WebHookAPI is created and if there is a new transcript available, it stores it to the chrome storage API. The text is then summarized using a open-source javascript NLP called JS Summarize 😃 and stored using the chrome.storage API. This is then dynamically rendered on the Notes page 📃.
Since Chrome no longer allows packed extensions which are not on the Chrome Web Store, the extension must be added as a developer. I will look to upload this extension so this method will soon be deprecated.
- Enable chrome developer mode
- Download the latest .zip from the releases page found at https://github.com/andr-dev/Scribr/releases
- Unzip the file and place the folder somewhere it won't be deleted
- Go to chrome://extensions/ and enable the "Developer Mode" toggle in the top right corner
- Click the "Load unpacked" button and select the folder where you unziped the files.
- You should now see Scribr installed!
If you would like to contribute to this project, follow these steps.
-
Option 1
- 🍴 Fork 🍴 this repo!
-
Option 2
- 👯 Clone 👯 this repo to your local machine using
https://github.com/andr-dev/Scribr
- 👯 Clone 👯 this repo to your local machine using
-
🔨 Hack Away At It! 🔨
-
Note: To build the project, use the provided build.sh file. If you just run
yarn build, it will cause errors because of the way I created the react project.
- Create a new pull request 🔁 using
https://github.com/andr-dev/Scribr/compare/.
If you have any questions or would like to contact me directly, reach out at [email protected].
- Samir Haque for making the logo
- @wkallhof for creating JS Summarize