Predictions Speech to text doesn't work in fr-FR, en-AU, en-GB and fr-CA

[SebSchwartz]

Describe the bug
The speech to text cannot work in fr-FR, en-AU, en-GB and fr-CA as the sample rate for the streaming transcription is limited to 8 kHz and amplify always send 16 kHz in the request.
As explained in the doc -> https://docs.aws.amazon.com/transcribe/latest/dg/streaming.html

To Reproduce

1. Take the example from -> https://docs.amplify.aws/lib/predictions/sample/q/platform/js#sample-react-app
2. Specify the language in one of the listed one.
3. You will receive 2 errors
   1. One that should be fixed in Missing return in AmazonAIConvertPredictionsProvider.ts #7803 -> missing return in code
   2. The real error -> The requested language doesn't support the specified sample rate. Use the correct sample rate then try again.
4. You could try to resample to 8 kHz as requested in the aws doc with wavefile library for example (code is working for en-US)

let wav = new WaveFile();
wav.fromBuffer(new Uint8Array(fileBlob));
wav.toSampleRate(8000);

7. But the request made to Streaming Transcription endpoint is hardcoded to 16 kHz as you can see in the code (L402):
   

   amplify-js/packages/predictions/src/Providers/AmazonAIConvertPredictionsProvider.ts

   Lines 397 to 402 in ac0ef4b

   	private generateTranscribeUrl({ credentials, region, languageCode }): string {
   	const url = [
   	`wss://transcribestreaming.${region}.amazonaws.com:8443`,
   	'/stream-transcription-websocket?',
   	`media-encoding=pcm&`,
   	`sample-rate=16000&`,

Expected behavior
Amplify should detect the sample rate or allow user to pass it as param (more easy to do).

Code Snippet

const speechToTextOutput = await Predictions.convert({
                      transcription: {
                        source: {
                          bytes: rawPcm,
                        },
                        language: 'fr-FR',
                      },
                    });

[wlee221]

Hi @SebSchwartz, thanks for your research! You are right, and we should downsample the buffer to 8khz depending on the language code. We already have code for downsampling that should make this fix easier:

amplify-js/packages/predictions/src/Providers/AmazonAIConvertPredictionsProvider.ts

Line 332 in ac0ef4b

private downsampleBuffer({ buffer }) {

and we were hard setting the target downsample samplerate to 8khz here

amplify-js/packages/predictions/src/Providers/AmazonAIConvertPredictionsProvider.ts

Line 330 in ac0ef4b

private outputSampleRate = 16000;

We would just need to set that to 8khz if the target code is one of the four you mentioned (maybe add that as a parameter to downsampleBuffer). Then finally we would dynamically set

amplify-js/packages/predictions/src/Providers/AmazonAIConvertPredictionsProvider.ts

Line 402 in ac0ef4b

`sample-rate=16000&`,

to 8khz or 16khz. Do you think you can work on creating a PR for this?

[SebSchwartz]

@wlee221 I've pushed my PR #7835 ;)

[wlee221]

Thanks a lot @SebSchwartz ! I'll take a look :)

[sammartinez]

From the above pull request being merged, I am going to close this issue.

[github-actions]

This issue has been automatically locked since there hasn't been any recent activity after it was closed. Please open a new issue for related bugs.

Looking for a help forum? We recommend joining the Amplify Community Discord server *-help channels or Discussions for those types of questions.