anonymous_openasr20_system_description.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Application of NVidia NeMo Quartz 15x5 model trained from scratch for 16000 Hz sample rate for Somali"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Anonymous team\n",
    "\n",
    "12 November 2020"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Abstract\n",
    "\n",
    "We describe our entry for NIST OpenASR20.  In EVAL Constrained condition, this system scored a WER of 1.13849 and 4th place out of 5 on the Leaderboard for Somali.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Core algorithmic approach\n",
    "\n",
    "We used the NVidia NeMo ASR package [1] and followed their instructions [2] for training a new language from scratch (Constrained condition) using the QuartzNet 15x5 model [3].  This involved creating a YAML file for Somali by modifying the example YAML file [4].\n",
    "\n",
    "The main modifications were to \n",
    "\n",
    "* Input the **grapheme set** for Somali\n",
    "\n",
    "* Decide on **maximum duration** in seconds of input sample.  We chose 10 seconds and limited our training samples to transcriptions that were 10 seconds or less in the BUILD set.\n",
    "\n",
    "* Decide on the **sample rate**. Because we initially worked with the pretrained model (Unconstrained condition), which uses 16000Hz sample rate, we stayed with 16000Hz rate for the Constrained condition (probably a mistake, as it increases parameter size for no added value).\n",
    "\n",
    "* Decide on the initial **learning rate**.  We chose a relatively high rate of 0.02 which is double the normal Novograd recommended starting rate of 0.01, because of use of highly augmented samples for training.\n",
    "\n",
    "* Decide on the **batch size**.  We chose a batch size of 180 to fit our GPU, because we felt that larger batch size would minimize overfitting.\n",
    "\n",
    "So, the following entries were changed in the base YAML file to make the Somali YAML file:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "sample_rate: &sample_rate 16000\n",
    "labels: &labels  [' ', \"'\", 'a', 'b', 'c', 'd', 'e',\n",
    " 'f', 'g', 'h', 'i', 'j',\n",
    " 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', \n",
    " 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']\n",
    "\n",
    "model:\n",
    "  train_ds:\n",
    "    sample_rate: 16000\n",
    "    batch_size: 180\n",
    "    max_duration: 10.0\n",
    "\n",
    "  optim:\n",
    "    lr: .002"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A new model from scratch is created by instantiating the `nemo_asr.models.EncDecCTCModel` class in NeMo."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional features and tools used, including software packages and publicly available external resources\n",
    "\n",
    "We used:\n",
    "\n",
    "* Python 3.7.9\n",
    "* `sph2pip_v2.5` for SPH to WAV conversion [5]\n",
    "* Python modules `IPython`, `Levenshtein`, `OpenASR_convert_reference_transcript`, `argparse`, `audioread`, `csv`, `datetime`, `glob`, `itertools`, `json`, `json`, `librosa`, `logging`, `matplotlib`, `multiprocessing`, `nemo`, `numpy`, `omegaconf`, `operator`, `os`, `pandas`, `pathlib`, `pickle`, `pprint`, `pytorch_lightning`, `random`, `re`, `ruamel`, `scipy`, `shutil`, `soundfile`, `sys`, `tarfile`, `torch`, `torchtext`, `tqdm`, `unidecode`, `warnings`\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other data used (outside provided data)\n",
    "\n",
    "Only NIST BABEL Somali BUILD samples were used for training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Significant data pre-/post-processing\n",
    "\n",
    "### Data augmentation\n",
    "\n",
    "Training audio was split according to the transcript into smaller samples per the timecodes on the scripts.\n",
    "\n",
    "Each sample was then augmented with random variations using NeMo provided perturbations [6][7].\n",
    "\n",
    "In particular we applied the following 3 perturbations in sequence 10 times to get 10 new samples:\n",
    "\n",
    "* **Time stretch** from 0.8 to 1.2.  (Not pitch preserving.)\n",
    "* **Speed change** from 0.8 to 1.2.  (Pitch preserving.)\n",
    "* **White noise** from -70db to -35 db.\n",
    "\n",
    "This is implemented in the following class:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "from nemo.collections.asr.parts import perturb\n",
    "\n",
    "class Disturb:\n",
    "\n",
    "    def __init__(self, _sample_rate):\n",
    "        self.sample_rate = _sample_rate\n",
    "        self.white_noise = \\\n",
    "           perturb.WhiteNoisePerturbation(min_level=-70, max_level=-35)\n",
    "        self.speed = perturb.SpeedPerturbation(self.sample_rate,\n",
    "            'kaiser_best', min_speed_rate=0.8, \n",
    "            max_speed_rate=1.2, num_rates=-1)\n",
    "        self.time_stretch = \\\n",
    "            perturb.TimeStretchPerturbation(min_speed_rate=0.8, \n",
    "                    max_speed_rate=1.2, num_rates=3)\n",
    "\n",
    "    def __call__(self, _sample):\n",
    "        sample=deepcopy(_sample)\n",
    "        self.time_stretch.perturb(sample)\n",
    "        self.speed.perturb(sample)\n",
    "        self.white_noise.perturb(sample)\n",
    "        return sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Speaker activity detection and translation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To reduce the unlabelled DEV and EVAL data to clips of at most 10 seconds in length, it is necessary to implement a Speaker Activity Detection function.  We explored NeMo templates for training a neural network for this purpose. This approach resulted in a very slow function.  We chose instead to implement an ad hoc manually tuned method which relies on the absolute value of the dB level of the mel spectogram to find suitably long periods of silence to cut the clips at.  The method is implemented as follows:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "def smooth(y, w):\n",
    "    box = np.ones(w)/w\n",
    "    y_smooth = np.convolve(y, box, mode='same')\n",
    "    return y_smooth\n",
    "\n",
    "def smoothhtooms(A,w):\n",
    "    A1=smooth(A,w)\n",
    "    A2=smooth(A1[::-1],w)\n",
    "    return A2[::-1]\n",
    "\n",
    "def listen_and_transcribe(C, model, max_duration, gold, audio):\n",
    "    audio /= max(abs(audio.min()), abs(audio.max()))\n",
    "    size=audio.shape[0]\n",
    "    T=size/C.sample_rate\n",
    "    X=np.arange(size)/C.sample_rate\n",
    "    Z=np.zeros(size)\n",
    "    S = librosa.feature.melspectrogram(y=audio, \n",
    "        sr=C.sample_rate, n_mels=64, fmax=8000)\n",
    "    dt_S=T/S.shape[1]\n",
    "    samples_per_spect=int(dt_S*C.sample_rate)\n",
    "    S_dB = librosa.power_to_db(S, ref=np.max)\n",
    "    s_dB_mean=np.mean(S_dB,axis=0)\n",
    "    max_samples=int(max_duration/dt_S)\n",
    "    min_samples=1\n",
    "    pred=[]\n",
    "    cutoffs = np.linspace(-80,-18,200)\n",
    "    max_read_head=s_dB_mean.shape[0]\n",
    "    max_read_head, max_samples, min_samples\n",
    "    read_head=0\n",
    "    transcriptions=[]\n",
    "    read_heads=[]\n",
    "    read_heads=[read_head]\n",
    "    while read_head < max_read_head:\n",
    "        finished = False\n",
    "        while not finished and read_head < max_read_head:\n",
    "            for cutoff in cutoffs:\n",
    "                speech_q=(s_dB_mean[read_head:]>cutoff)\n",
    "                silences=collect_false(speech_q)\n",
    "                silences=[(x,y) for x,y in silences \n",
    "                          if x != y and y-x > min_samples]\n",
    "                n_silences = len(silences)\n",
    "                if n_silences==0:\n",
    "                    continue\n",
    "                elif silences[0][0] == 0 and silences[0][1] != 0:\n",
    "                    read_head +=silences[0][1]\n",
    "                    break\n",
    "                elif silences[0][0] > max_samples:\n",
    "                    continue\n",
    "                else:\n",
    "                    silences=[(x,y) for x,y in silences \n",
    "                              if x <= max_samples]\n",
    "                    if not len(silences):\n",
    "                        continue\n",
    "                    start_at = read_head\n",
    "                    stop_at= read_head + silences[0][0]\n",
    "                    read_head = stop_at\n",
    "                    finished = True\n",
    "                    break\n",
    "            if not finished:\n",
    "                display_start=read_head*samples_per_spect\n",
    "                display_end=display_start+max_samples\n",
    "                start_at = read_head\n",
    "                stop_at = min(max_read_head, \n",
    "                              read_head + max_samples)\n",
    "                read_head = stop_at\n",
    "                finished = True\n",
    "        read_heads.append(read_head)\n",
    "        start=start_at*samples_per_spect\n",
    "        end=start_at+stop_at*samples_per_spect\n",
    "        display_start=max(0, start-5*C.sample_rate)\n",
    "        display_end=end+5*C.sample_rate\n",
    "        smooth_abs=smoothhtooms(np.abs(audio[start:end]), 100)\n",
    "        smooth_abs_max=smooth_abs.max()\n",
    "        if smooth_abs_max >= 0.05:\n",
    "            try:\n",
    "                segment_transcript, timeline, \\\n",
    "                normalized_power, speech_mask, clip_audio=\\\n",
    "                    predicted_segment_transcript(C, \\\n",
    "                          model, audio, \\\n",
    "                          start, end, s_dB_mean, \\\n",
    "                          samples_per_spect, dt_S)\n",
    "                transcriptions.extend(segment_transcript)\n",
    "            except:\n",
    "\t        print(\"empty translation\")\n",
    "    transcriptions = [(time, time+duration, pred) \n",
    "                      for time, duration, pred in transcriptions]\n",
    "    return transcriptions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The translation of a clip of 10 seconds or less in duration is performed by function `predicted_segment_transcript`.  This relies on similar thinking to break the transcribed clip into silent and speech components, and then allocate the words of the result proportionally in word size to speech component size:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "def normalize(A):\n",
    "    A=np.copy(A)\n",
    "    A=A-A.min()\n",
    "    A=A/A.max()\n",
    "    return A\n",
    "\n",
    "def predicted_segment_transcript(C, model, audio, \n",
    "            start, end, s_dB_mean, samples_per_spect, dt_S):\n",
    "    clip_audio=audio[start:end]\n",
    "    prediction=transcribe(C, model, clip_audio)\n",
    "    print(f\"PRED {start/C.sample_rate:2f} {prediction}\")\n",
    "    spec_start=int(start/samples_per_spect)\n",
    "    spec_end=int(end/samples_per_spect)\n",
    "    clip_power=s_dB_mean[spec_start:spec_end]\n",
    "    normalized_power=normalize(np.copy(clip_power))\n",
    "    timeline=np.arange(spec_start,spec_end)*dt_S\n",
    "    w=min(30, normalized_power.shape[0])\n",
    "    smoothed_normalized_power=normalize(smooth(normalized_power,w))\n",
    "    speech_mask=extremize(smoothed_normalized_power, 0.2)\n",
    "    speech_segments=mask_boundaries(speech_mask)+spec_start\n",
    "    spec_to_words=allocate_pred_to_speech_segments(prediction, speech_segments)\n",
    "    if len(spec_to_words)==0:\n",
    "        return None\n",
    "    segment_transcript = \\\n",
    "        [(spec1*dt_S, (spec2-spec1)*dt_S, word) \n",
    "        for spec1, spec2, word in spec_to_words]\n",
    "    return segment_transcript, timeline, \\\n",
    "           normalized_power, speech_mask, clip_audio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This in turn relies on a function to call the model to transcribe the audio into graphemes:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "def transcribe(C, model, audio):\n",
    "    fn='tmp.wav'\n",
    "    sf.write(fn, audio, C.sample_rate)\n",
    "    translations=model.transcribe(paths2audio_files=[fn], batch_size=1)\n",
    "    translation=translations[0]\n",
    "    translation=translation.split(' ')\n",
    "    translation=' '.join([x.strip() for x in translation if len(x)])\n",
    "    return translation.replace(\"\\u200c\",'')  # Just Pashto but required"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and a function to do the allocation of predicted text to speech segments:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "def align_seg_words(seg_words):\n",
    "    ([seg_start, seg_end], seg_wrds) = seg_words\n",
    "    seg_duration=seg_end-seg_start\n",
    "    n_seg_wrds=len(seg_wrds)\n",
    "    word_duration=seg_duration//n_seg_wrds\n",
    "    seg_duration, word_duration\n",
    "    seg_word_boundaries=np.hstack([np.linspace(seg_start, \\\n",
    "        seg_end-word_duration, n_seg_wrds).astype(int), [seg_end]])\n",
    "    seg_aligned_wrds=[(seg_word_boundaries[i], \n",
    "                    seg_word_boundaries[i+1], seg_wrds[i]) \n",
    "        for i in range(n_seg_wrds)]\n",
    "    return seg_aligned_wrds\n",
    "\n",
    "def align_segment_words(segment_words):\n",
    "    return [z for y in [align_seg_words(x) for x in segment_words] for z in y]\n",
    "\n",
    "def allocate_pred_to_speech_segments(prediction, speech_segments):\n",
    "    pred_words=prediction.split(' ')\n",
    "    n_words=len(pred_words)\n",
    "    if n_words==0:\n",
    "        return []\n",
    "    segment_durations=np.diff(speech_segments)\n",
    "    speech_duration=segment_durations.sum()\n",
    "    segment_allocation=n_words*segment_durations/speech_duration\n",
    "    words_per_segment=np.round(segment_allocation).T.astype(int)[0]\n",
    "    # If count is under then add missing word to longest segment\n",
    "    words_per_segment[np.where(words_per_segment==\\\n",
    "                 words_per_segment.max())[0][0]] \\\n",
    "                 += n_words-words_per_segment.sum()\n",
    "    word_segment_boundaries=np.cumsum(np.hstack([[0],\\\n",
    "                   words_per_segment]))\n",
    "    segment_words=list(zip(speech_segments.tolist(),\n",
    "          [pred_words[word_segment_boundaries[i]:\\\n",
    "                  word_segment_boundaries[i+1]]\n",
    "           for i in range(len(words_per_segment))]))\n",
    "    return align_segment_words(segment_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Features that were the most novel or unusual and/or led to the biggest improvements in system performance\n",
    "\n",
    "The open source NVidia NeMo project claims that the Quartz 15x5 model is novel in the sense of using 10X fewer parameters than other models in the Encoder/Decoder class [8]."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It did not train well in the time we had available and performed poorly. This may be the fault of over-extreme or poorly thought out augmentation choices that we made.  We did not understand the model well enough to attempt any configuration changes to increase the learning capacity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### System configuration\n",
    "\n",
    "Our system configuration was\n",
    "\n",
    "* Intel core i9 processor\n",
    "* 3TB SSD\n",
    "* 64GB RAM\n",
    "* NVidia RTX 2080TI GPU with 11GB of VRAM\n",
    "* Ubuntu 20.04LTS operating system\n",
    "* Python 3.7.9"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Minimal required hardware specs to run your system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluation required less than 1GB of GPU VRAM and less than 5GB of CPU RAM.  Evaluation was fast, less than 5 minutes for a DEV or EVAL run."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Minimal required time and amount of data to train/tune your system\n",
    "\n",
    "In training we were able to keep the GPU about 88% loaded and using 11.2 out of 11.6GB of available RAM.   With the augmented data we were not converging to a training error loss of less than 50 with over 48 hours of training.  This may be due to bad choice of augmentations, and also to upsampling to 16KHz when 8KHz was the source level, which may result in unnecessary artifacts. About 24GB of RAM was used during training.  We allocated 8 cores for data loading.  The 8 cores were periodically busy to 100% but in general stayed in a lower range."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Diagram giving a visual representation of our system’s workflow\n",
    "\n",
    "### Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![training](training_flow.gv.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![training](eval_flow.gv.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "[1] https://github.com/NVIDIA/NeMo\n",
    "\n",
    "[2] https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/01_ASR_with_NeMo.ipynb\n",
    "\n",
    "[3] https://arxiv.org/abs/1910.10261\n",
    "\n",
    "[4] https://github.com/NVIDIA/NeMo/blob/main/examples/asr/conf/config\n",
    "\n",
    "[5] https://www.openslr.org/3/\n",
    "\n",
    "[6] https://docs.nvidia.com/deeplearning/nemo/neural_mod_bp_guide/index.html#data_augmentation\n",
    "\n",
    "[7] https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/05_Online_Noise_Augmentation.ipynb\n",
    "\n",
    "[8] https://arxiv.org/pdf/1910.10261.pdf"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "nemo",
   "language": "python",
   "name": "nemo"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}