tutorials/12_decode_back_HMM-DNN_and_WFST.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Welcome to ExKaldi\n",
    "\n",
    "In this section, we will decode the test data based on HMM-DNN model and WFST graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import exkaldi\n",
    "\n",
    "import os\n",
    "dataDir = \"librispeech_dummy\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Restorage the posteiori probability of AM from file (generated in 11_train_DNN_acoustic_model_with_tensorflow)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "probFile = os.path.join(dataDir, \"exp\", \"train_DNN\", \"amp.npy\")\n",
    "\n",
    "prob = exkaldi.load_prob(probFile)\n",
    "\n",
    "prob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prob.subset(nHead=1).data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As above, this is naive output without log softmax activation function. We need do softmax.\n",
    "\n",
    "Exkaldi Numpy achivements have a method __.map(...)__ . We map a softmax function to all matrixs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prob = prob.map( lambda x: exkaldi.nn.log_softmax(x, axis=1) )\n",
    "\n",
    "prob.subset(nHead=1).data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then decode based on WFST. HCLG graph file and HMM model file have been generated (07_train_triphone_HMM-GMM_delta)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "HCLGFile = os.path.join(dataDir, \"exp\", \"train_delta\", \"graph\", \"HCLG.fst\")\n",
    "\n",
    "hmmFile = os.path.join(dataDir, \"exp\", \"train_delta\", \"final.mdl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And for convenience, prepare lexicons."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lexFile = os.path.join(dataDir, \"exp\", \"lexicons.lex\")\n",
    "\n",
    "lexicons = exkaldi.decode.graph.load_lex(lexFile)\n",
    "\n",
    "lexicons"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use __nn_decode__ function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lat = exkaldi.decode.wfst.nn_decode(prob, hmmFile, HCLGFile, symbolTable=lexicons(\"words\"))\n",
    "\n",
    "lat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "outDir = os.path.join(dataDir, \"exp\", \"train_DNN\", \"decode_test\")\n",
    "\n",
    "exkaldi.utils.make_dependent_dirs(outDir, False)\n",
    "\n",
    "lat.save( os.path.join(outDir,\"test.lat\") )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From lattice get 1-best result and score it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "refIntFile = os.path.join(dataDir, \"exp\", \"train_delta\", \"decode_test\", \"text.int\")\n",
    "\n",
    "for penalty in [0., 0.5, 1.0]:\n",
    "    for LMWT in range(10, 15):\n",
    "    \n",
    "        newLat = lat.add_penalty(penalty)\n",
    "        result = newLat.get_1best(lexicons(\"words\"), hmmFile, lmwt=LMWT, acwt=0.5)\n",
    "\n",
    "        score = exkaldi.decode.score.wer(ref=refIntFile, hyp=result, mode=\"present\")\n",
    "        \n",
    "        print(f\"Penalty {penalty}, LMWT {LMWT}: WER {score.WER}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In step 10_process_lattice_and_score, the best WER based on HMM-GMM model is about 135% and here is 107% in our experiment. So it is a truth that the performance of HMM-DNN got better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Up to here, the simple tutorial is over."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}