micro_speech_uint8.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "micro_speech_uint8.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S9eLyYdy9t4I"
      },
      "source": [
        "# Automated B&Q on trained trained speech model\n",
        "\n",
        "This notebooks allows you to apply B&Q compression technique on the pre-trained speech command \"yes,no\" recognition model. \n",
        "\n",
        "The B&Q automation process consists of the following compression and decompression steps: \n",
        "\n",
        "Compression: \n",
        "1. Extract the original fully connected (FC) weights from uint8 tflite file to original_weights.bin\n",
        "2. Compress the extracted FC weights and generate the B&Q binary file. Automated B&Q to find the appropriate bin values without losing accuracy. \n",
        "3. Create a compressed .tflite file using the B&Q binary file. \n",
        "\n",
        "Decompression:\n",
        "4. Extract the B&Q binary file from the compressed tflite file \n",
        "5. Extract the original weights from the B&Q binary file\n",
        "6. Create the original.tflite file using the original weights binary\n",
        "7. Test the accurcy of the decompressed model \n",
        "\n",
        "This notebook allows you to load a model file with uint8 weights, test the uint8 model accuracy and modify the values using B&Q, and then observe the effect on the model's accuracy."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LN0slo1N-dMT"
      },
      "source": [
        "## Software installation\n",
        "\n",
        "To be able to modify TensorFlow Lite files we need flatbuffers and TF libraries installed."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v9eegi_vtxW4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f0b7b8ab-6847-4754-deea-5509f4ab5c03"
      },
      "source": [
        "%tensorflow_version 1.x\n",
        "import tensorflow as tf\n",
        "\n",
        "# Build and install the Flatbuffer compiler.\n",
        "%cd /content/\n",
        "!rm -rf flatbuffers\n",
        "!git clone https://github.com/google/flatbuffers\n",
        "%cd flatbuffers\n",
        "!git checkout 37a5dee10525cc58908aff99b0aa073bf91b9ba6\n",
        "!cmake -G \"Unix Makefiles\" -DCMAKE_BUILD_TYPE=Release\n",
        "!make\n",
        "!cp flatc /usr/local/bin/\n",
        "%cd /content/\n",
        "!rm -rf tensorflow\n",
        "!git clone --depth 1 https://github.com/tensorflow/tensorflow\n",
        "!flatc --python --gen-object-api tensorflow/tensorflow/lite/schema/schema_v3.fbs"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "TensorFlow 1.x selected.\n",
            "/content\n",
            "Cloning into 'flatbuffers'...\n",
            "remote: Enumerating objects: 75, done.\u001b[K\n",
            "remote: Counting objects: 100% (75/75), done.\u001b[K\n",
            "remote: Compressing objects: 100% (66/66), done.\u001b[K\n",
            "remote: Total 20180 (delta 23), reused 21 (delta 8), pack-reused 20105\u001b[K\n",
            "Receiving objects: 100% (20180/20180), 12.55 MiB | 19.18 MiB/s, done.\n",
            "Resolving deltas: 100% (14048/14048), done.\n",
            "/content/flatbuffers\n",
            "Note: checking out '37a5dee10525cc58908aff99b0aa073bf91b9ba6'.\n",
            "\n",
            "You are in 'detached HEAD' state. You can look around, make experimental\n",
            "changes and commit them, and you can discard any commits you make in this\n",
            "state without impacting any branches by performing another checkout.\n",
            "\n",
            "If you want to create a new branch to retain commits you create, you may\n",
            "do so (now or later) by using -b with the checkout command again. Example:\n",
            "\n",
            "  git checkout -b <new-branch-name>\n",
            "\n",
            "HEAD is now at 37a5dee1 Code cleanup + updates test and readme (#6004)\n",
            "-- The C compiler identification is GNU 7.5.0\n",
            "-- The CXX compiler identification is GNU 7.5.0\n",
            "-- Check for working C compiler: /usr/bin/cc\n",
            "-- Check for working C compiler: /usr/bin/cc -- works\n",
            "-- Detecting C compiler ABI info\n",
            "-- Detecting C compiler ABI info - done\n",
            "-- Detecting C compile features\n",
            "-- Detecting C compile features - done\n",
            "-- Check for working CXX compiler: /usr/bin/c++\n",
            "-- Check for working CXX compiler: /usr/bin/c++ -- works\n",
            "-- Detecting CXX compiler ABI info\n",
            "-- Detecting CXX compiler ABI info - done\n",
            "-- Detecting CXX compile features\n",
            "-- Detecting CXX compile features - done\n",
            "-- Looking for strtof_l\n",
            "-- Looking for strtof_l - found\n",
            "-- Looking for strtoull_l\n",
            "-- Looking for strtoull_l - found\n",
            "-- `tests/monster_test.fbs`: add generation of C++ code with '--no-includes;--gen-compare'\n",
            "-- `tests/monster_test.fbs`: add generation of binary (.bfbs) schema\n",
            "-- `tests/namespace_test/namespace_test1.fbs`: add generation of C++ code with '--no-includes;--gen-compare'\n",
            "-- `tests/namespace_test/namespace_test2.fbs`: add generation of C++ code with '--no-includes;--gen-compare'\n",
            "-- `tests/union_vector/union_vector.fbs`: add generation of C++ code with '--no-includes;--gen-compare'\n",
            "-- `tests/native_type_test.fbs`: add generation of C++ code with ''\n",
            "-- `tests/arrays_test.fbs`: add generation of C++ code with '--scoped-enums;--gen-compare'\n",
            "-- `tests/arrays_test.fbs`: add generation of binary (.bfbs) schema\n",
            "-- `tests/monster_test.fbs`: add generation of C++ embedded binary schema code with '--no-includes;--gen-compare'\n",
            "-- `tests/monster_extra.fbs`: add generation of C++ code with '--no-includes;--gen-compare'\n",
            "-- `samples/monster.fbs`: add generation of C++ code with '--no-includes;--gen-compare'\n",
            "-- `samples/monster.fbs`: add generation of binary (.bfbs) schema\n",
            "Proceeding with version: 1.12.0.91\n",
            "-- Configuring done\n",
            "-- Generating done\n",
            "-- Build files have been written to: /content/flatbuffers\n",
            "\u001b[35m\u001b[1mScanning dependencies of target flatbuffers\u001b[0m\n",
            "[  1%] \u001b[32mBuilding CXX object CMakeFiles/flatbuffers.dir/src/idl_parser.cpp.o\u001b[0m\n",
            "[  2%] \u001b[32mBuilding CXX object CMakeFiles/flatbuffers.dir/src/idl_gen_text.cpp.o\u001b[0m\n",
            "[  3%] \u001b[32mBuilding CXX object CMakeFiles/flatbuffers.dir/src/reflection.cpp.o\u001b[0m\n",
            "[  4%] \u001b[32mBuilding CXX object CMakeFiles/flatbuffers.dir/src/util.cpp.o\u001b[0m\n",
            "[  5%] \u001b[32m\u001b[1mLinking CXX static library libflatbuffers.a\u001b[0m\n",
            "[  5%] Built target flatbuffers\n",
            "\u001b[35m\u001b[1mScanning dependencies of target flatc\u001b[0m\n",
            "[  7%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_parser.cpp.o\u001b[0m\n",
            "[  8%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_text.cpp.o\u001b[0m\n",
            "[  9%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/reflection.cpp.o\u001b[0m\n",
            "[ 10%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/util.cpp.o\u001b[0m\n",
            "[ 11%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_cpp.cpp.o\u001b[0m\n",
            "[ 12%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_csharp.cpp.o\u001b[0m\n",
            "[ 14%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_dart.cpp.o\u001b[0m\n",
            "[ 15%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_kotlin.cpp.o\u001b[0m\n",
            "[ 16%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_go.cpp.o\u001b[0m\n",
            "[ 17%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_java.cpp.o\u001b[0m\n",
            "[ 18%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_js_ts.cpp.o\u001b[0m\n",
            "[ 20%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_php.cpp.o\u001b[0m\n",
            "[ 21%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_python.cpp.o\u001b[0m\n",
            "[ 22%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_lobster.cpp.o\u001b[0m\n",
            "[ 23%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_lua.cpp.o\u001b[0m\n",
            "[ 24%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_rust.cpp.o\u001b[0m\n",
            "[ 25%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_fbs.cpp.o\u001b[0m\n",
            "[ 27%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_grpc.cpp.o\u001b[0m\n",
            "[ 28%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_json_schema.cpp.o\u001b[0m\n",
            "[ 29%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_swift.cpp.o\u001b[0m\n",
            "[ 30%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/flatc.cpp.o\u001b[0m\n",
            "[ 31%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/flatc_main.cpp.o\u001b[0m\n",
            "[ 32%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/code_generators.cpp.o\u001b[0m\n",
            "[ 34%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/grpc/src/compiler/cpp_generator.cc.o\u001b[0m\n",
            "[ 35%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/grpc/src/compiler/go_generator.cc.o\u001b[0m\n",
            "[ 36%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/grpc/src/compiler/java_generator.cc.o\u001b[0m\n",
            "[ 37%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/grpc/src/compiler/python_generator.cc.o\u001b[0m\n",
            "[ 38%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/grpc/src/compiler/swift_generator.cc.o\u001b[0m\n",
            "[ 40%] \u001b[32m\u001b[1mLinking CXX executable flatc\u001b[0m\n",
            "[ 40%] Built target flatc\n",
            "\u001b[35m\u001b[1mScanning dependencies of target generated_code\u001b[0m\n",
            "[ 41%] \u001b[34m\u001b[1mRun generation: 'samples/monster.bfbs'\u001b[0m\n",
            "[ 42%] \u001b[34m\u001b[1mRun generation: 'tests/monster_test_generated.h'\u001b[0m\n",
            "[ 43%] \u001b[34m\u001b[1mRun generation: 'tests/monster_test.bfbs'\u001b[0m\n",
            "[ 44%] \u001b[34m\u001b[1mRun generation: 'tests/namespace_test/namespace_test1_generated.h'\u001b[0m\n",
            "[ 45%] \u001b[34m\u001b[1mRun generation: 'tests/namespace_test/namespace_test2_generated.h'\u001b[0m\n",
            "[ 47%] \u001b[34m\u001b[1mRun generation: 'tests/union_vector/union_vector_generated.h'\u001b[0m\n",
            "[ 48%] \u001b[34m\u001b[1mRun generation: 'tests/native_type_test_generated.h'\u001b[0m\n",
            "[ 49%] \u001b[34m\u001b[1mRun generation: 'tests/arrays_test_generated.h'\u001b[0m\n",
            "[ 50%] \u001b[34m\u001b[1mRun generation: 'tests/arrays_test.bfbs'\u001b[0m\n",
            "[ 51%] \u001b[34m\u001b[1mRun generation: 'tests/monster_test_bfbs_generated.h'\u001b[0m\n",
            "[ 52%] \u001b[34m\u001b[1mRun generation: 'tests/monster_extra_generated.h'\u001b[0m\n",
            "[ 54%] \u001b[34m\u001b[1mRun generation: 'samples/monster_generated.h'\u001b[0m\n",
            "[ 55%] \u001b[34m\u001b[1mAll generated files were updated.\u001b[0m\n",
            "[ 55%] Built target generated_code\n",
            "\u001b[35m\u001b[1mScanning dependencies of target flattests\u001b[0m\n",
            "[ 56%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/src/idl_parser.cpp.o\u001b[0m\n",
            "[ 57%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/src/idl_gen_text.cpp.o\u001b[0m\n",
            "[ 58%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/src/reflection.cpp.o\u001b[0m\n",
            "[ 60%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/src/util.cpp.o\u001b[0m\n",
            "[ 61%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/src/idl_gen_fbs.cpp.o\u001b[0m\n",
            "[ 62%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/tests/test.cpp.o\u001b[0m\n",
            "[ 63%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/tests/test_assert.cpp.o\u001b[0m\n",
            "[ 64%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/tests/test_builder.cpp.o\u001b[0m\n",
            "[ 65%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/tests/native_type_test_impl.cpp.o\u001b[0m\n",
            "[ 67%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/src/code_generators.cpp.o\u001b[0m\n",
            "[ 68%] \u001b[32m\u001b[1mLinking CXX executable flattests\u001b[0m\n",
            "[ 77%] Built target flattests\n",
            "\u001b[35m\u001b[1mScanning dependencies of target flatsamplebinary\u001b[0m\n",
            "[ 78%] \u001b[32mBuilding CXX object CMakeFiles/flatsamplebinary.dir/samples/sample_binary.cpp.o\u001b[0m\n",
            "[ 80%] \u001b[32m\u001b[1mLinking CXX executable flatsamplebinary\u001b[0m\n",
            "[ 81%] Built target flatsamplebinary\n",
            "\u001b[35m\u001b[1mScanning dependencies of target flathash\u001b[0m\n",
            "[ 82%] \u001b[32mBuilding CXX object CMakeFiles/flathash.dir/src/flathash.cpp.o\u001b[0m\n",
            "[ 83%] \u001b[32m\u001b[1mLinking CXX executable flathash\u001b[0m\n",
            "[ 83%] Built target flathash\n",
            "\u001b[35m\u001b[1mScanning dependencies of target flatsampletext\u001b[0m\n",
            "[ 84%] \u001b[32mBuilding CXX object CMakeFiles/flatsampletext.dir/src/idl_parser.cpp.o\u001b[0m\n",
            "[ 85%] \u001b[32mBuilding CXX object CMakeFiles/flatsampletext.dir/src/idl_gen_text.cpp.o\u001b[0m\n",
            "[ 87%] \u001b[32mBuilding CXX object CMakeFiles/flatsampletext.dir/src/reflection.cpp.o\u001b[0m\n",
            "[ 88%] \u001b[32mBuilding CXX object CMakeFiles/flatsampletext.dir/src/util.cpp.o\u001b[0m\n",
            "[ 89%] \u001b[32mBuilding CXX object CMakeFiles/flatsampletext.dir/samples/sample_text.cpp.o\u001b[0m\n",
            "[ 90%] \u001b[32m\u001b[1mLinking CXX executable flatsampletext\u001b[0m\n",
            "[ 91%] Built target flatsampletext\n",
            "\u001b[35m\u001b[1mScanning dependencies of target flatsamplebfbs\u001b[0m\n",
            "[ 92%] \u001b[32mBuilding CXX object CMakeFiles/flatsamplebfbs.dir/src/idl_parser.cpp.o\u001b[0m\n",
            "[ 94%] \u001b[32mBuilding CXX object CMakeFiles/flatsamplebfbs.dir/src/idl_gen_text.cpp.o\u001b[0m\n",
            "[ 95%] \u001b[32mBuilding CXX object CMakeFiles/flatsamplebfbs.dir/src/reflection.cpp.o\u001b[0m\n",
            "[ 96%] \u001b[32mBuilding CXX object CMakeFiles/flatsamplebfbs.dir/src/util.cpp.o\u001b[0m\n",
            "[ 97%] \u001b[32mBuilding CXX object CMakeFiles/flatsamplebfbs.dir/samples/sample_bfbs.cpp.o\u001b[0m\n",
            "[ 98%] \u001b[32m\u001b[1mLinking CXX executable flatsamplebfbs\u001b[0m\n",
            "[100%] Built target flatsamplebfbs\n",
            "/content\n",
            "Cloning into 'tensorflow'...\n",
            "remote: Enumerating objects: 23770, done.\u001b[K\n",
            "remote: Counting objects: 100% (23770/23770), done.\u001b[K\n",
            "remote: Compressing objects: 100% (17489/17489), done.\u001b[K\n",
            "remote: Total 23770 (delta 8782), reused 10979 (delta 5712), pack-reused 0\u001b[K\n",
            "Receiving objects: 100% (23770/23770), 56.81 MiB | 12.00 MiB/s, done.\n",
            "Resolving deltas: 100% (8782/8782), done.\n",
            "Checking out files: 100% (24302/24302), done.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GZMwDesb-xoE"
      },
      "source": [
        "## Speech Evaluation Setup\n",
        "\n",
        "To evaluate the accuracy of the speech model we need to load a test data set and some utility classes to read the files and convert them into the right input form for the network.\n",
        "\n",
        "The full dataset is several gigabytes in size, so it may take a few minutes to download."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oUdptxthqXq0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "55f3407f-f726-4a94-e6d6-bc357d786dc4"
      },
      "source": [
        "%cd /content/"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V4QSItXgum0N",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "96c56b35-88ec-4b0e-e5d1-a0a64805c79e"
      },
      "source": [
        "!pip install flatbuffers\n",
        "import flatbuffers\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import pprint\n",
        "import re\n",
        "import sys\n",
        "\n",
        "# This hackery allows us to import the Python files we've just generated.\n",
        "sys.path.append(\"/content/tflite/\")\n",
        "import Model\n",
        "\n",
        "sys.path.append(\"/content/tensorflow/tensorflow/examples/speech_commands/\")\n",
        "import input_data\n",
        "import models\n",
        "\n",
        "# A comma-delimited list of the words you want to train for.\n",
        "# The options are: yes,no,up,down,left,right,on,off,stop,go\n",
        "# All the other words will be used to train an \"unknown\" label and silent\n",
        "# audio data with no spoken words will be used to train a \"silence\" label.\n",
        "WANTED_WORDS = \"yes,no\"\n",
        "\n",
        "# The number of steps and learning rates can be specified as comma-separated\n",
        "# lists to define the rate at each stage. For example,\n",
        "# TRAINING_STEPS=12000,3000 and LEARNING_RATE=0.001,0.0001\n",
        "# will run 12,000 training loops in total, with a rate of 0.001 for the first\n",
        "# 8,000, and 0.0001 for the final 3,000.\n",
        "TRAINING_STEPS = \"12000,3000\"\n",
        "LEARNING_RATE = \"0.001,0.0001\"\n",
        "\n",
        "# Calculate the total number of steps, which is used to identify the checkpoint\n",
        "# file name.\n",
        "TOTAL_STEPS = str(sum(map(lambda string: int(string), TRAINING_STEPS.split(\",\"))))\n",
        "\n",
        "# Calculate the percentage of 'silence' and 'unknown' training samples required\n",
        "# to ensure that we have equal number of samples for each label.\n",
        "number_of_labels = WANTED_WORDS.count(',') + 1\n",
        "number_of_total_labels = number_of_labels + 2 # for 'silence' and 'unknown' label\n",
        "equal_percentage_of_training_samples = int(100.0/(number_of_total_labels))\n",
        "SILENT_PERCENTAGE = equal_percentage_of_training_samples\n",
        "UNKNOWN_PERCENTAGE = equal_percentage_of_training_samples\n",
        "\n",
        "# Constants which are shared during training and inference\n",
        "PREPROCESS = 'micro'\n",
        "WINDOW_STRIDE =20\n",
        "MODEL_ARCHITECTURE = 'tiny_conv' # Other options include: single_fc, conv,\n",
        "                      # low_latency_conv, low_latency_svdf, tiny_embedding_conv\n",
        "\n",
        "# Constants used during training only\n",
        "VERBOSITY = 'WARN'\n",
        "EVAL_STEP_INTERVAL = '1000'\n",
        "SAVE_STEP_INTERVAL = '1000'\n",
        "\n",
        "# Constants for training directories and filepaths\n",
        "DATASET_DIR =  'dataset/'\n",
        "LOGS_DIR = 'logs/'\n",
        "TRAIN_DIR = 'train/' # for training checkpoints and other files.\n",
        "\n",
        "SAMPLE_RATE = 16000\n",
        "CLIP_DURATION_MS = 1000\n",
        "WINDOW_SIZE_MS = 30.0\n",
        "FEATURE_BIN_COUNT = 40\n",
        "BACKGROUND_FREQUENCY = 0.8\n",
        "BACKGROUND_VOLUME_RANGE = 0.1\n",
        "TIME_SHIFT_MS = 100.0\n",
        "\n",
        "DATA_URL = 'https://storage.googleapis.com/download.tensorflow.org/data/speech_commands_v0.02.tar.gz'\n",
        "VALIDATION_PERCENTAGE = 10\n",
        "TESTING_PERCENTAGE = 10\n",
        "\n",
        "model_settings = models.prepare_model_settings(\n",
        "    len(input_data.prepare_words_list(WANTED_WORDS.split(','))),\n",
        "    SAMPLE_RATE, CLIP_DURATION_MS, WINDOW_SIZE_MS,\n",
        "    WINDOW_STRIDE, FEATURE_BIN_COUNT, PREPROCESS)\n",
        "audio_processor = input_data.AudioProcessor(\n",
        "    DATA_URL, DATASET_DIR,\n",
        "    SILENT_PERCENTAGE, UNKNOWN_PERCENTAGE,\n",
        "    WANTED_WORDS.split(','), VALIDATION_PERCENTAGE,\n",
        "    TESTING_PERCENTAGE, model_settings, LOGS_DIR)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: flatbuffers in /usr/local/lib/python3.6/dist-packages (1.12)\n",
            ">> Downloading speech_commands_v0.02.tar.gz 99.9%"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8p_5vLNQ_sFF"
      },
      "source": [
        "## Utility Functions\n",
        "\n",
        "To load, save, and evaluate models we need some helper functions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "acpsc72lvKRW"
      },
      "source": [
        "def load_model_from_file(model_filename):\n",
        "  with open(model_filename, \"rb\") as file:\n",
        "    buffer_data = file.read()\n",
        "  model_obj = Model.Model.GetRootAsModel(buffer_data, 0)\n",
        "  model = Model.ModelT.InitFromObj(model_obj)\n",
        "  return model\n",
        "\n",
        "def save_model_to_file(model, model_filename):\n",
        "  builder = flatbuffers.Builder(1024)\n",
        "  model_offset = model.Pack(builder)\n",
        "  builder.Finish(model_offset, file_identifier=b'TFL3')\n",
        "  model_data = builder.Output()\n",
        "  with open(model_filename, 'wb') as out_file:\n",
        "    out_file.write(model_data)\n",
        "\n",
        "def test_model_accuracy(model_filename):\n",
        "  with tf.Session() as sess:\n",
        "    test_data, test_labels = audio_processor.get_data(\n",
        "        -1, 0, model_settings, 0, 0,\n",
        "        0, 'testing', sess)\n",
        "\n",
        "  interpreter = tf.lite.Interpreter(model_filename)\n",
        "  interpreter.allocate_tensors()\n",
        "\n",
        "  input_index = interpreter.get_input_details()[0][\"index\"]\n",
        "\n",
        "  output_index = interpreter.get_output_details()[0][\"index\"]\n",
        "  model_output = interpreter.tensor(output_index)\n",
        "\n",
        "  correct_predictions = 0\n",
        "  for i in range(len(test_data)):\n",
        "    current_input = test_data[i]\n",
        "    current_label = test_labels[i]\n",
        "    flattened_input = np.array(current_input.flatten(), dtype=np.uint8).reshape(1, 49,40,1)\n",
        "    interpreter.set_tensor(input_index, flattened_input)\n",
        "    interpreter.invoke()\n",
        "    top_prediction = model_output()[0].argmax()\n",
        "    if top_prediction == current_label:\n",
        "      correct_predictions += 1\n",
        "\n",
        "  #print('Accuracy is %f%% (N=%d)' % ((correct_predictions * 100) / len(test_data), len(test_data)))\n",
        "  return (correct_predictions * 100) / len(test_data)"
      ],
      "execution_count": 30,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vtNYC984AILB"
      },
      "source": [
        "## Download a trained model\n",
        "\n",
        "This pulls down a trained model from the speech commands. It includes two files, the first is the original pb model in float and the second has weights stored in eight bits using post-training quantization."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jym3ab1Lk7ps"
      },
      "source": [
        "# Test the uint8 model\n",
        "Run the below code to identify the uint8 inference accuracy before B&Q compression. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XCroVFTllIfH",
        "outputId": "a9893d0e-4318-4972-a550-b5d0f88e35e3"
      },
      "source": [
        "inital_accuracy = test_model_accuracy('/content/model.tflite')\n",
        "print(\"Accuracy is:\", inital_accuracy)"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy is: 91.42394822006473\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mr5y_gHlkq10"
      },
      "source": [
        "# **B&Q applied on the trained saved model**\n",
        "The below script runs multiple bins for B&Q and tests inference performance. \n",
        "To start the automation process, run the below block. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sI67FSvfn3yO"
      },
      "source": [
        "# B&Q automation using sensitivity analysis\n",
        "\n",
        "This automation code performs B&Q with a inital 4 bins. The goal is to find a model which has least accuracy perturbation. \n",
        "The sensitivity code perturbs each bin with 20% of the original value and identifies the most sensitive bin. Based on that, each sensitive bin is spilt in the middle and bin values are updated based on these bin rangeValues. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ayHauOjjn3Iq",
        "outputId": "5daf3718-20fa-4f9b-ca6d-fe36ddb74783"
      },
      "source": [
        "#load the model \n",
        "model = load_model_from_file('/content/model.tflite')\n",
        "max_bins = '16'\n",
        "curr_acc = inital_accuracy - 3\n",
        "\n",
        "top_acc = curr_acc\n",
        "top_acc_bins = []\n",
        "\n",
        "#load the inital layer parameters of the FC layer\n",
        "for buffer in model.buffers:\n",
        "  if buffer.data is not None and len(buffer.data) > 1024:\n",
        "    original_weights = np.frombuffer(buffer.data, dtype=np.uint8)\n",
        "    v2 = np.add(original_weights,0)\n",
        "    v2_min = v2.min()\n",
        "    v2_max = v2.max()\n",
        "\n",
        "RangeValues = [v2_min, np.mean(v2) - np.std(v2), np.mean(v2), np.mean(v2) + np.std(v2), v2_max]\n",
        "while (curr_acc <= inital_accuracy - 0.2):\n",
        "  model = load_model_from_file('/content/model.tflite')\n",
        "  for buffer in model.buffers:\n",
        "    if buffer.data is not None and len(buffer.data) > 1024:\n",
        "      original_weights = np.frombuffer(buffer.data, dtype=np.uint8)\n",
        "      v2 = np.add(original_weights,0)      \n",
        "\n",
        "      for x in range(len(RangeValues) - 1):\n",
        "        indices = np.where(np.logical_and(v2>=RangeValues[x], v2<=RangeValues[x+1]))\n",
        "        v2[indices] = np.uint8((RangeValues[x] + RangeValues[x+1])/2)\n",
        "\n",
        "      buffer.data = v2\n",
        "\n",
        "  save_model_to_file(model, '/content/speech_commands_model_modified.tflite')\n",
        "  curr_acc = test_model_accuracy('/content/speech_commands_model_modified.tflite')\n",
        "  print(\"Accuracy for range values:\", curr_acc, RangeValues)\n",
        "\n",
        "  if curr_acc >= top_acc:\n",
        "    top_acc = curr_acc\n",
        "    top_bins = RangeValues\n",
        "    save_model_to_file(model, '/content/speech_commands_model_top_acc_model.tflite')\n",
        "\n",
        "  #perturbation code to identify the sensitive bin\n",
        "  bin_acc = []\n",
        "  for times in range(len(RangeValues) - 1):\n",
        "    model = load_model_from_file('/content/model.tflite')\n",
        "    for buffer in model.buffers:\n",
        "      if buffer.data is not None and len(buffer.data) > 1024:\n",
        "        original_weights = np.frombuffer(buffer.data, dtype=np.uint8)\n",
        "        v2 = np.add(original_weights,0)      \n",
        "\n",
        "        indices = np.where(np.logical_and(v2>=RangeValues[times], v2<=RangeValues[times+1]))\n",
        "        v2[indices] = np.uint8(v2[indices] + 0.2*v2[indices])\n",
        "\n",
        "        buffer.data = v2\n",
        "\n",
        "    save_model_to_file(model, '/content/speech_commands_model_modified.tflite')\n",
        "    bin_acc.append(test_model_accuracy('/content/speech_commands_model_modified.tflite'))\n",
        "\n",
        "  print(\"The most sensitive bin here is:\", bin_acc, np.array(bin_acc).argmin())\n",
        "  to_modify = np.array(bin_acc).argmin()\n",
        "  middle_bin = (RangeValues[to_modify] + RangeValues[to_modify+1]) / 2\n",
        "  RangeValues.insert(to_modify+1, middle_bin)\n",
        "  print(\"New Range Values:\", RangeValues, len(RangeValues))\n",
        "\n"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy for range values: 90.45307443365695 [1, 97.78707476597417, 128.78125, 159.77542523402582, 255]\n",
            "The most sensitive bin here is: [91.10032362459548, 90.85760517799353, 91.2621359223301, 89.07766990291262] 3\n",
            "New Range Values: [1, 97.78707476597417, 128.78125, 159.77542523402582, 207.3877126170129, 255] 6\n",
            "Accuracy for range values: 90.93851132686085 [1, 97.78707476597417, 128.78125, 159.77542523402582, 207.3877126170129, 255]\n",
            "The most sensitive bin here is: [90.93851132686085, 90.85760517799353, 91.2621359223301, 90.77669902912622, 86.24595469255664] 4\n",
            "New Range Values: [1, 97.78707476597417, 128.78125, 159.77542523402582, 207.3877126170129, 231.19385630850644, 255] 7\n",
            "Accuracy for range values: 90.61488673139158 [1, 97.78707476597417, 128.78125, 159.77542523402582, 207.3877126170129, 231.19385630850644, 255]\n",
            "The most sensitive bin here is: [91.18122977346279, 90.53398058252426, 91.18122977346279, 90.6957928802589, 90.53398058252426, 91.42394822006473] 1\n",
            "New Range Values: [1, 97.78707476597417, 113.28416238298709, 128.78125, 159.77542523402582, 207.3877126170129, 231.19385630850644, 255] 8\n",
            "Accuracy for range values: 90.53398058252426 [1, 97.78707476597417, 113.28416238298709, 128.78125, 159.77542523402582, 207.3877126170129, 231.19385630850644, 255]\n",
            "The most sensitive bin here is: [91.18122977346279, 91.99029126213593, 91.50485436893204, 91.2621359223301, 90.53398058252426, 90.45307443365695, 91.2621359223301] 5\n",
            "New Range Values: [1, 97.78707476597417, 113.28416238298709, 128.78125, 159.77542523402582, 207.3877126170129, 219.2907844627597, 231.19385630850644, 255] 9\n",
            "Accuracy for range values: 90.45307443365695 [1, 97.78707476597417, 113.28416238298709, 128.78125, 159.77542523402582, 207.3877126170129, 219.2907844627597, 231.19385630850644, 255]\n",
            "The most sensitive bin here is: [90.93851132686085, 91.74757281553399, 91.2621359223301, 91.2621359223301, 90.6957928802589, 91.58576051779936, 91.42394822006473, 91.18122977346279] 4\n",
            "New Range Values: [1, 97.78707476597417, 113.28416238298709, 128.78125, 159.77542523402582, 183.58156892551938, 207.3877126170129, 219.2907844627597, 231.19385630850644, 255] 10\n",
            "Accuracy for range values: 91.18122977346279 [1, 97.78707476597417, 113.28416238298709, 128.78125, 159.77542523402582, 183.58156892551938, 207.3877126170129, 219.2907844627597, 231.19385630850644, 255]\n",
            "The most sensitive bin here is: [91.10032362459548, 91.99029126213593, 91.42394822006473, 91.10032362459548, 91.10032362459548, 91.34304207119742, 91.50485436893204, 91.58576051779936, 91.42394822006473] 0\n",
            "New Range Values: [1, 49.393537382987084, 97.78707476597417, 113.28416238298709, 128.78125, 159.77542523402582, 183.58156892551938, 207.3877126170129, 219.2907844627597, 231.19385630850644, 255] 11\n",
            "Accuracy for range values: 90.93851132686085 [1, 49.393537382987084, 97.78707476597417, 113.28416238298709, 128.78125, 159.77542523402582, 183.58156892551938, 207.3877126170129, 219.2907844627597, 231.19385630850644, 255]\n",
            "The most sensitive bin here is: [91.50485436893204, 91.2621359223301, 92.07119741100324, 91.50485436893204, 90.93851132686085, 91.10032362459548, 91.34304207119742, 91.58576051779936, 91.58576051779936, 91.2621359223301] 4\n",
            "New Range Values: [1, 49.393537382987084, 97.78707476597417, 113.28416238298709, 128.78125, 144.2783376170129, 159.77542523402582, 183.58156892551938, 207.3877126170129, 219.2907844627597, 231.19385630850644, 255] 12\n",
            "Accuracy for range values: 91.42394822006473 [1, 49.393537382987084, 97.78707476597417, 113.28416238298709, 128.78125, 144.2783376170129, 159.77542523402582, 183.58156892551938, 207.3877126170129, 219.2907844627597, 231.19385630850644, 255]\n",
            "The most sensitive bin here is: [91.50485436893204, 91.2621359223301, 91.99029126213593, 91.34304207119742, 90.37216828478964, 91.58576051779936, 90.93851132686085, 91.34304207119742, 91.50485436893204, 91.42394822006473, 91.34304207119742] 4\n",
            "New Range Values: [1, 49.393537382987084, 97.78707476597417, 113.28416238298709, 128.78125, 136.52979380850644, 144.2783376170129, 159.77542523402582, 183.58156892551938, 207.3877126170129, 219.2907844627597, 231.19385630850644, 255] 13\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xoTWjfYT0BCj",
        "outputId": "239fef1c-3600-41f2-f0c9-ed350dd3e086"
      },
      "source": [
        "print(\"Reported top_accuracy and infered test accuracy :\", top_acc, test_model_accuracy('/content/speech_commands_model_top_acc_model.tflite'))\n",
        "print(\"The RangeValues are:\", RangeValues)\n",
        "print(\"total_bins\", len(RangeValues) - 1)\n",
        "print(\"The model is saved in /content/speech_commands_model_top_acc_model.tflite\")\n"
      ],
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Reported top_accuracy and infered test accuracy : 91.42394822006473 91.34304207119742\n",
            "The RangeValues are: [1, 49.393537382987084, 97.78707476597417, 113.28416238298709, 128.78125, 136.52979380850644, 144.2783376170129, 159.77542523402582, 183.58156892551938, 207.3877126170129, 219.2907844627597, 231.19385630850644, 255]\n",
            "total_bins 12\n",
            "The model is saved in /content/speech_commands_model_top_acc_model.tflite\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GhbtzP8Kr7C6"
      },
      "source": [
        "model = load_model_from_file('/content/model.tflite')\n",
        "bins = '6'\n",
        "for buffer in model.buffers:\n",
        "  if buffer.data is not None and len(buffer.data) > 1024:\n",
        "    print(\"buffer.data:\", buffer)\n",
        "    original_weights = np.frombuffer(buffer.data, dtype=np.uint8)\n",
        "    v2 = np.add(original_weights,0)\n",
        "    v2_min = v2.min()\n",
        "    v2_max = v2.max()\n",
        "\n",
        "    print(np.mean(v2), np.std(v2))\n",
        "    \n",
        "    # with open('/content/weights_to_file.bin', 'wb') as out_file:\n",
        "    #   out_file.write(original_weights)\n",
        "    # print(len(buffer.data))\n",
        "\n",
        "    if bins == '2':\n",
        "       RangeValues = [v2_min, 0, v2_max]\n",
        "    elif bins == '4':\n",
        "      RangeValues = [v2_min, np.mean(v2) - np.std(v2), np.mean(v2), np.mean(v2) + np.std(v2), v2_max]\n",
        "    elif bins == '6':\n",
        "      RangeValues = [v2_min, (RangeValues[0] + RangeValues[1]) /2, np.mean(v2) - np.std(v2), np.mean(v2), np.mean(v2) + np.std(v2), \\\n",
        "                     (RangeValues[3] + RangeValues[4]) /2, v2_max]\n",
        "\n",
        "\n",
        "    for x in range(len(RangeValues) - 1):\n",
        "            indices = np.where(np.logical_and(v2>=RangeValues[x], v2<=RangeValues[x+1]))\n",
        "            v2[indices] = np.uint8((RangeValues[x] + RangeValues[x+1])/2)\n",
        "\n",
        "    print(np.unique(v2))\n",
        "\n",
        "    # print(munged_weights)\n",
        "    buffer.data = v2\n",
        "    print(buffer.data)\n",
        "    # with open('/content/bin_data', \"rb\") as file:\n",
        "    #   buffer_data = file.read()\n",
        "\n",
        "\n",
        "\n",
        "save_model_to_file(model, '/content/speech_commands_model_modified.tflite')\n",
        "test_model_accuracy('/content/speech_commands_model_modified.tflite')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EaYw9sjo4tko"
      },
      "source": [
        "model = load_model_from_file('/content/inception_v1_224_quant.tflite')\n",
        "\n",
        "for buffer in model.buffers:\n",
        "  if buffer.data is not None and len(buffer.data) > 32:\n",
        "    original_weights = np.frombuffer(buffer.data, dtype=np.uint8)\n",
        "    # print(original_weights)\n",
        "\n",
        "    # This is the line where the weights are altered.\n",
        "    # Try replacing it with your own version, for example:\n",
        "    munged_weights = np.add(original_weights, 1)\n",
        "    #munged_weights = np.round(original_weights * (1/0.02)) * 0.02\n",
        "    \n",
        "    # print(munged_weights)\n",
        "    buffer.data = munged_weights.tobytes()\n",
        "\n",
        "save_model_to_file(model, '/content/inception_modified.tflite')\n",
        "#test_model_accuracy('/content/speech_commands_model/speech_commands_model_modified.tflite')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i2jo_Gl4vpLx"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}