OpenVINO backend for Intel Architectures

[dkurt]

To test this branch,

• Clone repository

git clone -b openvino https://github.com/dkurt/bonito

• Follow https://github.com/dkurt/bonito/tree/openvino#developer-quickstart

[iiSeymour]

Thanks @dkurt - I have tested on a Skylake 8168 and see a >20X speed up over PyTorch on the CPU 🔥

Validation samples	mean	median	time	samples/s
PyTorch --device=cpu Intel(R) Xeon(R) Platinum 8168)	96.51%	97.04%	75.05	3.20E+04
OpenVINO --device=cpu Intel(R) Xeon(R) Platinum 8168	96.51%	97.04%	3.59	6.68E+05
PyTorch --device=cuda --half Nvidia(R) V100	96.51%	97.04%	1.07	2.24E+06

To include OpenVino this patch would require my users to -

1. Sign up at intel.com and download of the toolkit.
2. Manually install per target machine.
3. Extra installation of requirements_onnx.txt.
4. Source ~/intel/openvino_2020.4.287/bin/setupvars.sh.

I'd like to find a better way to include OpenVino if possible. Ideally, prebuilt wheels on pypi.org. The package size limit of 60MB would restrict including everything but the inference engine development kit + CPU runtime or GPU runtime would fit openvino-cpu-runtime/openvino-gpu-runtime?

[dkurt]

@iiSeymour, thanks for testing it!

OpenVINO team currently works on PyPi wheels publication so I believe it will be available next coming release. Going to try test package and update PR.

[dkurt]

Pushed a commit with test OpenVINO package. Now installation requires only

export LD_LIBRARY_PATH=$(pwd)/venv3/lib:$LD_LIBRARY_PATH

but still users have to convert model to OpenVINO IR (once).

[iiSeymour]

Perfect - with supporting wheels on PyPI my intention will be the make the default the CPU backend OpenVINO, so I would package the required files and avoid the conversion step.

Do you intend to stick with dynamic library loading? Check out auditwheel https://github.com/pypa/auditwheel

[dkurt]

Just pushed a commit with runtime OpenVINO model creation (without Model Optimizer call or model serialization).

There is some weak point with residual connection - had to wrap it to Sum class to detect by forward hook. Hope I'll fix it.

[dkurt]

Comparing formulas from my script (see #52 (comment))

pads = 498
out_size = chunksize // 3
length = out_size - 2 * pads
num = 2 + (inp.shape[-1] - chunksize) // (3 * length)
num = 2 + (inp.shape[-1] - chunksize) // (chunksize - 6 * pads)

with chunk method from utils.py:

num_chunks = raw_data.shape[0] // (chunksize - overlap) + 1

we can say that overlap = 6 * pads is correct:

output for different chunk size

$ bonito basecaller dna_r9.4.1 reads/test1 --device cpu
AAACAACACATACACACACACACACACACACACACACACACTCTCTCAGTAAATGTGCAGGAAGAAGAGAGACACTATCAATCGAAAACGTATGTAGCTCGAAATTAGGGAGGCAGTAAGTGGGAGGGAGATCTATAAAAGGAAGAAAGAACAGGAAGTTATCCAAGAGAGAGAGATGTATATTGCTCGCGTGCAGAGAGAGAGAGTAGTACATATATATATATATATATATATATATGTGTGTGTGTGTATTTTTAAATCATCGGGTGGAAGCCGGAATAAAAAGCAGGAGATCCCTTTGAAGTAAGTCTCATGATCCATTGCGGGGGTCGTATGCGTGCTGTGGATTAATATAAATGGGTGATCAAAAAAAATTTAGTAAAGGTGAGGCAGCATCATGTATCTCGAGTCCAAAGAAATAATCAAGAGAGAGAGAGATGATAGATAGATCTTGGAGGAGTAAATGAATAATCAAAGA

$ bonito basecaller dna_r9.4.1 reads/test1 --device cpu --chunksize 3600 --overlap 2988
AAACAACACATACACACACACACACACACACACACACACACTCTCTCAGTAAATGTGCAGGAAGAAGAGAGACACTATCAATCGAAAACGTATGTAGCTCGAAATTAGGGAGGCAGTAAGTGGGAGGGAGATCTATAAAAGGAAGAAAGAACAGGAAGTTATCCAAGAGAGAGAGATGTATATTGCTCGCGTGCAGAGAGAGAGAGTAGTACATATATATATATATATATATATATATGTGTGTGTGTGTATTTTTAAATCATCGGGTGGAAGCCGGAATAAAAAGCAGGAGATCCCTTTGAAGTAAGTCTCATGATCCATTGCGGGGGTCGTATGCGTGCTGTGGATTAATATAAATGGGTGATCAAAAAAAATTTAGTAAAGGTGAGGCAGCATCATGTATCTCGAGTCCAAAGAAATAATCAAGAGAGAGAGAGATGATAGAAGTAGTTCTTGGAGAGAGTGAATGAAAATCTATTCAGAAAGAAAAA

$ bonito basecaller dna_r9.4.1 reads/test1 --device cpu --chunksize 6000 --overlap 2988
AAACAACACATACACACACACACACACACACACACACACACTCTCTCAGTAAATGTGCAGGAAGAAGAGAGACACTATCAATCGAAAACGTATGTAGCTCGAAATTAGGGAGGCAGTAAGTGGGAGGGAGATCTATAAAAGGAAGAAAGAACAGGAAGTTATCCAAGAGAGAGAGATGTATATTGCTCGCGTGCAGAGAGAGAGAGTAGTACATATATATATATATATATATATATATGTGTGTGTGTGTATTTTTAAATCATCGGGTGGAAGCCGGAATAAAAAGCAGGAGATCCCTTTGAAGTAAGTCTCATGATCCATTGCGGGGGTCGTATGCGTGCTGTGGATTAATATAAATGGGTGATCAAAAAAAATTTAGTAAAGGTGAGGCAGCATCATGTATCTCGAGTCCAAAGAAATAATCAAGAGAGAGAGAGATGATAGAAGTAGTTCTTGGAGAGAGATAAATGAAGAATCATATCAGAAGAGAAAA

As you may see only difference in the last chunk when total length is not divided by 3 and last chunk is shifted.

[dkurt]

@iiSeymour, OpenVINO 2021.1 is now available officially: https://pypi.org/project/openvino-python/. I've pushed commit with update

[iiSeymour]

@dkurt that's great - I will aim to evaluate before the end of the week.

[dkurt]

Hi, @iiSeymour! I just added OpenVINO dependency which is built with manylinux2014 toolchain with auditwheel postprocessing, as you suggested. Now it works fine with Python 3.6, 3.7, 3.8, 3.9 and different Linux distros.

[dkurt]

I'm happy to say that this PR is finally depends on the official OpenVINO PyPI package. It's based on manylinux2014 toolchain for Linux. Python 3.6, 3.7, 3.8 are supported for every OS (Linux, Windows, Mac).

[iiSeymour]

Hey @dkurt really great to see an official release of OpenVINO on PyPI! The latest release of Bonito today drops support for Python 3.5 with the move to PyTorch 1.8 so this a good time to revisit this. There are a few minor conflicts from the release and the latest models freeze all blank scores to a single fixed value.

[colindaven]

@dkurt It would be great to see a full tutorial on how to use OpenVino with bonito sometime. There would be a lot of interest.

[dkurt]

@colindaven, Hi! Thanks for feedback! Actually, you may just follow README: https://github.com/dkurt/bonito/blob/openvino/README.md. Everything you need is just specify --use_openvino --device=cpu

[dkurt]

Hi, @iiSeymour! I've updated proposal to make integration smoother for you. OpenVINO is now optional backend and regular setup won't have extra deps. For users who want use it from source, python setup.py develop can be replaced to pip install develop .[openvino]. For users who will use PyPI package, pip install ont-bonito installs a regular set of requirements but pip install ont-bonito[openvino] additionally downloads OpenVINO and enables --use_openvino option.

So with such changes proposed code can live in the master branch without any overhead 😊. Also, I see that number of Bonito users have CPU only environment. This PR also helps them to unblock such configurations.

[iiSeymour]

Hey @dkurt, that's great, I've been tied up with the v0.5.0 release this last couple of weeks but let's look at getting this merged next week.

[dkurt]

@iiSeymour, great news, thanks! Will resolve merge conflicts as fast as possible ;)

[dkurt]

@iiSeymour, I'd like to add a config for GitHub Actions to test install and evaluate steps (on CPU). Should I try it in this PR or by a separate one?

[dkurt]

Hi, @iiSeymour! Just a reminder 😃

[iiSeymour]

Hey @dkurt - I've merged the branch from your fork to a local branch so I can add back CPU decoding - should be good to go down to master after that #228

[dkurt]

Hey @dkurt - I've merged the branch from your fork to a local branch so I can add back CPU decoding - should be good to go down to master after that #228

Thanks! Do you need some help with it from me?