Discuss about MMDeploy pre-build.

[grimoire]

We plan to create some pre-build package to ease the installation of MMDeploy. Here are some details:

Environment and Platform

environment

• linux-x86_64 （cpu/cuda11.x)
• windows-x86_64 (cpu/cuda11.x)
• Jetson

backend

• TensorRT 8
• ONNX Runtime 1.8

We will support these combinations. More systems and backend would be added in the future ( if needed).

Plan

Model Converter and SDK would be built in two different packages.

Model Converter

Most of the code in the model converter is python, so we plan to align the pre-build package with PyTorch.

• Decouple export IR and convert backend model is a good choice. Since user might want to deploy their model in some edge device (jetson for example).It is not easy to install some third-party library on these devices. Instead, we can export ONNX on PC and convert TensorRT model on jetson device with NO third-party dependency.
• All compiled libs (custom ops, export tools etc) should be placed in mmdeploy/lib so they can be packed in the same wheel with the python code.
• Some tools can be mapped to mmdeploy so we can use them like python -m mmdeploy deploy ....
• The wheel will be placed in our server (or somewhere else that can be access public), user can install the package with pip install mmdeploy=={version} -f https://<url_to_wheels>

SDK

The behavior of SDK looks like an inference engine, so the package would be aligned with TensorRT pre-build package.

• SDK would be packed as tar or zip package, which includes heads, libraries, python API, and anything that might be necessary. It would be published after each release of MMDeploy.
• SDK could be used independently of the model converter.
• The python API (I mean mmdeploy_python) will be another wheel, which is placed in <package_dir>/python, just like TensorRT. User can optionally install the API.

I have made a private repo mmdeploy_builder.py to do the model converter build. It is not well tested, and is buggy(I guess). SDK build tools have not yet been developed.

There are two PR related:

• Reduce torch dependency #273 which tries to remove the dependency of PyTorch and any other third-party libs.
• [Refactor][tools] Add prebuild tools. #347 move the library to the mmdeploy/lib so we can pack them together with python code.

I plan to close #273 since it is an old PR with a lot of conflict with the current master. It would be done in #347 again.

There is still a lot to do:

• SDK build tools, include heads, libs, cmake, python API (most of them can be provided by the installation besides the python API). @PeterH0323
• Decouple export IR and convert backend model. (remove third-party dependency, make sure we can convert TensorRT model on the jetson platform without installing PyTorch)
• windows support.
• bug fix
• Anything I might forget to mention.
• build script or tools to ease the installation.

@tehkillerbee @PeterH0323 Please feel free to choose the tasks (or is there any task I missed?). Let's work together to make MMDeploy Better!

[tehkillerbee]

@grimoire So just to understand, the plan is to have a tool that can build MMDeploy + SDK for various platforms and with various backends? To allow pre-build releases of MMDeploy that can be easily downloaded by the user?

Initially, I thought a single build script would be used, similar to how it is done with ppl.cv. This might be the easiest solution but it requires the user to build locally. This is what I ended up doing myself.

If we want to do pre-build packages, my main concern is how to handle building for other platforms - eg. aarch64, as I assume we would crosscompile the packages on a non aarch64 build server?

I have previously tried crosscompiling ppl-cv, MMDeploy and the dependencies, but I did not manage to get it working properly due to the way the dependencies are detected. Instead, I wrote a bash script that handles building and installing all dependencies (including spdlog, pplcv) and MMDeploy SDK on the Jetson itself. Then I was able to crosscompile my custom inference tools on my dev. machine using the Jetson rootfs containing the pre-built MMDeploy SDK libs.

Decoupling IR and convert backend model to avoid pytorch on Jetson sounds like a good idea, that would simplify install steps if you only need to do TensorRT model conversion on Jetson. Then all other steps can be done in advance on the development machine.

By the way, how do we handle misc. dependencies that are not installed by MMDeploy pre-build package? Eg. mmcv-full and other python packages that are needed for mmdeploy? Can they be installed automatically as part of the installation of MMDeploy wheel?

[PeterH0323]

By the way, how do we handle misc. dependencies that are not installed by MMDeploy pre-build package? Eg. mmcv-full and other python packages that are needed for mmdeploy? Can they be installed automatically as part of the installation of MMDeploy wheel?

Yes. We will install the dependencies which we need when we install the mmdeploy wheel package.

[PeterH0323]

If we want to do pre-build packages, my main concern is how to handle building for other platforms - eg. aarch64, as I assume we would crosscompile the packages on a non aarch64 build server?

We need to use the aarch64 to build it and then relesae it, just like some pre-build package need to choose x86 or aarch64.

[grimoire]

@tehkillerbee
A build script is a good idea (Actually I am planning to add an interact script, or even install tools with a panel, which we can fill whatever dependency we need then click next). I always want to add this. It can also be part of this plan (now).
But still, people might want some packages that they can download and use without any work to do. We do not have to cover all platforms. They can build it with source code by themself if their platforms has not been pre-built yet. (Just like mmcv, we can install the pre-built one or build from source).
We do not need to worry about cross-compile for now. The pre-build for jetson can be built on the device for now until we figure out how to build it on x86-64.
As for the misc you mentioned. We can add part of it into the requirements, but I guess some dependency should be installed manually(such as the installation requires login to their main page). That is not a big deal since user should always install these package whatever they use pre-built package or not.

[PeterH0323]

So just to understand, the plan is to have a tool that can build MMDeploy + SDK for various platforms and with various backends? To allow pre-build releases of MMDeploy that can be easily downloaded by the user?

Yes, cuz some users have trouble when they deploy into some device. At the same time, they need to install all mmdeploy dependencies step by step according to the doc which is little harder for some users. I think this pre-build package will help more users to use mmdeploy more fluently.

[PeterH0323]

There is still a lot to do:

• SDK build tools, include heads, libs, cmake, python API (most of them can be provided by the installation besides the python API).
• Decouple export IR and convert backend model. (remove third-party dependency, make sure we can convert TensorRT model on the jetson platform without installing PyTorch)
• windows support.
• bug fix
• Anything I might forget to mention.

Please feel free to choose the tasks (or is there any task I missed?). Let's work together to make MMDeploy Better!

Hi @grimoire , I want to pick SDK build tools, include heads, libs, cmake, python API (most of them can be provided by the installation besides the python API). 😄

[tehkillerbee]

@grimoire I could probably assist with the build script, since I have already worked on something like that.
I can also show the simple bash script I wrote for installing dependencies + MMDeploy SDK.

export IR and convert backend model sounds like quite a large task and would probably require some pointers if I were to look at that. Do you have any suggestions on where to start with that task?

[grimoire]

Root folder is OK. And I think you can keep the author information if you want.
You can try adding other backend support in the script, or just push the script and discuss it with other reviewers.

[tehkillerbee]

@grimoire @PeterH0323 Since the latest python build script also builds the SDK, does it make sense to add this bash script as well?

[grimoire]

Sounds like a good idea.

[tehkillerbee]

@grimoire Ok, I will add the script if you think it is a good idea. Should I add it to your repo or MMDeploy?

EDIT: Maybe it would make sense to use the .yaml config files for the bash script as well? Only problem is the way pplcv paths are handled by my script as I build ppclv together with mmdeploy.

[tehkillerbee]

I have created PR #399 for further discussion with other reviewers, I hope this is OK with you.