Generation of single tflite file for hand geustures recognition

[Deepakpy]

Issue Description
I am new to mediapipe. I want to use the mediapipe hands (hand detection and hand landmarks models) to be combined along with a simple gesture recognition model. All three to be combined to a single tflite file for inference on a microcontroller.
Steps to Reproduce
Is it possible to take the blaze hand models weights and make those layers non trainable and add few additional FC layers which take two inputs the keypoints from the blaze hand models and additional label input for predefined geustures. The output should be a single prediction of hand gesture.

Any leads will be helpful. To generate the model.
The overall idea is to have this multistage model compressed into a single model which can be trained with just few epochs and with less data for new gestures leaveraging the already trained weights from blaze hand.

[vladmandic]

possible, but very much out of scope here. first, i'm not using tflite, so it wouldn't be part of this project.

but there are bigger conceptual issues:

• tflite makes sense for models that are designed around int, not float as most microcontrollers have very slow fp execution. so its not about taking any model and just converting it, it needs to be designed in that way to really make sense
• having single execution pipeline (typically that would be done with attention linking) makes sense if both stages are always executed. but in all my tests, hand/palm detection is much slower than landmark detection (finger joints). so what i'm doing is caching detection boxes and recalculating them within an area (e.g. if hand only moved 10-15% between frames, new hand box can be calculated instead of forcing detection from scratch). only when hand cannot be calculated, it is re-detected - that results in much higher performance. so keeping models separate to be executed on-demand makes much more sense.
• i'm not sure you want to have gestures in ml as that limits which gestures can be recognized by a lot. i prefer to have landmark detection and do gesture calculation separately - its not that cpu intensive.

[Deepakpy]

Microcontroller with 2 to 4x Arm® Cortex®-A53s and 1x Cortex-M4 cores. It brings High-Performance with Low Power, Flexible options of Memory and High-Speed Interfaces as well as advanced Audio and Video capabilities.
OS : Yocto Linux
The microcontroller planned to use has a dedecated neural network accelerator and gpu. Most microcontrollers have support to run neural network model which are in tflite which made me to choose the tflite format.

The idea of using the landmarks from keypoints detected and computing geustures from it is one thing which i have to explore. I initially thought having a not so computationally intensive neural network would be super easy to add in newer geustures to be detected with less training.

[vladmandic]

btw, i've converted the issue into a discussion as its a good topic, but not really a product issue.

[vladmandic]

The microcontroller planned to use has a dedecated neural network accelerator and gpu. Most microcontrollers have support to run neural network model which are in tflite which made me to choose the tflite format.

in that case, floating-point is not an issue, so you could convert to tflite. still, the recommendation remains to keep hand landscape detectors separate as its much faster if you can skip re-running detector each time.

The idea of using the landmarks from keypoints detected and computing geustures from it is one thing which i have to explore. I initially thought having a not so computationally intensive neural network would be super easy to add in newer geustures to be detected with less training.

my approach is to run landscape points through generic analysis first so i get things like "finger x is half-curled and pointing right".
and then its very easy to define actual gestures using something thats almost like a high-level macro language (plus its very extensible).

take a look at https://github.com/vladmandic/human/blob/main/src/hand/fingergesture.ts for some examples..

[vladmandic]

btw, i've created a script that converts all model to tflite - see https://github.com/vladmandic/human-models/blob/main/src/tfjs2tflite.py