add support for alternative emotion and body models

[vladmandic]

Discussed in #334

^{Originally posted by nicoeiris11 January 31, 2023}
Hi Vlad!

I'm using human in the browser and currently experiencing a very bad accuracy for emotion detection.
Using blazeface default model and the distance to the webcam is always around 2-3 meters

I'd like to know if there's an alternative model for emotion detection supported by human taking into account Oarriaga Emotion model has %66 test accuracy. In your model's wiki page you mention GEAR as an alternative. Is it possible to experiment with the GEAR emotion model instead of Oarriaga's for emotion? Looking at GEAR repo I can see the emotion classifier is independent of the GenderAgeRace model but not sure if the emotion model is supported by human or not.

Some additional questions:

• High resolution should help, right?
• Is human doing all necessary preproc of the cropped face before predicting emotion? (grayscale, normalization, etc.)
• Is it possible to visualize the input of the emotion model for debugging purposes? (the 64x64 image sent to the model)

Thank you!

[vladmandic]

i took a quick look and decided its faster to just do it than to keep it on the to-do list
new code is published and example usage is in https://github.com/vladmandic/human/blob/main/test/test-node-emotion.js
model is pretty clean, only required strange input normalization.

i've also added one more feature:

• you can control face crop scale using config.face.scale setting (as before)
• added config.face.emotion.crop (hidden) setting to perform additional tighter crop if desired.

for example, face detector would detect face box and then according to config.face.scale=1.4 (that is the default) it would enlarge a box by 40% before performing crop.

but emotion model is trained on much tighter cropped image so instead of messing with face in global, you can set config.face.emotion.crop = 0.15 which would remove 15% of each edge of the cropped face image.
(default is not set and means that entire cropped face will be used as-is, just resized to fit the model).

i'm closing the issue as complete, but feel free to continue posting any questions.

[nicoeiris11]

Thank you so much Vlad for publishing the new code so quickly! 🙏

[nicoeiris11]

Considering some times my pipeline processes bad-quality face images (e.g: small faces because of distance), I'm thinking about discarding blazeface detected boxes with less than configurable resolution (before crop and resizing).

Given subsequent face models X (emotion) and Y (description), before resizing cropped face boxes, discard face boxes where:
min_face_res = min(input_resolution(X), input_resolution(Y))
box_size = getBoxSize(blazeface_box)
if ( box_size[0] < min_face_res && box_size[1] < min_face_res ) discard_box(blazeface_box)

What do you think about it? Not sure if there's an existing preprocessing function to achieve this, otherwise if you think this could be helpful I can create a PR for your review.

[vladmandic]

i like the idea. and all necessary code is already pretty much in place, just needed to add a filter statement.
i've made it a configurable item config.face.detector.minSize. also face results object now has a size property.
new code is published.

[nicoeiris11]

My god! Are you an AI bot or a real developer? Again thank you so much!! I couldn't be more grateful.

I'll be sponsoring you later today for your hard work and responsiveness.

[nicoeiris11]

Hi Vlad! Does human npm contains latest changes related emotion models & configs?
I upgraded the human npm package in my electron app but experiencing erros when using affectnet-mobilenet.json emotion config.

I see the latest release is from Nov 22.

[vladmandic]

not yet, it will be in version 3.0.5 which i'll publish either end of this week or early next week.
btw, latest current version is 3.0.4 which is just 9 days old, not from nov 22. latest version in v2 branch in from nov 22.

[nicoeiris11]

Hi Vlad! Do you happen to have any updates regarding the next npm release?

On other notes, did you ever consider integrating AlphaPose into the human library as an alternative body model?

Thank you and have a great weekend!

[vladmandic]

most likely monday.
its ready, but awaiting one user confirmation on resolved issue around react-native as i don't currently have an android dev kit around.

re: alphapose - i've looked at it before, looks really nice, but

• its really a lot of custom code that would nee to be ported from python to javascript, its not just the model
• models are huge - smallest one is 150mb. which kind of defeats a purpose for lightweight solution such as human.

if i'm going in a direction of heavy-weight model, might as well go all the way - for that i'm exploring metrabs model.
you can take a look at https://github.com/vladmandic/body-pose

[nicoeiris11]

Thank you for the update regarding the next deployment.

btw, great work with the body-pose project (added a star on that one also).

Totally understand is too much work to integrate AlphaPose.

• I'm looking at the available body models and noticed that the only multi-person one is PoseNet, right?
• I see in the models table there is a MoveNet-MultiPose (movenet-multipose.json) in human-models repo, but when I test it in the demo it's not working. I'm missing something or it's not working at all?
• Is any of the supported body models able to detect multi-person and also detect fingers as BlazePose? I'd like to avoid using hand models for performance purposes. I saw that BlazePose detects fingers apart from elbow, wrist, etc.

[vladmandic]

I see in the models table there is a MoveNet-MultiPose (movenet-multipose.json) in human-models repo, but when I test it in the demo it's not working. I'm missing something or it's not working at all?

It should be working, what's the problem?

Is any of the supported body models able to detect multi-person and also detect fingers as BlazePose?

Not really. I'd suggest to use hand model for that. Even blazepose estimation of hand is not that precise.

I'd like to avoid using hand models for performance purposes.

Is it really such a performance hit?

[nicoeiris11]

It should be working, what's the problem?

Body detections are always empty even when the entire body is visible in the camera.

[vladmandic]

i just took a look, seems like model is working, but there is something wrong with keypoint scores returned by latest version of the movenet-multipose

to confirm, model works fine if minConfidence check is disabled:

  config.body = { enabled: true, minConfidence: -1, modelPath: 'https://raw.githubusercontent.com/vladmandic/human-models/main/models/movenet-multipose.json' },

update: yup, movenet-multipose model sometimes is randomly reporting zero scores for valid keypoints (and that is raw output from model, prior to any postprocessing). also checked input normalization and its fine, this is a bug in the model itself.

i'll report it to mediapipe and add a workaround for the time being (basically disable minConfidence check just for this model, there is no bad sideeffects since human performs a lot of postprocessing on results, so it all looks good at the end, just score is missing).

[nicoeiris11]

I pulled the changes you uploaded earlier today and executed the demo using movenet-multipose with minConfidence: -1, and is always detecting as many bodies as defined in body.maxDetected.

In the following screenshot you can see what's happening:

[vladmandic]

in my case, background was clear so there were no ghosts. but i'm affraid i cant do much until model itself returns a valid score (in any format)

[nicoeiris11]

This depends on MediaPipe updating the model itself, right? Thank you so much for checking it out Vlad!

On other notes, the npm human package is updated with the latest changes from what I can see, could you please confirm?

[vladmandic]

This depends on MediaPipe updating the model itself, right?

right.

On other notes, the npm human package is updated with the latest changes from what I can see, could you please confirm?

updated few hours ago and published on npm.
github version additionally includes the minConfidence = -1 workaround for multipose, but i'm not going to bump a version and republish on npmjs just for that.
(i've also added it to list of known issues in TODO.md)

[nicoeiris11]

I don't need the minConfidence workaround for multipose in npm version so don't worry.
Thank you so much Vlad!

[nicoeiris11]

Hi Vlad,

Any update from MediaPipe regarding the broken movenet-multipose model?

update: yup, movenet-multipose model sometimes is randomly reporting zero scores for valid keypoints (and that is raw output from model, prior to any postprocessing). also checked input normalization and its fine, this is a bug in the model itself.

i'll report it to mediapipe and add a workaround for the time being (basically disable minConfidence check just for this model, there is no bad sideeffects since human performs a lot of postprocessing on results, so it all looks good at the end, just score is missing).

I need multi-person body detection, and posenet doesn't have enough accuracy for my requirements (detecting elbow and wrist for user actions + count & track people in the video).

I'd really like to test movenet-multipose for its usage in production.

UPDATE: I don't think movenet-multipose model is broken on their side since I found a demo where it's working fine even with not clear background in the frame: https://storage.googleapis.com/tfjs-models/demos/pose-detection/index.html?model=movenet

[vladmandic]

no feedback so far. and version i have is definitely buggy and its the latest version, i'll take a look at tfjs version.

[nicoeiris11]

Do you think tfjsversion might be the issue?

I don't understand why their demo works perfectly, attached you can find a screenshot of the detection and config.

Is there any chance that the conversion of the model you made could be the problem? I see that when downloading the model from the hub, the weights are divided into 3 bin files (shards).

Should I create an issue in the repo to track this model troubleshooting?

Thank you so much for looking into this Vlad!

[vladmandic]

first, no need to open a separate issue anymore since i've figured out whats going on, but in the future please do not mix multiple issues - the subject line here is about alternative emotion models, movenet-multipose hardly has anything to do with that.

anyhow, the issue is NOT with model weights (i prefer to have monolithic weights instead of split by 2mb which i feel is more like going back to 90ies, but it is the same model).
and issue is not in the model itself (i wish mediapipe actually investigated when i reported, but thats another story).

the issue is in one specific tfjs webgl optimization which is disabled by default (so online tfjs demo works just fine), but i've been using for over a year now. anyhow, i've enabled a workaround so that optimization gets autodisabled if movenet model is requested.

related issue is tensorflow/tfjs#7430

new code is published on github.

[nicoeiris11]

Thank you so much for fixing this issue, and apologies for mixing conversations related to different issues, it won't happen again.

Do you have an estimated date for the next npm release?

I've been doing some research on alternatives pose estimation models that I'd like to share with you soon.
Do you mind if I open a new discussion thread to present you with these alternatives? This would enable human users to leverage all the amazing features you built for different use cases regarding body detection. I'm really looking forward to it!

Thank you for the support, Vlad! I'll continue sponsoring your hard work 💪

[vladmandic]

Do you have an estimated date for the next npm release?

no plans right now. right now changes are too small to warrant a release, but tfjs 4.3.0 should be coming out soon, so i'll then do a full refresh and publish a release to be in-sync with latest version of tfjs. my best-guess is in 2-3 weeks tops.

but you can install using npm direct from github, so that shouldn't be an issue?

Do you mind if I open a new discussion thread to present you with these alternatives?

of course not!