Gender reversal bug

[ericwhyne]

Openbr seems to consistently get the gender wrong. Josh Klontz has seen same behaviour and believes there's a bug in the master branch causing gender reversals.
re: mailing list Thu, Nov 21, 2013 at 11:26 AM, Eric Whyne

Here's the script I've been using to take an image and output a single word gender. I give it cropped faces and reverse it's output.

 #!/bin/bash
 br -algorithm GenderEstimation -enroll $1 tmp/metadata.csv 2> /dev/null
 openbr_gender=`awk -F'^|,|$' '{print $18}' tmp/metadata.csv | tail -n 1`
 #echo $openbr_gender
 # Dirty hack to reverse gender to get correct result for our data.
 if [ $openbr_gender == 'Male' ]
 then
   echo "Female"
 else
   echo "Male"
 fi

[jklontz]

Unfortunately it isn't the quick fix I thought it would be. Looks like the algorithm is simply getting worse accuracy than it used to (down to ~86% from ~95%):

Overall Accuracy = 0.864375
Count Precision Recall F-score
Male 8000 0.983416 0.74125 0.845331
Female 8000 0.792377 0.9875 0.879243

@ericwhyne You might need to revert to the 0.4, or even 0.3, release for the time being.
@caotto Do you recall what accuracy were you seeing the last time you trained it? Odd that the precision/recall are so skewed.

[caotto]

I don't remember, so presumably it was still 90-something.
I'll take a look this weekend, under the current labeling scheme it's hard to see how a label reversal would even happen, but seems like something is definitely going on.

[caotto]

Well after some work with git bisect, I can say that 1. bad submodule refs were pushed like 3 times in early August, and 2. in between those breaks, the real culprit was e9c9e03

@imaus10: is my read correct that that commit transposes the order in which points are added? If so I think that's a reasonably clear compatibility break.

Anyway I re-trained gender classification and that is back to ~95% so, I will train age estimation and face recognition again, and push that later.

[jklontz]

Yikes, it's amazing we weren't getting 50% accuracy! Thanks for looking into and fixing this. @imaus10 it's good that you identified and fixed the bug in e9c9e03, just be aware that when you change the behavior of existing plugins it may have ramifications on the pre-trained algorithms, so you should inform the owner of the plugin before pushing the change. Or perhaps you did and I signed off on the change, in which case I'm to blame :)

[caotto]

@ericwhyne: Please try using the updated models from commit fea31bb

[imaus10]

@jklontz Got it, sorry for the mix up.

[caotto]

Closing, re-open if you believe there is still an issue.

[ericwhyne]

Thanks! I haven't gotten back around to this part of the project yet but I
will soon. I'll report back if it acts strange again.

R/
Eric

On Mon, Dec 2, 2013 at 1:39 PM, caotto [email protected] wrote:

Closed #135 #135.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/135
.

[torrmal]

Hi Caotto,

I am having the same problem, I am a mac user, I cloned and installed from https://github.com/biometrics/openbr.git

Do you know how can I merge the commit that has your fix?

Thank you,

Jorge

[torrmal]

actually it seems like I do have that commit:
git log | grep models
Address #135, update models ref due to compatibility break

However, I do get Female all the time.
Do you know if I am missing something?

Thank you,

Jorge

[jklontz]

Can you share an example image on the mailing list?