Example precision?

[RafaRuiz]

Hello, thanks again for answering my last question, I think this is the last one.

I've used facial landmarks before but not this library, so I'm discovering it indeed :).

So, I tried to copy and paste your example in Xcode, the video_input.cpp. Just to be correct, I'm pasting it as I have it: (I just removed or added some irrelevant information):

int main( int argc, const char** argv )
{

    //CvCapture* capture = 0x0;
    //CvVideoWriter* writer = 0x0;
    VideoCapture capture;
    VideoWriter writer;

    Mat frame;

    double tic;
    bool saveoutput = false;
    string out_fname;

    Flandmark *flandmark = Flandmark::getInstanceOf(landmarks_path.c_str());

    //-- 1. Load the cascades
    if( !face_cascade.load( cascade_path ) )
    {
        cerr << "Couldn't load the haar cascade. Exiting..." << endl;
        return -1;
    };


    capture.open(0);
    // window
    namedWindow(window_name, CV_WINDOW_KEEPRATIO);
    //    namedWindow(window_name, WINDOW_KEEPRATIO);

    //-- 2. Read the video stream
    if( capture.isOpened() )
    {
        while( true )
        {
            tic = (double)getTickCount();

            capture >> frame;

            resize(frame, frame, Size(frame.size().width/3, frame.size().height/3));

            //-- 3. Apply the classifier to the frame
            if( !frame.empty() )
            {
                detectAndDisplay( frame, flandmark );

                tic = ((double)getTickCount() - tic)/getTickFrequency() * 1000;

                stringstream fps;
                fps << "fps: " << setprecision(4) << setw(4) << 1000.0 / tic << " ";
                putText(frame, fps.str(), Point(10, 25), CV_FONT_HERSHEY_COMPLEX, 1, Scalar(255, 255, 0));
                //              putText(frame, fps.str(), Point(10, 25), FONT_HERSHEY_COMPLEX, 1, Scalar(255, 255, 0));

                imshow( window_name, frame );

            } else {
                cerr << "No frame --- break." << endl;
                break;
            }

            if (saveoutput)
            {
                writer << frame;
            }

            int c = waitKey(10);
            if( (char)c == 'c' )
            {
                break;
            }
        }
    }

    delete flandmark;

    return 0;
}

and the result is this at the moment:

GIF:

seems like flies around my face :)

Do you have any clue? these are the cascade and model:

string cascade_path = "haarcascade_frontalface_alt.xml";
string landmarks_path = "joint_mv_models/JOINT_MV_AFLW_SPLIT_1_frontal.xml";

thank you very much in advance.

[uricamic]

Hi,

I think the problem is that the learned model available on the webpage is still quite weak. It was learned on around 3,500 examples only for the whole multi-view scenario (i.e. 5 yaw ranges).
I will upload some new models on the project webpage soon. It should be a bit better, however, we will soon learn a new models with much more examples.

It could be also interesting to visualize the face box as well. I believe some precision gain might be obtained by stabilizing the noisy face detection (e.g. by a Kalman filter).

[uricamic]

Hi @RafaRuiz,

I almost forgot to ask, do you use CFeaturePool * in your code? If not then the problem is that the model which you are using is supposing that you use it. So the resulting precision would be totally unpredictable.

[RafaRuiz]

Hello @uricamic, This is the new code. Basically I added the example of the CFeaturePool* from static_input.cpp to the code after declaring the Flandmark flandmark:

main function:

int main( int argc, const char** argv )
{

    //CvCapture* capture = 0x0;
    //CvVideoWriter* writer = 0x0;
    VideoCapture capture;
    VideoWriter writer;

    Mat frame;

    double tic;
    bool saveoutput = false;
    string out_fname;

    Flandmark *flandmark = Flandmark::getInstanceOf(landmarks_path.c_str());
    CFeaturePool *featurePool = new CFeaturePool(flandmark->getBaseWindowSize()[0], flandmark->getBaseWindowSize()[1]);
    featurePool->addFeaturesToPool(
                                   new CSparseLBPFeatures(
                                                          featurePool->getWidth(),
                                                          featurePool->getHeight(),
                                                          featurePool->getPyramidLevels(),
                                                          featurePool->getCumulativeWidths()
                                                          )
                                   );

    flandmark->setNFfeaturesPool(featurePool);

    //-- 1. Load the cascades
    if( !face_cascade.load( cascade_path ) )
    {
        cerr << "Couldn't load the haar cascade. Exiting..." << endl;
        return -1;
    };


    capture.open(0);
    // window
    namedWindow(window_name, CV_WINDOW_KEEPRATIO);
    //    namedWindow(window_name, WINDOW_KEEPRATIO);

    //-- 2. Read the video stream
    if( capture.isOpened() )
    {
        while( true )
        {
            tic = (double)getTickCount();

            capture >> frame;

            resize(frame, frame, Size(frame.size().width/3, frame.size().height/3));

            //-- 3. Apply the classifier to the frame
            if( !frame.empty() )
            {
                detectAndDisplay( frame, flandmark );

                tic = ((double)getTickCount() - tic)/getTickFrequency() * 1000;

                stringstream fps;
                fps << "fps: " << setprecision(4) << setw(4) << 1000.0 / tic << " ";
                putText(frame, fps.str(), Point(10, 25), CV_FONT_HERSHEY_COMPLEX, 1, Scalar(255, 255, 0));
                //              putText(frame, fps.str(), Point(10, 25), FONT_HERSHEY_COMPLEX, 1, Scalar(255, 255, 0));

                imshow( window_name, frame );

            } else {
                cerr << "No frame --- break." << endl;
                break;
            }

            if (saveoutput)
            {
                writer << frame;
            }

            int c = waitKey(10);
            if( (char)c == 'c' )
            {
                break;
            }
        }
    }

    delete flandmark;

    return 0;
}

Detect and Display function

void detectAndDisplay( Mat &frame, Flandmark *flandmark)
{
    std::vector<Rect> faces;
    Mat frame_gray;
    //  int bbox[4];
    int bbox[8];
    fl_double_t *landmarks;

    cvtColor( frame, frame_gray, CV_BGR2GRAY );
    //    cvtColor( frame, frame_gray, COLOR_BGR2GRAY );
    equalizeHist( frame_gray, frame_gray );

    //-- Detect faces
    face_cascade.detectMultiScale( frame_gray, faces, 1.1, 2, 0|CV_HAAR_SCALE_IMAGE, Size(30, 30) );
    //  face_cascade.detectMultiScale( frame_gray, faces, 1.1, 2, 0|CASCADE_SCALE_IMAGE, Size(30, 30) );

    //    CImage *frm_gray = new CImage();
    cimg_library::CImg<unsigned char>* frm_gray = 0x0; //= new cimg_library::CImg<unsigned char>();

    for( uint32_t i = 0; i < faces.size(); i++ )
    {
        // Get detected face bounding box
        bbox[0] = faces[i].x;
        bbox[1] = faces[i].y;
        bbox[2] = faces[i].x + faces[i].width;
        bbox[3] = faces[i].y;
        bbox[4] = faces[i].x + faces[i].width;
        bbox[5] = faces[i].y + faces[i].height;
        bbox[6] = faces[i].x;
        bbox[7] = faces[i].y + faces[i].height;

        // Detect facial landmarks
        frm_gray = cvImgToCImg(frame_gray);
        flandmark->detect(frm_gray, bbox);

        delete frm_gray;

        // Get detected landmarks
        landmarks = flandmark->getLandmarks();

        // Draw bounding box and detected landmarks
        //      rectangle(frame, Point(bbox[0], bbox[1]), Point(bbox[2], bbox[3]), Scalar(255, 0, 0));
        circle(frame, Point(int(landmarks[0]), int(landmarks[1])), 2, Scalar(255, 0, 0), -1);
        rectangle(frame, faces.at(i), Scalar(0,255,0), 3);
        for (int i=2; i < 2*flandmark->getLandmarksCount(); i+=2)
        {
            circle(frame, Point(int(landmarks[i]), int(landmarks[i+1])), 2, Scalar(0, 0, 255), -1);
        }

        // Textual output
        printTimingStats(flandmark->timings);
        printLandmarks(landmarks, flandmark->getLandmarksCount());
        printLandmarks(flandmark->getLandmarksNF(), flandmark->getLandmarksCount());
    }

    //    delete frm_gray;
}

and this is the result:

[RafaRuiz]

Also tried with:

string landmarks_path = "/Users/rafaelruizmunoz/Downloads/independent_mv_models/INDIVIDUAL_FRONTAL_AFLW_SPLIT_1.xml";

working in the same way

[uricamic]

Hi @RafaRuiz,

call flandmark->detect_optimized(frm_gray, bbox); instead of flandmark->detect(frm_gray, bbox);
That should do the job.

I am sorry for this inconvenience, it is not described anywhere yet, but there is an old version of computing features for compatibility reasons (our previous version of the library called flandmark).

The difference in INDIVIDUAL_FRONTAL_AFLW_SPLIT_1.xml and JOINT_MV_AFLW_SPLIT_1_frontal is that the first one was learned as an independent single-view detector, while the second is learned jointly with all 5 yaw ranges.

[RafaRuiz]

now it seems to be more stable!
no sorry to be about :)

Thank you very much for your answers, I can't see the moment to start playing with this ;)

[uricamic]

Glad to hear that! :)

I will upload the learning scripts for MATLAB soon and Python interface will follow up shortly.

[karlhugle]

@RafaRuiz Hi, can you write any instructions about setting up it using Xcode, tried but failed....