Impress that special someone with Haarcascades

It is always hard to think of something for valentines day. Sure you could get her flowers, chocolate, but nothing says I love you like a cute piece of software.

In this tutorial I will show you what I did to train my computer to recognize hand hearts, and how you can do the same. I followed the tutorial from here. It is very informative but it took a while to get it to work and I only used a few parts of the tutorial. I will also include the cascade that I ended up with,  but it may be far more romantic and accurate if you generate one on your own, but this requires patience.

My goal was simple use haarcascades to detect hand hearts the same way a smartphone detects a person's face. Naotoshi Seo's tutorial suggested taking 2000 pictures containing the object to be detected and use 5000 negatives that do not contain the object. This seemed like a lot and instead I used 1000 positives and 1000 negatives. It took a while but I was able to speed up the process with some handy software.

The first piece of code I wrote was to take 1000 pictures and save them. To make my life easier, the code takes a picture every 250 milliseconds. It is important to note that this code assumes there is a directory called images and it will overwrite any previous images so make sure to back up your positives before taking your negatives.

Here is the code:

#include <stdio.h>
#include <opencv2/opencv.hpp>

int main (int argc, const char * argv[])
{
    // Initialize capturing live feed from the camera
        CvCapture* capture = 0;
        capture = cvCaptureFromCAM(0);
        // Couldn't get a device? Throw an error and quit
        if(!capture) {
                printf("Could not initialize capturing...\n");
                return -1;
        }

        // The two windows we'll be using
        cvNamedWindow("video");

        for(int i=0; i<1000; i++) {
                // Will hold a frame captured from the camera
                IplImage* frame = 0;
                frame = cvQueryFrame(capture);
                if(!frame)
                        break;

                cvShowImage("video", frame);

                int c = cvWaitKey(10);
                if(c == 'q')
                        break;
                usleep(250000);
                char name[32];
                sprintf(name, "images/%d.jpg", i);
                cvSaveImage(name, frame);
        }
        // We're done using the camera. Other applications can now use it
        cvReleaseCapture(&capture);
    return 0;
}

After you take all your positives and negatives your hands should be thoroughly sore from making all those lovely hand hearts. The next tedious task you have to complete is to crop all your positives. To do this I followed Naotoshi Seo's advice and used imageclipper it helped make the cropping go fast. If you made it this far without chickening out and using my cascade congratulations.

At this point you are probably thinking, "Oh great I'm almost done!" Well you aren't, but now the work is handed over to the computer. You will have to run a few shell commands and I'm sorry windows users this might not work for you. You will have to at least install mingw or cygwin to get these commands to work. If you are running a mac you are in luck especially if you have macports installed. If you don't thats okay but It makes this next step less painful.

You have to install ImageMagick you can find the binaries here or if you have macports installed just run:

sudo port install ImageMagick

Now we will generate some descriptor files that are needed for haartraining. The negatives are easy you just need a big list of the image files so assuming your negatives are all in the subdirectory called negatives run:

find negatives -name \*.jpg > negatives.dat

The positives are a bit trickier because the file must also include the size of all the cropped images so run the following command assuming the cropped images are in the dir called positives:

find positives -name '*.png'-exec identify -format '%i 1 0 0 %w %h' \{\} \; > samples.dat

HaarTraining needs one more file so run this command:

opencv_createsamples -info samples.dat -vec samples.vec -w 20 -h 20

Now we are ready to hand this off to the computer. Expect this to take all day and night:

opencv_haartraining -data haarcascade -vec samples.vec -bg negatives.dat -nstages 20 -nsplits 2 -minhitrate 0.999 -maxfalsealarm 0.5 -npos 7000 -nneg 3019 -w 20 -h 20 -mem 512 -mode ALL

When that completes give yourself a pat on the back as you now are finished with that nightmare and are ready to write the final piece of code.

The detection code was simple. It was as easy as renaming the cascade xml in a simple face detection program, and inserting a heart. The face detection program I started off with simply drew a rectangle around the face. As I wanted it to place a heart instead of a box I just had to copy an image of a heart into the box. To do this I set the ROI or Region Of Interest to the box returned by cvHaarDetectObjects and scaled the image of the heart to and used a temporary image as the destination. The temporary image is then added to the ROI causing a translucent effect and makes black a transparency.

Here is what I ended up with:

#include <stdio.h>
#include <opencv2/opencv.hpp>

CvHaarClassifierCascade *cascade;
CvMemStorage            *storage;

void detectObjects(IplImage *img, IplImage *heart, IplImage *temp)
{
        int i;

        CvSeq *objs = cvHaarDetectObjects(
                        img,
                        cascade,
                        storage,
                        1.1,
                        3,
                        0 /*CV_HAAR_DO_CANNY_PRUNNING*/,
                        cvSize(60, 60 ) );

        for(i = 0 ; i < (objs ? objs->total : 0) ; i++) {
                CvRect *r = (CvRect*)cvGetSeqElem(objs, i);
                cvSetImageROI(img, *r);
                cvSetImageROI(temp, *r);

                cvResize(heart, temp);
                cvAdd(temp, img, img);

                cvResetImageROI(temp);
                cvResetImageROI(img);
        }

        cvShowImage("video", img);
}

int main(int argc, char *argv[])
{
        CvCapture *capture;
        IplImage  *frame;
        IplImage  *heart = cvLoadImage("heart1.jpg");
        int       key;
        char      *filename = "handheart.xml";

        cascade = (CvHaarClassifierCascade*)cvLoad(filename, 0, 0, 0);
        storage = cvCreateMemStorage(0);
        capture = cvCaptureFromCAM(0);

        assert(cascade && storage && capture);

        cvNamedWindow("video", 1);
        frame = cvQueryFrame(capture);
        IplImage *temp = cvCreateImage(

                         cvSize(frame->width, frame->height),

                         IPL_DEPTH_8U, 3);

        while(key != 'q') {
                frame = cvQueryFrame(capture);

                if(!frame) {
                        fprintf(stderr, "Cannot query frame!\n");
                        break;
                }

                detectObjects(frame, heart, temp);

                key = cvWaitKey(10);
        }

        cvReleaseImage(&heart);
        cvReleaseImage(&temp);
        cvReleaseCapture(&capture);
        cvDestroyWindow("video");
        cvReleaseHaarClassifierCascade(&cascade);
        cvReleaseMemStorage(&storage);

        return 0;
}

For this to run properly you will need the following image:

For those of you that were not patient enough here is the haarcascade that I trained myself.

handheart.xml

If this works for you as well as it did for me the ladies should now be all over you. I am not responsible for any unexpected results. For those of you who completed your own haartraining and of course my special girl this next picture is for you.