Subscribe:

Ads 468x60px

Sunday, May 26, 2013

Eye blink detection using OpenCV in Ubuntu

Hi all

This is a simple project that i did recently. The purpose of this project is to detect the blinking of eyes. The application of this project is to detect whether the driver inside the car is sleeping or not.

Using this program, we can detect the eyes of the user and can track their eyes.  Using template matching technique, we will check whether the eyes are closed or not .. If the eyes are closed, it will print a message in terminal also execute a python script to pass control the event into external devices such as arduino or anyother boards

 Opencv code :blink_detection.cpp



 // OpenCV Sample Application: blink_detction.cpp  
 // Include header files  
 #include "cv.h"  
 #include "highgui.h"  
 #include <stdio.h>  
 #include <stdlib.h>  
 #include <string.h>  
 #include <assert.h>  
 #include <math.h>  
 #include <float.h>  
 #include <limits.h>  
 #include <time.h>  
 #include <ctype.h>  
 #include <iostream>  
 #include "opencv2/objdetect/objdetect.hpp"  
 #include "opencv2/highgui/highgui.hpp"  
 #include "opencv2/imgproc/imgproc.hpp"  
 // Create memory for calculations  
 static CvMemStorage* storage = 0;  
 // Create a new Haar classifier  
 static CvHaarClassifierCascade* cascade = 0;  
 // Function prototype for detecting and drawing an object from an image  
 bool detect_and_draw( IplImage* image ,CvHaarClassifierCascade* cascade);  
 // Create a string that contains the cascade name  
 /*  "eyes.xml*/  
 const char *cascade_name[1]={"eyes.xml"};  
 cv::Mat roiImg;  
 int threshold_value = 200;  
 int threshold_type = 3;;  
 int const max_value = 255;  
 int const max_type = 4;  
 int const max_BINARY_value = 255;  
 int hough_thr = 35;  
 cv::Mat src_gray, dst;  
 using namespace cv;  
 Mat img1; Mat img2; Mat templ; Mat result;  
 const char* image_window = "Source Image";  
 const char* result_window = "Result window";  
 int match_method=0;  
 int max_Trackbar = 5;  
 int eye_open=0;  
 int eye_close=0;  
 /*  
 **  
  * @function MatchingMethod  
  * @brief Trackbar callback  
 */  
 //Matching with 2 images ,eye closed or open  
 void MatchingMethod(cv::Mat templ,int id )  
 {  
  /// Source image to display  
  cv::Mat img_display;  
  roiImg.copyTo( img_display );  
  /// Create the result matrix  
  int result_cols = roiImg.cols - templ.cols + 1;  
  int result_rows = roiImg.rows - templ.rows + 1;  
  result.create( result_cols, result_rows, CV_32FC1 );  
  /// Do the Matching and Normalize  
  cv::matchTemplate( roiImg, templ, result, match_method );  
  cv::normalize( result, result, 0, 1, NORM_MINMAX, -1, Mat() );  
  /// Localizing the best match with minMaxLoc  
  double minVal; double maxVal; Point minLoc; Point maxLoc;  
  cv::Point matchLoc;  
  cv::minMaxLoc( result, &minVal, &maxVal, &minLoc, &maxLoc, Mat() );  
  ///Justing checkin the match template value reaching the threashold  
  if(id == 0 && (minVal < 0))  
      {  
      eye_open=eye_open+1;  
      if(eye_open == 10)  
           {  
           std::cout<<"Eye Open"<<std::endl;  
           eye_open=0;  
           eye_close=0;  
           }  
      }  
   else if(id == 1 && (minVal < 0))  
      eye_close=eye_close+1;  
      if(eye_close == 10)  
           {  
           std::cout<<"Eye Closed"<<std::endl;  
           eye_close=0;  
           system("python send_arduino.py");  
           }  
  /// For SQDIFF and SQDIFF_NORMED, the best matches are lower values. For all the other methods, the higher the better  
  if( match_method == CV_TM_SQDIFF || match_method == CV_TM_SQDIFF_NORMED )  
   { matchLoc = minLoc; }  
  else  
   { matchLoc = maxLoc; }  
  /// Show me what you got  
  cv::rectangle( img_display, matchLoc, Point( matchLoc.x + templ.cols , matchLoc.y + templ.rows ), Scalar::all(0), 2, 8, 0 );  
  cv::rectangle( result, matchLoc, Point( matchLoc.x + templ.cols , matchLoc.y + templ.rows ), Scalar::all(0), 2, 8, 0 );  
  cv::imshow( image_window, img_display );  
  cv::imshow( result_window, result );  
  return;  
 }  
 void detect_blink(cv::Mat roi)  
 {  
      try  
      {        
      MatchingMethod(img1,0);  
       MatchingMethod(img2,1);  
      }  
      catch( cv::Exception& e )  
      {  
           std::cout<<"An exception occued"<<std::endl;  
      }  
 }  
 // Main function, defines the entry point for the program.  
 int main( int argc, char** argv )  
 {  
   if(argc <= 1)  
      {  
      std::cout<<"\n Help "<<std::endl;  
      std::cout<<"\n ------------------------------------\n"<<std::endl;  
      std::cout<<"./blink_detect open_eye.jpg close_eye.jpg\n"<<std::endl;  
     std::cout<<"Eg :: ./blink_detect 2.jpg 3.jpg\n"<<std::endl;  
      std::cout<<"\n ------------------------------------\n"<<std::endl;  
      exit(0);  
      }  
   // Structure for getting video from camera or avi  
   CvCapture* capture = 0;  
   // Images to capture the frame from video or camera or from file  
   IplImage *frame, *frame_copy = 0;  
   // Used for calculations  
   int optlen = strlen("--cascade=");  
   // Input file name for avi or image file.  
   const char* input_name;  
    img1 = imread( argv[1], 1 );  
    img2 = imread( argv[2], 1 );  
   // Load the HaarClassifierCascade  
    /// Create windows  
    cv::namedWindow( image_window, CV_WINDOW_AUTOSIZE );  
    cv::namedWindow( result_window, CV_WINDOW_AUTOSIZE );  
   // Allocate the memory storage  
   storage = cvCreateMemStorage(0);  
   capture = cvCaptureFromCAM( 0);  
   // Create a new named window with title: result  
   cvNamedWindow( "original_frame", 1 );  
   // If loaded succesfully, then:  
   if( capture )  
   {  
     // Capture from the camera.  
     for(;;)  
     {  
       // Capture the frame and load it in IplImage  
       if( !cvGrabFrame( capture ))  
         break;  
       frame = cvRetrieveFrame( capture );  
       // If the frame does not exist, quit the loop  
       if( !frame )  
         break;  
       // Allocate framecopy as the same size of the frame  
       if( !frame_copy )  
         frame_copy = cvCreateImage( cvSize(frame->width,frame->height),  
                       IPL_DEPTH_8U, frame->nChannels );  
       // Check the origin of image. If top left, copy the image frame to frame_copy.   
       if( frame->origin == IPL_ORIGIN_TL )  
         cvCopy( frame, frame_copy, 0 );  
       // Else flip and copy the image  
           for(int i=0;i<1;i++)  
           {  
        cascade = (CvHaarClassifierCascade*)cvLoad( cascade_name[i], 0, 0, 0 );  
        // Check whether the cascade has loaded successfully. Else report and error and quit  
        if( !cascade )  
             {  
                fprintf( stderr, "ERROR: Could not load classifier cascade\n" );  
                return -1;  
             }  
       // Call the function to detect and draw the face  
            if(detect_and_draw(frame_copy,cascade))  
           {  
                std::cout<<"Detected"<<std::endl;  
           }                                
           }  
       // Wait for a while before proceeding to the next frame  
       if( cvWaitKey( 1 ) >= 0 )  
        break;  
    }  
     // Release the images, and capture memory  
     cvReleaseHaarClassifierCascade(&cascade);  
     cvReleaseImage( &frame_copy );  
     cvReleaseCapture( &capture );  
     cvReleaseMemStorage(&storage);  
   }  
  return 0;  
 }  
 // Function to detect and draw any faces that is present in an image  
 bool detect_and_draw( IplImage* img,CvHaarClassifierCascade* cascade )  
 {  
   int scale = 1;  
   // Create a new image based on the input image  
   IplImage* temp = cvCreateImage( cvSize(img->width/scale,img->height/scale), 8, 3 );  
   // Create two points to represent the face locations  
   CvPoint pt1, pt2;  
   int i;  
   // Clear the memory storage which was used before  
   cvClearMemStorage( storage );  
   // Find whether the cascade is loaded, to find the faces. If yes, then:  
   if( cascade )  
   {  
     // There can be more than one face in an image. So create a growable sequence of faces.  
     // Detect the objects and store them in the sequence  
     CvSeq* faces = cvHaarDetectObjects( img, cascade, storage,  
                       1.1, 8, CV_HAAR_DO_CANNY_PRUNING,  
                       cvSize(40, 40) );  
     // Loop the number of faces found.  
     for( i = 0; i < (faces ? faces->total : 0); i++ )  
     {  
       // Create a new rectangle for drawing the face  
       CvRect* r = (CvRect*)cvGetSeqElem( faces, i );  
       // Find the dimensions of the face,and scale it if necessary  
       pt1.x = r->x*scale;  
       pt2.x = (r->x+r->width)*scale;  
       pt1.y = r->y*scale;  
       pt2.y = (r->y+r->height)*scale;  
       // Draw the rectangle in the input image  
       cvRectangle( img, pt1, pt2, CV_RGB(255,0,0), 3, 8, 0 );  
       cv::Mat image(img);  
       cv::Rect rect;  
       rect = cv::Rect(pt1.x,pt1.y,(pt2.x-pt1.x),(pt2.y-pt1.y));  
       roiImg = image(rect);  
         cv::imshow("roi",roiImg);       
 ///Send to arduino  
       detect_blink(roiImg);  
     }  
   }  
   // Show the image in the window named "result"  
   cvShowImage( "original_frame", img );  
   if(i > 0)  
           return 1;  
      else  
           return 0;  
   // Release the temp image created.  
   cvReleaseImage( &temp );  
 }  


Python code : send_arduino.py

Which will execute when both eyes are closed

 #/usr/bin/python  
 print "Eyes closed from Python script"  


Execution bash script compile.sh

 g++ blink_detection.cpp `pkg-config opencv --cflags --libs` -o blink_detect  


Video




Code Download

34 comments:

vijayalaxmi biradar said...

Hello this post is very informative. i am also working on eye detection using opencv i am able to detect eyes but how to find eye open and close. this is good post but when i followed the procedure which u gave i could not get video displayed will u please help me.

vijayalaxmi
laxmi214@yahoo.co.in

Lentin Joseph said...

@Lakshmi, Can you share the error you getting ?

Tiagow said...

Hello. This is really good stuff, gratz.
I am also doing something like this but in JAVA and it's kind of hard because i can't find enough materials about this subject in this language.
I was able to detect the face and eyes of the user but just using a picture retrieved from my webcam.
For some reason I can't do it as a Video and because of that i'm not able do progress and detect the blinks of the eye. If you could help me in anyway i'd be very grateful. Thanks

Lentin Joseph said...

Hi

Actually i am doing a match template process ie taking an image and compared with the camera frames. There is a function in opencv for doing this process called match template function. If you can find a match template function in java, u can do this thing

Tiagow said...

Yeah, I tried that. But my problem is that i can't find the correspondent funtion in java. Like "cvClearMemStorage" or "cvCreateImage", for example. And when i search the documentation there's just nothing about it, and if there is.. the examples are just in c++ or python and i really dont know how to do it in java because the parameters and stuff are really different. It makes things hard.

adityonoegroho said...

Can it implemented in Visual Studio 2010?

Lentin Joseph said...

Yes, You can do it

MrFred said...

Hi, thanks for share this code!
I downloaded the code for university project but when I run the program I receive this error from try-catch:
OpenCV Error: Assertion failed ((img.depth() == CV_8U || img.depth() == CV_32F) && img.type() == templ.type()) in matchTemplate, file C:/slave/WinInstallerMegaPack/src/opencv/modules/imgproc/src/templmatch.cpp, line 249
An exception occued
Detected

How can I solve?

Firi Musro said...

I'm using visual studio 10 to implement this code, which section that I have to eliminate to amke it works on visual studio?

Anonymous said...

Can you tell me which version of opencv are you using because i am having some difficulty with libraries in the code while compiling using the 2.4.6 version

Ehsan Ab said...

Just like Firi Musro, i'm trying to implement the code on visual studio, do i have to eliminate some part of the code or change it ?
Thanks

Anonymous said...

Dear Brothers & Sisters, I am trying to implement this code on OpenCV prela 1.1 & Visual studio 2010. but i am facing some problem with library functions.please tell me which part i have to change in this code.please send me correct code so that it executes in OpenCV pre1.1 please help me.
shakela_cse@yahoo.com

Anonymous said...

please send the executable code of EYE BLINK DETECTION & TRACKING in OpenCV pre1.1 and send it to my inbox.I really need your help. please help me to accomplish this project. my mail id is: shakela_cse@yahoo.com

Lentin Joseph said...

Hi

It is build using opencv 2.4.3. It may not be work with opencv 1.1

Natasa Angelidou said...
This comment has been removed by the author.
eXp said...

Hi,

First of all, thanks for sharing your work on this.

I've tried your code and doesn't seem very accurate. It appears to be able to detect opened or closed eyes randomly. I tested it several times and still doubt whether this is correctly detecting anything but just my eyes. I guess it depends on light intensity, quality of the picture, etc. Have you measured the sensitivity of this implementation varying these conditions? How can these features be improved?

Would it be possible to count the number of blinks in a period of time?

Lentin Joseph said...

Hi eXp
The accuracy depends on following things

1. Light intensity
2. Matching template images

If you choose good lighting conditions and take a match template image of your eye instead me, it will give better accuracy

Ginson Mathew said...

Hi Lentin,

Thanks for the code.
I tried the code but nothing was getting displayed in the result window. Do we need to add blink detect file? Is send_arduino.py really required for this program.
If yes, how to add these files to visual studio 2010.

If possible can you mail me the complete project to my mail id ginson.gm@gmail.com or ginson.mathew@yahoo.com.

Thanks a billion Lentin!!!

Lentin Joseph said...

Hi Ginson
Before executing the code you need to give some command line arguments to this application, its like

blink_detect.exe 2.jpg 3.jpg

so blink_detect.exe is the application and 2.jpg and 3.jpg are open and closed eye images. I think i have attached a sample image for download

Anonymous said...

hi,
the code compilation is fine but when i am executing the cod it is not opening my webcam. could u pls tel me wht is the problem.

thank you

Harish Babu said...

Hi,
I have installed open cv 2.4.7 but not able to perform face detect program, when executed it shows up an error like "error while loading shared libraries: libopencv_core.so.2.4: cannot open shared object file: No such file or directory".Can u please help me a bit

Lentin Joseph said...

Hi Harish
Can you compile some simple opencv program other than this code, i think the installation of opencv is not properly done.

Harish Babu said...

Hi,
I got that right.But when i executed your program.I found an error after executing the command " ./blink_detect 2.jpg 3.jpg ", the error was "Could not load classifier cascade". Why is it like that?

Richard Tenorio said...

Hello sir, can i ask question about implementing your program in raspberry pi with its own camera? is it possible? what should i replace? thanks..

Lentin Joseph said...

Hi Richard
You can implement using rpi cam, i didnt tried yet, i think you can try the following link

http://thinkrpi.wordpress.com/2013/05/22/opencv-and-camera-board-csi/

Sweety Ramnani said...

Hey Lentin ,
I work on python , Can you tell me how to make this program run on to python Please.


Thanks,
Sweety

Lentin Joseph said...

Hi Sweety
The concepts of this program is like this

1. Detect eyes from camera frame using haard cascade
2. Extract the ROI of 2 eyes
3. Do template matching algorithm with this two extracted frame with already stored images of close and open images of eye

4. If it detect closing and open, it detect as blink

This method is not very accurate, but it will work ..

Anonymous said...

Hi Lentin,
I couldn't find the templates i.e 2 and 3.jpg here. Can you please upload them or send me on my id snaz303@gmail.com

Thanks,

Lentin Joseph said...

Hi
The 2.jpg and 3.jpg are open and closed images of your eyes. You can crop the eye area from the camera frame and give the input to this code

Harish Babu said...

Hi,
I did run your program in raspberry pi.. but i used a normal zebronics cam....n there is considerable lag...if i use a rpi camera module..whether this problem be solved?...

Lentin Joseph said...

Hi Harish
You can run this code in raspberry pi with some modification, using rpi camera it will work better than usb camera

Harish Babu said...

Hi,
Modification? if i connect the rpi camera module to CSI connector in raspberry pi and then run the code will it be taking the video?.. or should I include some extra instruction for that in the program..?

Harish Babu said...

Hi,
I tried with a Logitech C310 on Raspberry Pi, it did gave a decent output...eye portion is getting extraced.. n eye open or eye closed command is also appearing on the terminal..but there is slight delay..the one which I experienced in a pentium 4 laptop... :) .. any suggestions to improve the performance?

Lentin Joseph said...

Hi
What is the image resolution? May be u can reduce the image resolution for getting the performance

Post a Comment