Subscribe:

Ads 468x60px

Sunday, May 26, 2013

Eye blink detection using OpenCV in Ubuntu

Hi all

This is a simple project that i did recently. The purpose of this project is to detect the blinking of eyes. The application of this project is to detect whether the driver inside the car is sleeping or not.

Using this program, we can detect the eyes of the user and can track their eyes.  Using template matching technique, we will check whether the eyes are closed or not .. If the eyes are closed, it will print a message in terminal also execute a python script to pass control the event into external devices such as arduino or anyother boards

 Opencv code :blink_detection.cpp


Code Download





 // OpenCV Sample Application: blink_detction.cpp  
 // Include header files  
 #include "cv.h"  
 #include "highgui.h"  
 #include <stdio.h>  
 #include <stdlib.h>  
 #include <string.h>  
 #include <assert.h>  
 #include <math.h>  
 #include <float.h>  
 #include <limits.h>  
 #include <time.h>  
 #include <ctype.h>  
 #include <iostream>  
 #include "opencv2/objdetect/objdetect.hpp"  
 #include "opencv2/highgui/highgui.hpp"  
 #include "opencv2/imgproc/imgproc.hpp"  
 // Create memory for calculations  
 static CvMemStorage* storage = 0;  
 // Create a new Haar classifier  
 static CvHaarClassifierCascade* cascade = 0;  
 // Function prototype for detecting and drawing an object from an image  
 bool detect_and_draw( IplImage* image ,CvHaarClassifierCascade* cascade);  
 // Create a string that contains the cascade name  
 /*  "eyes.xml*/  
 const char *cascade_name[1]={"eyes.xml"};  
 cv::Mat roiImg;  
 int threshold_value = 200;  
 int threshold_type = 3;;  
 int const max_value = 255;  
 int const max_type = 4;  
 int const max_BINARY_value = 255;  
 int hough_thr = 35;  
 cv::Mat src_gray, dst;  
 using namespace cv;  
 Mat img1; Mat img2; Mat templ; Mat result;  
 const char* image_window = "Source Image";  
 const char* result_window = "Result window";  
 int match_method=0;  
 int max_Trackbar = 5;  
 int eye_open=0;  
 int eye_close=0;  
 /*  
 **  
  * @function MatchingMethod  
  * @brief Trackbar callback  
 */  
 //Matching with 2 images ,eye closed or open  
 void MatchingMethod(cv::Mat templ,int id )  
 {  
  /// Source image to display  
  cv::Mat img_display;  
  roiImg.copyTo( img_display );  
  /// Create the result matrix  
  int result_cols = roiImg.cols - templ.cols + 1;  
  int result_rows = roiImg.rows - templ.rows + 1;  
  result.create( result_cols, result_rows, CV_32FC1 );  
  /// Do the Matching and Normalize  
  cv::matchTemplate( roiImg, templ, result, match_method );  
  cv::normalize( result, result, 0, 1, NORM_MINMAX, -1, Mat() );  
  /// Localizing the best match with minMaxLoc  
  double minVal; double maxVal; Point minLoc; Point maxLoc;  
  cv::Point matchLoc;  
  cv::minMaxLoc( result, &minVal, &maxVal, &minLoc, &maxLoc, Mat() );  
  ///Justing checkin the match template value reaching the threashold  
  if(id == 0 && (minVal < 0))  
      {  
      eye_open=eye_open+1;  
      if(eye_open == 10)  
           {  
           std::cout<<"Eye Open"<<std::endl;  
           eye_open=0;  
           eye_close=0;  
           }  
      }  
   else if(id == 1 && (minVal < 0))  
      eye_close=eye_close+1;  
      if(eye_close == 10)  
           {  
           std::cout<<"Eye Closed"<<std::endl;  
           eye_close=0;  
           system("python send_arduino.py");  
           }  
  /// For SQDIFF and SQDIFF_NORMED, the best matches are lower values. For all the other methods, the higher the better  
  if( match_method == CV_TM_SQDIFF || match_method == CV_TM_SQDIFF_NORMED )  
   { matchLoc = minLoc; }  
  else  
   { matchLoc = maxLoc; }  
  /// Show me what you got  
  cv::rectangle( img_display, matchLoc, Point( matchLoc.x + templ.cols , matchLoc.y + templ.rows ), Scalar::all(0), 2, 8, 0 );  
  cv::rectangle( result, matchLoc, Point( matchLoc.x + templ.cols , matchLoc.y + templ.rows ), Scalar::all(0), 2, 8, 0 );  
  cv::imshow( image_window, img_display );  
  cv::imshow( result_window, result );  
  return;  
 }  
 void detect_blink(cv::Mat roi)  
 {  
      try  
      {        
      MatchingMethod(img1,0);  
       MatchingMethod(img2,1);  
      }  
      catch( cv::Exception& e )  
      {  
           std::cout<<"An exception occued"<<std::endl;  
      }  
 }  
 // Main function, defines the entry point for the program.  
 int main( int argc, char** argv )  
 {  
   if(argc <= 1)  
      {  
      std::cout<<"\n Help "<<std::endl;  
      std::cout<<"\n ------------------------------------\n"<<std::endl;  
      std::cout<<"./blink_detect open_eye.jpg close_eye.jpg\n"<<std::endl;  
     std::cout<<"Eg :: ./blink_detect 2.jpg 3.jpg\n"<<std::endl;  
      std::cout<<"\n ------------------------------------\n"<<std::endl;  
      exit(0);  
      }  
   // Structure for getting video from camera or avi  
   CvCapture* capture = 0;  
   // Images to capture the frame from video or camera or from file  
   IplImage *frame, *frame_copy = 0;  
   // Used for calculations  
   int optlen = strlen("--cascade=");  
   // Input file name for avi or image file.  
   const char* input_name;  
    img1 = imread( argv[1], 1 );  
    img2 = imread( argv[2], 1 );  
   // Load the HaarClassifierCascade  
    /// Create windows  
    cv::namedWindow( image_window, CV_WINDOW_AUTOSIZE );  
    cv::namedWindow( result_window, CV_WINDOW_AUTOSIZE );  
   // Allocate the memory storage  
   storage = cvCreateMemStorage(0);  
   capture = cvCaptureFromCAM( 0);  
   // Create a new named window with title: result  
   cvNamedWindow( "original_frame", 1 );  
   // If loaded succesfully, then:  
   if( capture )  
   {  
     // Capture from the camera.  
     for(;;)  
     {  
       // Capture the frame and load it in IplImage  
       if( !cvGrabFrame( capture ))  
         break;  
       frame = cvRetrieveFrame( capture );  
       // If the frame does not exist, quit the loop  
       if( !frame )  
         break;  
       // Allocate framecopy as the same size of the frame  
       if( !frame_copy )  
         frame_copy = cvCreateImage( cvSize(frame->width,frame->height),  
                       IPL_DEPTH_8U, frame->nChannels );  
       // Check the origin of image. If top left, copy the image frame to frame_copy.   
       if( frame->origin == IPL_ORIGIN_TL )  
         cvCopy( frame, frame_copy, 0 );  
       // Else flip and copy the image  
           for(int i=0;i<1;i++)  
           {  
        cascade = (CvHaarClassifierCascade*)cvLoad( cascade_name[i], 0, 0, 0 );  
        // Check whether the cascade has loaded successfully. Else report and error and quit  
        if( !cascade )  
             {  
                fprintf( stderr, "ERROR: Could not load classifier cascade\n" );  
                return -1;  
             }  
       // Call the function to detect and draw the face  
            if(detect_and_draw(frame_copy,cascade))  
           {  
                std::cout<<"Detected"<<std::endl;  
           }                                
           }  
       // Wait for a while before proceeding to the next frame  
       if( cvWaitKey( 1 ) >= 0 )  
        break;  
    }  
     // Release the images, and capture memory  
     cvReleaseHaarClassifierCascade(&cascade);  
     cvReleaseImage( &frame_copy );  
     cvReleaseCapture( &capture );  
     cvReleaseMemStorage(&storage);  
   }  
  return 0;  
 }  
 // Function to detect and draw any faces that is present in an image  
 bool detect_and_draw( IplImage* img,CvHaarClassifierCascade* cascade )  
 {  
   int scale = 1;  
   // Create a new image based on the input image  
   IplImage* temp = cvCreateImage( cvSize(img->width/scale,img->height/scale), 8, 3 );  
   // Create two points to represent the face locations  
   CvPoint pt1, pt2;  
   int i;  
   // Clear the memory storage which was used before  
   cvClearMemStorage( storage );  
   // Find whether the cascade is loaded, to find the faces. If yes, then:  
   if( cascade )  
   {  
     // There can be more than one face in an image. So create a growable sequence of faces.  
     // Detect the objects and store them in the sequence  
     CvSeq* faces = cvHaarDetectObjects( img, cascade, storage,  
                       1.1, 8, CV_HAAR_DO_CANNY_PRUNING,  
                       cvSize(40, 40) );  
     // Loop the number of faces found.  
     for( i = 0; i < (faces ? faces->total : 0); i++ )  
     {  
       // Create a new rectangle for drawing the face  
       CvRect* r = (CvRect*)cvGetSeqElem( faces, i );  
       // Find the dimensions of the face,and scale it if necessary  
       pt1.x = r->x*scale;  
       pt2.x = (r->x+r->width)*scale;  
       pt1.y = r->y*scale;  
       pt2.y = (r->y+r->height)*scale;  
       // Draw the rectangle in the input image  
       cvRectangle( img, pt1, pt2, CV_RGB(255,0,0), 3, 8, 0 );  
       cv::Mat image(img);  
       cv::Rect rect;  
       rect = cv::Rect(pt1.x,pt1.y,(pt2.x-pt1.x),(pt2.y-pt1.y));  
       roiImg = image(rect);  
         cv::imshow("roi",roiImg);       
 ///Send to arduino  
       detect_blink(roiImg);  
     }  
   }  
   // Show the image in the window named "result"  
   cvShowImage( "original_frame", img );  
   if(i > 0)  
           return 1;  
      else  
           return 0;  
   // Release the temp image created.  
   cvReleaseImage( &temp );  
 }  


Python code : send_arduino.py

Which will execute when both eyes are closed

 #/usr/bin/python  
 print "Eyes closed from Python script"  


Execution bash script compile.sh

 g++ blink_detection.cpp `pkg-config opencv --cflags --libs` -o blink_detect  


Video





68 comments:

Unknown said...

Hello this post is very informative. i am also working on eye detection using opencv i am able to detect eyes but how to find eye open and close. this is good post but when i followed the procedure which u gave i could not get video displayed will u please help me.

vijayalaxmi
laxmi214@yahoo.co.in

Lentin Joseph said...

@Lakshmi, Can you share the error you getting ?

Tiagow said...

Hello. This is really good stuff, gratz.
I am also doing something like this but in JAVA and it's kind of hard because i can't find enough materials about this subject in this language.
I was able to detect the face and eyes of the user but just using a picture retrieved from my webcam.
For some reason I can't do it as a Video and because of that i'm not able do progress and detect the blinks of the eye. If you could help me in anyway i'd be very grateful. Thanks

Lentin Joseph said...

Hi

Actually i am doing a match template process ie taking an image and compared with the camera frames. There is a function in opencv for doing this process called match template function. If you can find a match template function in java, u can do this thing

Tiagow said...

Yeah, I tried that. But my problem is that i can't find the correspondent funtion in java. Like "cvClearMemStorage" or "cvCreateImage", for example. And when i search the documentation there's just nothing about it, and if there is.. the examples are just in c++ or python and i really dont know how to do it in java because the parameters and stuff are really different. It makes things hard.

Anonymous said...

Can it implemented in Visual Studio 2010?

Lentin Joseph said...

Yes, You can do it

MrFred said...

Hi, thanks for share this code!
I downloaded the code for university project but when I run the program I receive this error from try-catch:
OpenCV Error: Assertion failed ((img.depth() == CV_8U || img.depth() == CV_32F) && img.type() == templ.type()) in matchTemplate, file C:/slave/WinInstallerMegaPack/src/opencv/modules/imgproc/src/templmatch.cpp, line 249
An exception occued
Detected

How can I solve?

Anonymous said...

I'm using visual studio 10 to implement this code, which section that I have to eliminate to amke it works on visual studio?

Anonymous said...

Can you tell me which version of opencv are you using because i am having some difficulty with libraries in the code while compiling using the 2.4.6 version

Unknown said...

Just like Firi Musro, i'm trying to implement the code on visual studio, do i have to eliminate some part of the code or change it ?
Thanks

Anonymous said...

Dear Brothers & Sisters, I am trying to implement this code on OpenCV prela 1.1 & Visual studio 2010. but i am facing some problem with library functions.please tell me which part i have to change in this code.please send me correct code so that it executes in OpenCV pre1.1 please help me.
shakela_cse@yahoo.com

Anonymous said...

please send the executable code of EYE BLINK DETECTION & TRACKING in OpenCV pre1.1 and send it to my inbox.I really need your help. please help me to accomplish this project. my mail id is: shakela_cse@yahoo.com

Lentin Joseph said...

Hi

It is build using opencv 2.4.3. It may not be work with opencv 1.1

Unknown said...
This comment has been removed by the author.
eXp said...

Hi,

First of all, thanks for sharing your work on this.

I've tried your code and doesn't seem very accurate. It appears to be able to detect opened or closed eyes randomly. I tested it several times and still doubt whether this is correctly detecting anything but just my eyes. I guess it depends on light intensity, quality of the picture, etc. Have you measured the sensitivity of this implementation varying these conditions? How can these features be improved?

Would it be possible to count the number of blinks in a period of time?

Lentin Joseph said...

Hi eXp
The accuracy depends on following things

1. Light intensity
2. Matching template images

If you choose good lighting conditions and take a match template image of your eye instead me, it will give better accuracy

Ginson Mathew said...

Hi Lentin,

Thanks for the code.
I tried the code but nothing was getting displayed in the result window. Do we need to add blink detect file? Is send_arduino.py really required for this program.
If yes, how to add these files to visual studio 2010.

If possible can you mail me the complete project to my mail id ginson.gm@gmail.com or ginson.mathew@yahoo.com.

Thanks a billion Lentin!!!

Lentin Joseph said...

Hi Ginson
Before executing the code you need to give some command line arguments to this application, its like

blink_detect.exe 2.jpg 3.jpg

so blink_detect.exe is the application and 2.jpg and 3.jpg are open and closed eye images. I think i have attached a sample image for download

Anonymous said...

hi,
the code compilation is fine but when i am executing the cod it is not opening my webcam. could u pls tel me wht is the problem.

thank you

Unknown said...

Hi,
I have installed open cv 2.4.7 but not able to perform face detect program, when executed it shows up an error like "error while loading shared libraries: libopencv_core.so.2.4: cannot open shared object file: No such file or directory".Can u please help me a bit

Lentin Joseph said...

Hi Harish
Can you compile some simple opencv program other than this code, i think the installation of opencv is not properly done.

Unknown said...

Hi,
I got that right.But when i executed your program.I found an error after executing the command " ./blink_detect 2.jpg 3.jpg ", the error was "Could not load classifier cascade". Why is it like that?

Unknown said...

Hello sir, can i ask question about implementing your program in raspberry pi with its own camera? is it possible? what should i replace? thanks..

Lentin Joseph said...

Hi Richard
You can implement using rpi cam, i didnt tried yet, i think you can try the following link

http://thinkrpi.wordpress.com/2013/05/22/opencv-and-camera-board-csi/

Unknown said...

Hey Lentin ,
I work on python , Can you tell me how to make this program run on to python Please.


Thanks,
Sweety

Lentin Joseph said...

Hi Sweety
The concepts of this program is like this

1. Detect eyes from camera frame using haard cascade
2. Extract the ROI of 2 eyes
3. Do template matching algorithm with this two extracted frame with already stored images of close and open images of eye

4. If it detect closing and open, it detect as blink

This method is not very accurate, but it will work ..

Anonymous said...

Hi Lentin,
I couldn't find the templates i.e 2 and 3.jpg here. Can you please upload them or send me on my id snaz303@gmail.com

Thanks,

Lentin Joseph said...

Hi
The 2.jpg and 3.jpg are open and closed images of your eyes. You can crop the eye area from the camera frame and give the input to this code

Unknown said...

Hi,
I did run your program in raspberry pi.. but i used a normal zebronics cam....n there is considerable lag...if i use a rpi camera module..whether this problem be solved?...

Lentin Joseph said...

Hi Harish
You can run this code in raspberry pi with some modification, using rpi camera it will work better than usb camera

Unknown said...

Hi,
Modification? if i connect the rpi camera module to CSI connector in raspberry pi and then run the code will it be taking the video?.. or should I include some extra instruction for that in the program..?

Unknown said...

Hi,
I tried with a Logitech C310 on Raspberry Pi, it did gave a decent output...eye portion is getting extraced.. n eye open or eye closed command is also appearing on the terminal..but there is slight delay..the one which I experienced in a pentium 4 laptop... :) .. any suggestions to improve the performance?

Lentin Joseph said...

Hi
What is the image resolution? May be u can reduce the image resolution for getting the performance

Anonymous said...

Can this be implemented using AVI files and OpenCV 2.4.7 or OpenCV 2.4.9 in Microsoft Visual Studio 2008 on Windows?

Lentin Joseph said...

Hi
Yea, its possible. But you need to change camera into video file. Other things will work fine i hope.

megz said...

Hi
We tried running this code in Visual Studio 2013. There are no compilation errors but it does not open the webcam and hence no output is seen. And on the output screen there are some garbage values printed. What do we do? Please help.
Thank you

megz said...

and also v r using on windows 7 os for this.will it work on that?

Lentin Joseph said...

Hi Megz
After building you have to copy eyes.xml and 2.jpg and 3.jpg into the exe path and take a cmd promt in that path. And execute like this

name_of_executable 2.jpg 3.jpg

In this 2.jpg is the opened eye and 3.jpg is the closed eye.. You can put ur own eye for better detection.

megz said...

Hi
Thanks a lot, your input helped. The output is showing "ERROR: Could not load classifier cascade". Is it that the openCV CvHaarClassifierCascade is not there in the openCV library that I downloaded? If yes, what should I do?

Anonymous said...

Hi im having issues with executing the program, whenever I try and execute, my video stream does not pop up and the shell returns this "init done"
"opengl support available"

Unknown said...
This comment has been removed by the author.
Unknown said...

Good day sir! We tried your code on Ubuntu and it worked fine but when we tried to run it on Raspberry Pi Model B running Raspbian with a R.Pi IR camera installed, we managed to install all the necessary libraries and get it to run without errors but it doesnt seem to be doing anything. The windows such as the rio, output, etc. and the view of the camera did not appear. Do you have any idea as to why nothing happened and what do you think we can do to make this work? Thank You.

Tabassum said...

hey lentin I'm trying to write a blink detection system and would just like to know whose eyes you used as a original template?

akiresu said...

Hi

Thank you very much for the code, but I need to implement it for the web (an HTML page), how can I do that?

Thanks again!

Have a great day!

Anonymous said...

Hi Lantin,
Can you explain, How to generate cascade xml file for eye blink detection? Can you explain which images should be positive and which should be negative images?

Anonymous said...

Hi;
Really would appreciate someone help me how can I run this nice code by raspicam.

Unknown said...

Hi.
It's nice.
I have a question.
is working with ras pi?
openCV + ras pi or just open CV?

Lentin Joseph said...

Hi

This code only work with opencv in PC.

Anonymous said...

sir I want a code in c++ ? ?

Anonymous said...

thanks

Unknown said...

Hi , I have few doubts ,will you please provide your mail ID.

Unknown said...

eyes.xml you have used is the one opencv provides ? haarcascades_eye.xml ?
or you trained that file?

Unknown said...

how to run the code in raspberry pi

Unknown said...

Hello, i have downloaded the zip file for above code.
I have installed opencv 2.4.10, visual studio 2010 n python 2.7.
What all extra things do I need to install so that this code works.
Please help me with this as I am a beginner in open cv.
Thank You.
Email ID: darshan.more@vit.edu.in

Anonymous said...

Urgent:
Header file missing #include "cv.h"
#include "highgui.h",.......solution??

Unknown said...
This comment has been removed by the author.
Unknown said...
This comment has been removed by the author.
Anonymous said...

its really gud and cngrts. am wrking for the same project...
can u explation the step by step procedure for code u hav done

Ramanand Bhat said...

I am new to this and am using vb 2013. I copied and pasted your code in VB .cpp page and there was only one error. cv::Mat image(img);. The error says: No instance of constructor "cv::Mat::Mat" matches the argument list argument types are :(iplimage). Can you please help me. I have not installed python yet.

Unknown said...

Hi i have the same problem as MrFred
OpenCV Error: Assertion failed ((img.depth() == CV_8U || img.depth() == CV_32F) && img.type() == templ.type()) in matchTemplate, file /build/buildd/opencv-2.4.8+dfsg1/modules/imgproc/src/templmatch.cpp, line 249
An exception occued

i'm using opencv 3

testblog said...

@Powi, you should downoad the complete code from the following link

https://bitbucket.org/lentinjoseph/opencv-blink-detection/downloads/detect_eye_bink.zip

Unknown said...
This comment has been removed by the author.
Unknown said...

Hi Mr. Lentin Joseph, my name is Samuel Susanto.
I want to ask something. I have run your code and i got :

HIGHGUI ERROR: V4L/V4L2: VIDIOC_S_CROP

Can you help me?

testblog said...

is your camera properly detecting on linux ?

Unknown said...

would you please put your the algorithm explanation

testblog said...

there are two images, one for open eye and one for closed eye. These two images will check with the camera frames using opencv match template algorithm. and check the detection count is > threshold then detected eye blink !!

Megana Babu said...

Hello sir,
I have the same issues as mentioned above.
Your program executes just fine without any errors. But at end it displays -
"init done"
"opengl support available" and terminates the program.
I would be glad if I could some assistance here.

Post a Comment