Wednesday, March 25, 2015

Face detection with Raspberry Pi

One nice thing that can be done with the latest technology is a really powerful image analysis. Also using a small unit like a Raspberry Pi can be enough to create tasks such as face detection and recognition.

These two tasks are quite similar. Detection is just the task of finding a face in an image (could be static image or a frame from a video stream). Recognition is finding a known face in a picture. 

For this post I will show you how to configure a small face detection system using a RPi and a PiCamera.

As always, remember to keep your system updated.

For this task I will use OpenCV libraries. I think these are the best open source libraries available to perform image processing and they are not too complicated. Also they can be used directly with Python. To install them use:

sudo apt-get install libopencv-dev python-opencv python-dev 

This is enough to create a script for detecting a face in a saved image. If you wish to get the image directly from the PiCamera, then you should also install the python libraries (if you do not have already installed them):

sudo apt-get install python-picamera

Now look at the following code:

import cv2

#Load an image from file
image = cv2.imread("face.jpg", 1)

#Load a cascade file for detecting faces
face_cascade = cv2.CascadeClassifier('/usr/share/opencv/haarcascades/haarcascade_frontalface_alt.xml')

#Convert to grayscale
gray = cv2.cvtColor(image,cv2.COLOR_BGR2GRAY)

#Look for faces in the image using the loaded cascade file
faces = face_cascade.detectMultiScale(gray, 1.1, 5)

print "Found "+str(len(faces))+" face(s)"

#Draw a rectangle around every found face
for (x,y,w,h) in faces:
    cv2.rectangle(image,(x,y),(x+w,y+h),(255,0,0),2)

#Save the result image
cv2.imwrite('result.jpg',image)

First of all we load the picture we want to analyze. After that we need to tell OpenCV what kind of "feature" we want to find opening a cascade file. As we are looking for faces, we just use the right one for this feature.
In the opencv folder there are several ready-made cascade files, but you could also create one on your own (these will be mandatory for face recognition).

We could talk about cascades for days. For our scope you just need to know that Haar cascades are a bit slower (because they use floating point math), but more precise. LBP are faster (use integer math) but have less precision. It's up to you to choose the one that's right for you.
You just need to change the file specified in the CascadeClassifier function.

Let's go further. The next step is to convert the loaded picture into grayscale. This is not really needed, but will improve speed and detection rate.

After converting the image, we will perform the real process executing the cascade. As you can see it's quite simple!
Here you may need to change the middle parameter (here is 1.1) to make it perform better. This is a scale factor and can greatly affect the detecting process. Unfortunately there is no "perfect" value for this and could change for different pictures...

The process will return a list (empty if no faces have been found) with the coordinates of the rectangle surrounding the detected faces. The length of the list gives of course the number of found features (used in the print statement).

After printing the message, we get all the values from the list and draw every rectangle around the detected faces. Then the resulting picture is saved in a new file.

Here you can see an example with the original image and the result after the processing:


If you prefer to get the image directly from the PiCamera you need to modify the script like this:

import io
import picamera
import cv2
import numpy

#Create a memory stream so photos doesn't need to be saved in a file
stream = io.BytesIO()

#Get the picture (low resolution, so it should be quite fast)
#Here you can also specify other parameters (e.g.:rotate the image)
with picamera.PiCamera() as camera:
    camera.resolution = (320, 240)
    camera.capture(stream, format='jpeg')

#Convert the picture into a numpy array
buff = numpy.fromstring(stream.getvalue(), dtype=numpy.uint8)

#Now creates an OpenCV image
image = cv2.imdecode(buff, 1)

#Load a cascade file for detecting faces
face_cascade = cv2.CascadeClassifier('/usr/share/opencv/haarcascades/haarcascade_frontalface_alt.xml')

#Convert to grayscale
gray = cv2.cvtColor(image,cv2.COLOR_BGR2GRAY)

#Look for faces in the image using the loaded cascade file
faces = face_cascade.detectMultiScale(gray, 1.1, 5)

print "Found "+str(len(faces))+" face(s)"

#Draw a rectangle around every found face
for (x,y,w,h) in faces:
    cv2.rectangle(image,(x,y),(x+w,y+h),(255,0,0),2)

#Save the result image
cv2.imwrite('result.jpg',image)

What we are doing here is just capturing an image into a memory stream and converting it to a numpy stream so that we can use it for OpenCV. The processing part of the script is just the same as before.

Let's see another example with a bunch of faces (NASA Astronaut Group 18 from Wikipedia):


Face detection could be used for several scopes.
In an home automation system it could be useful to detect an intrusion. Instead of saving the result image, we could for example send the picture to our smartphone using Whatsapp, so we could personally check if we need to take further actions.

In a future post I will describe how to make face recognition on a Raspberry Pi.

55 comments:

  1. The Best Delphi sample code I ever had for Face Detection.
    GOD bless the good hearted Author.
    http://delphimagic.blogspot.com.es/2011/08/reconocimiento-de-caras-con-delphi.html

    ReplyDelete
    Replies
    1. Not exactly relatore to rpi, but seems quite useful!

      Delete
  2. When the author states, "If you prefer to get the image directly from the PiCamera..." do they mean like with the PiCamera in "video mode?" As in, it is taking a video? I am trying to figure out if it's better to use the PiCamera for video (essentially real-time) face detection or a normal webcam (think logitech or what-have-you).

    ReplyDelete
    Replies
    1. With PiCamera I'm just taking still photos (at least in the example above), but if you wish you can use a video stream from the picamera and get the single frames to check for face presence. An mjpeg video stream is just a sequence of jpg still images and there are many examples on the internet on how to retrieve them from a live stream.
      With python you could not get high framerates, but I think that 5-10fps can be achieved.

      Delete
    2. Hey Carlos, do you still plan on making a facial recognition tutorial for rpi? I hope you do; I'm doing a project on it.

      Delete
    3. Not in short times, sorry. Lately I'm busy with different things (and as you can see I'm not posting so often...). Anyway It's a thing I will surely do in the future and I will definitely post something about it when ready.

      Delete
    4. ok cool, thanks for taking the time to reply.

      Delete
    5. can you please help me in making this program real-time ? please i will be very grateful

      Delete
    6. You can visit https://www.pyimagesearch.com/ as this guy has tons of info on using OpenVC on the RPi and also gives several optimization tips for increasing the speed.

      Delete
  3. awesome tutorial!, but could also detect who is it???

    ReplyDelete
    Replies
    1. Not with this code, but it's surely possible! I saw someone that used face recognition to open a box only with his face.
      As I wrote on another comment, I'm planning to add face recognition, but not in short time...

      Delete
  4. I'm getting an error called 'Incorrect buffer length for resolution 320x240'

    ReplyDelete
    Replies
    1. Are you trying to do this more than one time? The script above should work correctly for one frame.
      If you wish to check several frames one after another, than you maybe should reset the "stream" buffer using the truncate method like this:

      stream.truncate(0)

      This statement should be put after you finished the process on an image (above it could be put after the saving of the image, at the end).

      This happens because PiCamera checks the stream size and if it's not the exact size of the frame it raises that error. After the first image, if the buffer is not cleared, the forther data is appended to the old one, so the buffer size just increase and the size check of course fails.

      Another option could be putting the capture routine inside a function and using a local variable for the stream buffer.

      Delete
  5. Hi your tutorial was great and so I decided to test it out for video instead of picture by using "camera.capture_continuous(rawCapture, format="rgb", use_video_port_True):"

    However I got 'Incorrect buffer length for resolution 320x240' which I probably need to add "truncate" to reset the buffer. I was wondering if I add under my "capture_continuous"

    cv2.imshow("Frame", image)
    key = cv2.waitKey(1) & 0xff
    rawCapture.truncate(0)
    rawCapture.seek(0)

    and after adding the 4 lines , proceed converting the rgb to grayscale and draw rectangle when face are detected will actually work.

    ReplyDelete
    Replies
    1. Your solution should be right. RPi camera driver has this kind of issue (also stated in the user manual) and the truncate option is what they suggest to use to avoid the problem.

      Delete
    2. Thanks for the fast reply. I just tried it again but no face was detected. The xml file is definitely working as I used your code to test it. The conversion is a success as well because the live feed was gray. The thing is when I end the process of the video, 'Incorrect buffer length for resolution 320x240' error appears again. Does that mean I had to add truncate again after the haar cascade command? This is so confusing as I'm not really sure if the errors actually obstruct the detection.

      Delete
    3. I searched a bit and you should do something like this:

      ....
      rawCapture = io.BytesIO()
      for foo in camera.capture_continuous(rawCapture, format='jpeg'):
      # Truncate the stream to the current position (in case
      # prior iterations output a longer image)
      rawCapture.truncate()
      rawCapture.seek(0)
      ....

      of course you should adapt to your needs

      Delete
    4. Incorrect buffer length appears when you use different resolution in rawCapture and camera.resolution(). Make sure you have same resolution in these two functions. Something like this:

      # initialize the camera and grab a reference to the raw camera capture
      camera = PiCamera()
      camera.resolution = (320, 240)
      camera.framerate = 32
      rawCapture = PiRGBArray(camera, size=(320, 240))

      I am able to do face detection with video instead of picture. Thanks Carlo for the great tutorial.

      The problem with the video face detection is the code is dame slow on my Rpi 3 even with the resolution of 160 X 120. Any idea how to increase the speed of the function face_cascade.detectMultiScale() function on raspberry pi. Guess GPU memory allocation for this process would increase the speed but no idea how to do it.

      Delete
    5. Yes, speed is not really big, but consider that you are using a really small computer.
      Anyway there are tricks to speed up things. You can search in the articles of the site: http://www.pyimagesearch.com/
      The owner wrote a couple of articles about this problem and found simple and effective solutions.

      Delete
  6. Please could you upload the code for face recognition also .Thank you for your amazing face detection code.

    ReplyDelete
  7. Or could you just give pseudo code for facial recognition in open cv on raspberry pi ?

    ReplyDelete
    Replies
    1. Sorry, but I do not have the code (yet). Anyway there are some projects for RPi on the internet that can be quite useful for you. Try seraching for "raspberry pi treasure box" and you should find one of the best abot this topic.

      Delete
  8. Great code piece. Helped me a lot.
    I wonder, how fast the above code runs on other systems (may I have something misconfigured) - my Raspi B+ runs almost 10sec on a single frame. I wonder how to achieve 8-10 fps as mentioned in an earlier post?

    ReplyDelete
    Replies
    1. 10 sec for every frame is really too much. Maybe you are using a too high resolution. For face detection you do not need hi-res and 320x240 is usually enough for that. Remember that doubling the resolution means the data is multiplied by 4.
      If your resolution is low enough, well there should be something elsewhere, but I really cannot tell you what could be the issue...
      Overclocking can be a bit useful to speed up things.

      Delete
  9. hey, Great work.!
    But I am gettin zero faces.
    output isnt working. I am doing the one on rpi camera. but it is not showing any image or any face detection.
    No error in the code.

    ReplyDelete
    Replies
    1. Do the PiCamera works? Try to just capture the image to a file and see if the image is correct. To save the image use camera.capture('file.jpg') instead of camera.capture(stream, format='jpeg')
      If it's ok, remember that the code above works for almost faces that are both frontal and vertical. Also light conditions can greatly affect the detection...

      Delete
    2. camera works fine. I use command raspistill -o name.jpg for image capture.
      Also , the code works if i load a saved image. I am just unable to detect faces in live cam. For an image, it shows, zero faces detected and for a live video it gives following error :
      OpenCV Error: Assertion failed (scn ==3 || scn == 4) in cvtColot, file /build/opencv-ISmtKH/opencv-2.4.9.1+dfsg/modules/imgproc/src/color.cpp, line 3737
      Traceback (most recent call last):

      Delete
    3. Hi, thanx for replying.
      pi camera works fine. I use command raspistill -o image.jpg to capture images. and it works fine.
      It is not working when i am trying to detect faces on live cam. It either shows zero faces or the following error :
      cv2.error: ..\..\..\opencv-2.4.10\modules\imgproc\src\color.cpp:3739: error: (-2

      15) scn == 3 || scn == 4 in function cv::cvtColor

      Please help.

      Delete
    4. Hey, it just worked.!!
      i reinstalled my os and started all over again. and it worked.
      Now could you help in detecting faces in video stream? I have been looking for it and unable to find any solutions. I want to detect faces in rpi camera module. no computer webcam.

      Delete
    5. The code above will detect faces from pi camera...
      Anyway I found a really nice website specific for RPi, Python and OpenCV. There are really tons of info about these topics.
      For getting frames from picamera video there is a nice tutorial there: http://www.pyimagesearch.com/2015/03/30/accessing-the-raspberry-pi-camera-with-opencv-and-python/

      Delete
  10. the code gives error "xlib: extension missing on display 10.0"
    what should i do to solve this?

    ReplyDelete
    Replies
    1. This is an X11 issue. Are you using a vnc software and launch the script under X11?
      Try to connect using SSH (so just a terminal) and launch the script from there, this should work.

      Anyway the error doesn't seems correct as it should also specicy what extension has the problem (usually it's RANDR).

      Delete
  11. could you please upload the code for multiple face detection and recognition using open cv from raspberry pi ,this is a part of my project and I have completed detection in open cv using raspberry pi but not recognition ,please help me out ..@carlo

    ReplyDelete
    Replies
    1. As I already wrote you above, I have no code about face recognition and this year I don't think I will have enough time to make something like that. However there are other projects for RPi on the web about face recognition, like the treasure box I suggested above...

      Delete
  12. Hey, what is the procedure for doing it in the USB camera?

    ReplyDelete
    Replies
    1. I never used an usb camera with a Raspberry Pi. Anyway there are several guides about using them, also with Python, on the internet.
      The only difference in the code above would be the image acquisition as I used what is specific for the PiCamera...

      Delete
  13. I am getting an error at the very beginning ('import cv2'). I am getting "ImportError: No module named 'cv2'"

    I do not understand this as I loaded the libraries!

    Thank you in advance for the help!

    ReplyDelete
    Replies
    1. Did you installed them as specified at the top of the article? If not, that's the problem. Maybe also some error during the install...

      Delete
    2. does the image have to be the in the same folder as the .py i am running?

      Delete
    3. Yes, unless you add a path to the filename.

      Delete
    4. Oh, man. Now I am getting the error:

      error: /build/opencv-ISmtkH/open-cv2.4.9.1+dfsg/modules/imgproc/src/color.cpp:37
      37: error: (-215) scn == 3 || 4 in function cvtColor


      Is this normal?

      Delete
    5. At which line of the script this happens? Is the image a jpg?

      Delete
    6. Line 7 in module.

      The image is a .jpg, yes.

      Delete
    7. So it happens when the script tryes to open the cascade classifier... is that file in the specified folder?
      If the file has been moved from /usr/share/opencv/haarcascades/haarcascade_frontalface_alt.xml
      in the current CV version maybe that's the problem. Try to find it. Eventually you can copy to the script folder and remove the path when opening it...

      Delete
  14. thanks so much for showing this directly on python.
    this has been very helpful.
    it was nice to see the first recognized image.
    took like 1-2 minutes though for the first try.

    keep up the good work dude

    ReplyDelete
  15. OpenCV Error: Assertion failed (!empty()) in detectMultiScale, file /home/pi/opencv-3.0.0/modules/objdetect/src/cascadedetect.cpp, line 1634
    Traceback (most recent call last):
    File "face2.py", line 29, in
    faces = face_cascade.detectMultiScale(gray, 1.1, 5)
    cv2.error: /home/pi/opencv-3.0.0/modules/objdetect/src/cascadedetect.cpp:1634: error: (-215) !empty() in function detectMultiScale

    getting this error please help

    ReplyDelete
  16. The above download command for OpenCV did not work for me. I modified it to
    $ sudo apt-get install ipython python-opencv python-scipy python-numpy python-setuptools python-pip
    and it worked fine.

    ReplyDelete
  17. In my project, i use raspberry pi2 and do: Human detection and alarm. So, how can i detect human with cameraPi?. May you help me? plz.

    ReplyDelete
    Replies
    1. If you do not need to discriminate between humans and animals there are libraries for doing this. These libraries usually make a difference between two frames from a camera and if there are too many different pixels, than there is something moving in front of the camera. Try searching the internet for something like "detect movement with raspberry pi"...

      Delete
  18. no module named picamera, I checked I previously installed picamera using sudo apt- get python picamera, and I have open cv version 2.7 installed. what should I do now?

    import error no module named picamera...

    please help me

    ReplyDelete
    Replies
    1. edited: raspistill -o image.jpg works for me...waiting for your reply

      Delete
    2. The module is separated from the system commands like raspistill. It seems that python-picamera has not been installed. Did you get any error while installing with apt-get?

      Delete
    3. no I didn't get any error, I install sudo apt-get install python-picamera
      it shows that previously installed.

      Delete
  19. Pls can I get a sample using video stream? Thanks

    ReplyDelete
    Replies
    1. Take a look at http://www.pyimagesearch.com/2015/03/30/accessing-the-raspberry-pi-camera-with-opencv-and-python/
      There also is a sample code for video got from the PiCamera. In case you want to get the stream from a webcam you need to acquire the single frames with something like this:

      req = urllib.urlopen('http://127.0.0.1:8080/?action=snapshot')
      arr = numpy.asarray(bytearray(req.read()), dtype=numpy.uint8)
      image = cv2.imdecode(arr,-1)

      The first line gets the jpeg frame (the right syntax for the address depends on the streaming source; this is just an example), the second line transforms the image into a numpy array and the third line creates an OpenCV image that can be processed.
      You need of course to import numpy and urllib libraries in the Python script.

      Delete