Real -Time Video Processing in iPhone

I was really impressed with the snapdragon features integrated to new Snapdragon enabled Android devices. It has really brought the changes in the market, specifically pointing the facial processing features. I just wanted to have a glance on how facial processing is been implemented. Before to lean on some framework i just had a quick glance at some of the SDKs and libraries available for facial processing. Found too many falling into this category, most with desktop implementation samples as-well.

Like OpenCv, SimpleCv, Faint and the list went long. I also found some list of commercial vendors that provide packages for facial recognition like Cybula, NeuroTechnology, Pittsburgh Pattern Recognition, Sensible Vision.

Freezing the requirement made me filter the choices. I just wanted to explore and find some library or framework doing the facial processing in iOS, may be not exactly as the snapdragon's facial processing for Android but something alternate to that. Before going for an open source or third party libraries its always wise looking at the native iOS SDK options. For any one who just need to identify the face real time or from an image CIDetector of the Core Image framework is an easy choice. Just a couple of line of code will get the work done. Have a look at the implementation below.

Have a look at the native implementation doc by apple to get this done here

But for anyone who need to do much more processing than just detection of face eyes, mouth, using a good open source libraries designed for specific purpose is a good choice.

OpenCV gives a framework for iOS that can be downloaded here. Also the clean and clear documentation of the API is also available in the CV website. They have also got sample implementation of desk top and mobile versions useful for any one to take a quick start.

The real time video processing makes you write some codes to capture the frames at the rate you need before to get processed. You have a wonderful apple documentation to get know the base and start here.

With the motive to detect face and eyes real time,  i just started with extending OpenCV sample for iOS found here.

Face detection is easily possible by using cascade classifier for Haar features. A cascade classifier basically tells OpenCV what to search in the image you specify. I used lbpcascade_frontalface classifier in my project to detect the face. There are too many cascade classifiers available in the internet. You can also train a cascade classifier tell OpenCV to recognize an object of our choice.

Extending the project i just detected the eyes in the reduced face area by just loading the haarcascade_lefteye_2splits classifier. But even though OpenCV is one of the fastest among the machine vision libraries, the speed is bottle neck when it comes to mobile devices. I found it even depended on the device capability. Your processing speed differ from an iPod to iPhone 4 to iPhone5. You can have a really faster processing on latest iPhone5. But we can have a little control over the processing of frames. Loading of more cascade for identifying face, left eye, right eye will slow down the process. Reducing the number of cascade loading will improve your speed. Also reducing the area of the image for search is also a best practice to improve your speed.

Reducing the search area is all about fixing your region of interest (ROI). You don't have to search an eye in lower half of the face. But to reduce the usage of classifies you got to have something to be done. Template matching is one thing that can be tried out. Ofcourse you will have the limitations and constraints.

But it really speeds up the processing. This is how i got it done.

 

Load the specific classifiers.

NSString *faceCascadePath = [[NSBundle mainBundle] pathForResource:@"lbpcascade_frontalface" ofType:@"xml"];

NSString *lefteyeCascadePath = [[NSBundle mainBundle]pathForResource:@"haarcascade_lefteye_2splits" ofType:@"xml"];

NSString *righteyeCascadePath = [[NSBundle mainBundle]pathForResource:@"haarcascade_lefteye_2splits" ofType:@"xml"];

if (!_faceCascade.load([faceCascadePath UTF8String])) {

NSLog(@"Could not load face cascade: %@", faceCascadePath);

}

if (!_leftEyeCascade.load([lefteyeCascadePath UTF8String])) {

NSLog(@"Could not load leftEye cascade: %@", lefteyeCascadePath);

}

if (!_rightEyeCascade.load([righteyeCascadePath UTF8String])) {

NSLog(@"Could not load rightEye cascade: %@", righteyeCascadePath);

}

Implement the delegate that captures the frame as cv::Mat. This is where your additional codes for processing is done.

cvtColor(image, image, CV_RGBA2GRAY);

std::vector<cv::Rect> faces;

_faceCascade.detectMultiScale(image, faces, 1.1, 2, kHaarEyeOptions, cv::Size(40, 40));

Loop through the result face array detected.

for (int i = 0; i < faces.size(); i++) {

std::vector<cv::Rect> lefteyes;

std::vector<cv::Rect> righteyes;

cv::Rect faceRect = faces[i];

Draw the face rect on screen.

cv::rectangle(image, faceRect.tl(), faceRect.br(), cvScalar(255,0,0));

 

CGRect eyearea_right = CGRectMake(faces[i].x +faces[i].width/16,(faces[i].y + (faces[i].height/4.5)),(faces[i].width - 2*faces[i].width/16)/2,( faces[i].height/3.0));

 

cv::Rect  eROI(eyearea_right.origin.x,eyearea_right.origin.y,eyearea_right.size.width,eyearea_right.size.height);

 

cv::Mat croppedReye = image(eROI);

cv::Mat cropReye = image(eROI).clone();

CGRect eyearea_left = CGRectMake(faces[i].x +faces[i].width/16 +(faces[i].width - 2*faces[i].width/16)/2,(faces[i].y + (faces[i].height/4.5)),(faces[i].width - 2*faces[i].width/16)/2,( faces[i].height/3.0));

 

cv::Rect 
eLOI(eyearea_left.origin.x,eyearea_left.origin.y,eyearea_left.size.width,eyearea_left.size.height);

 

cv::Mat croppedLeye = image(eLOI);

cv::Mat cropLeye = image(eLOI).clone();

Learn few frames to get the template of both the eyes, which is to be used further for template matching implementation.

if(learnTemplate<50){

_lEyeTemplate = [self getTemplate:croppedReye];

_rEyeTemplate = [self getTemplate:croppedLeye];

}

This function takes the reduced eye area and creates the templates for the eyes. The cascade is used to find the eyes in the reduced eyes areas left and right respectively. And from the area the iris spot is been located with simple MinMax calculation of OpenCV which is finding the darkest spot in an image. Further this area is cropped to get the exact eye area excluding the eyebrow and that is been saved as the eye templates.

-(cv::Mat)getTemplate:(const cv::Mat&)croppedEyearea{

std::vector<cv::Rect> eyes;

cv::Mat EyeTemplate;

cv::Mat temlMat;

_leftEyeCascade.detectMultiScale(croppedEyearea, eyes, 1.1, 2, kHaarEyeareaOptions, cv::Size(30, 30));

for(int i=0; i<eyes.size(); i++){

CGRect eye_only_rectangle = CGRectMake(eyes[i].tl().x,(eyes[i].tl().y + eyes[i].height*0.4),eyes[i].width,(eyes[i].height*0.6));

cv::Rect mylROI(eye_only_rectangle.origin.x,eye_only_rectangle.origin.y ,eye_only_rectangle.size.width,eye_only_rectangle.size.height);

temlMat = croppedEyearea(mylROI);

 

double minval, maxval;

cv::Point minloc, maxloc;

CGPoint iris ;

cv::minMaxLoc(temlMat, &minval, &maxval, &minloc, &maxloc);

iris.x = minloc.x + eye_only_rectangle.origin.x;

iris.y = minloc.y + eye_only_rectangle.origin.y;

 

cv::Rect newlROI((int)iris.x-12,(int)iris.y-12 ,30,30);

EyeTemplate = croppedEyearea(newlROI).clone();

}

learnTemplate ++;

return EyeTemplate;

}

 

So what happens after the learning of templates. We really dont have to load the eye classifiers any more instead we can use the template created for just finding the eyes in face frame. This speeds the processing after learning phase.

cv::Mat res(cropReye.rows-_lEyeTemplate.rows+1, cropReye.cols-_lEyeTemplate.cols+1, CV_32FC1);

 

cv::matchTemplate(cropReye, _lEyeTemplate, res, CV_TM_CCOEFF_NORMED);

 

double minval, maxval;

cv::Point minloc, maxloc;

cv::minMaxLoc(res, &minval, &maxval, &minloc, &maxloc);

cv::Point px(maxloc.x + eyearea_right.origin.x , maxloc.y+ eyearea_right.origin.y);

 

cv::Point py(maxloc.x + _lEyeTemplate.cols + eyearea_right.origin.x , maxloc.y + _lEyeTemplate.rows + eyearea_right.origin.y);

 

cv::rectangle(image, px,py, *new Scalar(255, 255, 0, 255));

 Do the same to find the other eye also.

So whats next is to really make something out of this results. Keep tuned to look at the next post with detail description on blink detection using OpenCV in iPhone.

iAd in iPhone app

This is a short blog post on integrating iAd feature into your iPhone application. So, for people aiming at money making from their applications have got their way either by putting a price tag on their app or integrating iAd framework. IOS 4 and later started supporting this feature through which user's application permits apple to push advertisements into the application and earn out from it. Apple pays 70% of what they get from the Add companies that pushes their adds into the application's addbanner.

Before to put your hands on keyboard to code, you have got something to do with your Apple account. Thats nothing but enabling iAd Network by filling up your contacts, Tax and Banking informations.

Once you are done with the above mentioned you can start with app development.

Reference the iAd framework:

For older versions of xCode you can Right click on the framework folder and select Add Exsisting framework>iAd framework and add it to application else you can open our buildphases expand Link Binary With Libraries tab and click on the plus symbol to open the existing framework list then add iAd framework to application.

You have got less to do with the designing part. If you wish using the IBOutlet just drag that and drop to the view where u would like to have a addbanner. If not you can just create a addbannerView via code as-well.

Drag and drop the AddBannerView IBOutlet into your view controller and so in your .h file you will have to declare the IBOutlet for instance adbannerView and map it to the control. In addition declare a boolean variable to store the banner's visible status. Add another subview (contentView) to the view to hold the entire contends to manage the appearance of the contents when the ad view scrolls into the screen whenever an ad is available.

Naturally your header file should appear as below

#import <UIKit/UIKit.h>

#import <iAd/iAd.h>

@interface ViewController : UIViewController<ADBannerViewDelegate>

@property (nonatomic,assign) BOOL bannerIsVisible;

@property(nonatomic,retain) IBOutlet ADBannerView *adbannerView;

@property(nonatomic,retain)IBOutlet UIView *contentView;

@end

You have to allocate the bannerview and added it to the subview. To do this add the following codes to the viewDidLoad in implementation file.

Note: The content size differs from the below for landscapeview.

adbannerView = [[ADBannerView alloc] initWithFrame:CGRectZero];

[adbannerView setRequiredContentSizeIdentifiers:[NSSet setWithObject: ADBannerContentSizeIdentifier320x50]];

[adbannerView setCurrentContentSizeIdentifier:ADBannerContentSizeIdentifier320x50];

[adbannerView setFrame:CGRectOffset([adbannerView frame], 0, -50)];

[self.view addSubview:adbannerView];

Now you have to add the following two delegates to the implementation file to handle the display of adds in the banner.

- (void)bannerViewDidLoadAd:(ADBannerView *)banner

{

if (!self.bannerIsVisible)

{

self.bannerIsVisible = YES;

[self setAdView:[UIDevice currentDevice].orientation];

}

}

- (void)bannerView:(ADBannerView *)banner didFailToReceiveAdWithError:(NSError *)error

{

if (self.bannerIsVisible)

{

self.bannerIsVisible = NO;

[self setAdView:[UIDevice currentDevice].orientation];

}

}

The function setAdView here is used to set the frame for iAd and display it when an add is available.

- (void)setAdView:(UIInterfaceOrientation)toInterfaceOrientation {

if (adbannerView != nil) {

[adbannerView setCurrentContentSizeIdentifier:ADBannerContentSizeIdentifier320x50];

[UIView beginAnimations:@"fixupViews" context:nil];

if (self.bannerIsVisible) {

CGRect adBannerViewFrame = [adbannerView frame];

adBannerViewFrame.origin.x = 0;

adBannerViewFrame.origin.y = 0;

[adbannerView setFrame:adBannerViewFrame];

CGRect contentViewFrame = _contentView.frame;

contentViewFrame.origin.y = 50;

contentViewFrame.size.height = self.view.frame.size.height -50;

_contentView.frame = contentViewFrame;

} else {

CGRect adBannerViewFrame = [adbannerView frame];

adBannerViewFrame.origin.x = 0;

adBannerViewFrame.origin.y = -50;

[adbannerView setFrame:adBannerViewFrame];

CGRect contentViewFrame = _contentView.frame;

contentViewFrame.origin.y = 0;

contentViewFrame.size.height = self.view.frame.size.height;

_contentView.frame = contentViewFrame;

}

[UIView commitAnimations];

}

}

If your application need to support landscape orientation you can add the  ADBannerContentSizeIdentifier480x32 for landscape with an if condition that check for the orientation. This is all what you have to do as the major part of development. Build and run the application. If add is available the banner will scroll in to the screen and rest of contents will shrink in the contentview.
You can download the sample app here.