ID card Border detection and image processing using Emgu CV

 The article talks about different ways of image processing and the technology and issues which we faced while developing this solution for a client. Consider a scenario of an end-user application which processes different ID cards such as driving licenses, passports and other similar photo ID proof cards. There are many business scenario where this requirement fits such as online application for credit/debit cards, online bank forms etc. Normally in such cases, a user takes a photo of their ID cards and uploads it. More often than not, photo gets captured awkwardly resulting in a lot of white spaces on each side of the ID card. Sometimes the captured image is even skewed. Eventually, all these improper scans of ID proofs gets uploaded in the system and causes a hindrance at the time of processing of application as they dont meet the pre-defined criterion. Therefore, in order to ensure that the images meets the pre-defined system criterion, certain level of image processing needs to be done on these images before they are uploaded.

So the requirement we had in hand was to:

  • ID card border detection automatically and the image should then be cropped
    Remove if any skew found
    Ensure it gets rotated or the orientation be corrected.
    Image Processing

For any kind of Image processing requirement, OpenCV is one of the widely used libraries and is built on the open source platform. Its mainly developed in C++ and in order to use it, a separate component is required to be implemented which should run independently. In order to use OpenCV in other languages, there are several wrappers available.

EmguCV is one such cross platform wrapper for .NET compatible languages (C#, VB, VC++ etc). Its an open source wrapper and can also be compiled for Xamarin and Unity applications along with usual .NET applications.

This blog will cover some of the APIs provided by EmguCV and how it can be used for border detection, image cropping and checking for skew. Emgu CV library is available as a nuget package and can be installed using following command:
Install-Package EmguCV -Version 3.1.0.1

Pre-processing

To start processing on image there are few pre-processing steps which are required to performed. The first step of pre-processing is to convert the image to greyscale and reduce the noise.

//Convert the image to grayscale and filter out the noise
UMat uimage = new UMat();
CvInvoke.CvtColor(img, uimage, ColorConversion.Bgr2Gray);

//use image pyr to remove noise
UMat pyrDown = new UMat();
CvInvoke.PyrDown(uimage, pyrDown);
CvInvoke.PyrUp(pyrDown, uimage);

Here PyrDown method downsamples the image and reduces the noise then by using PyrUp method performs up-sampling, thus enlarging the image. This improves the overall image quality as well as the size of image.

Next step for pre-processing is edge detection using canny algorithm.
UMat boundary = new UMat();
CvInvoke.Canny(uimage, boundary, cannyThreshold, cannyThresholdLinking);
CvInvoke.Canny(boundary, cannyEdges, cannyThreshold, cannyThresholdLinking);

Below is the output of this

Below is the output of this

Gray scale image
Gray scale image

Now that everything in image is marked with lines, it becomes easy to detect the most outer lines. Our image is now ready for edge detection.

Border detection

As we need to detect the borders of ID cards, we can assume that it be rectangular. Going by this assumption there can be 2 ways of detecting borders and both can be used in conjunction.

1. Detect outer most lines

Since Canny algorithm already converted our image into collection of lines, if we can find the outermost lines then we may be able to find the border. Here we used probabilistic Hough transform to filter out lines based on width & length. From Hough transform we get a collection of lines, which we can iterate to find out points at extreme positions of the image

LineSegment2D[] lines = CvInvoke.HoughLinesP(cannyEdges,
1, //Distance resolution in pixel-related units
Math.PI / 180.0, //Angle resolution measured in radians.
80, //threshold
2, //min Line width
20);
if (null != lines && lines.Length > 0)
{
PointF[] pts = new PointF[]
{
new PointF(lines[0].P1.X, lines[0].P1.Y),
new PointF(lines[0].P1.X, lines[0].P1.Y),
new PointF(lines[0].P1.X, lines[0].P1.Y),
new PointF(lines[0].P1.X, lines[0].P1.Y)
};
foreach (LineSegment2D line1 in lines)
{
pts[0].X = pts[0].X < line1.P1.X ? pts[0].X : line1.P1.X; pts[0].Y = pts[0].Y > line1.P1.Y ? pts[0].Y : line1.P1.Y;

pts[1].X = pts[1].X < line1.P1.X ? pts[1].X : line1.P1.X;
pts[1].Y = pts[1].Y < line1.P1.Y ? pts[1].Y : line1.P1.Y; pts[2].X = pts[2].X > line1.P2.X ? pts[2].X : line1.P2.X;
pts[2].Y = pts[2].Y < line1.P2.Y ? pts[2].Y : line1.P2.Y; pts[3].X = pts[3].X > line1.P2.X ? pts[3].X : line1.P2.X;
pts[3].Y = pts[3].Y > line1.P2.Y ? pts[3].Y : line1.P2.Y;
}
outline = CvInvoke.MinAreaRect(pts);
}

 

Id card with outer boundary
Id card with outer boundary

The above code snippet can give the coordinates of the lines which are at the extreme position in our image using which we can form a rectangle and that will be our border. But this technique wont work in all the scenarios such as ID card having more prominent rectangle drawn on it or image with white background of a white ID card. These cases wont give a correct collection of lines, thus may result in incorrect border detection. To overcome this problem we can go for another similar solution.

ID card border detection
ID card border detection

2. Detect outer most points

In this approach instead of finding lines, we will find objects at outer most location using FindContours. Contours are the curves joining continuous points with same color and intensity. As we have already applied Canny edge detection, all the objects are marked as lines, thus best suited for FindContours. Its always recommended to use FindContours on binary image.

using (VectorOfVectorOfPoint contours = new VectorOfVectorOfPoint())
{
CvInvoke.FindContours(cannyEdges, contours, null, RetrType.List, ChainApproxMethod.ChainApproxSimple);
.
.
}

Once we get the list of contours, we can filter the contours based on the area to further filter out noise.

for (int i = 0; i < count; i++)
{
using (VectorOfPoint contour = contours[i])
{
using (VectorOfPoint approxContour = new VectorOfPoint())
{
CvInvoke.ApproxPolyDP(contour, approxContour, CvInvoke.ArcLength(contour, true) * 0.05, true);

if (CvInvoke.ContourArea(approxContour, false) < MIN_CONTOUR_AREA) //only consider contours with area greater than 250
continue;
}
var newRect = CvInvoke.MinAreaRect(contour);
var newRectVerices = newRect.GetVertices();
pts[0].X = pts[0].X < newRectVerices[0].X ? pts[0].X : newRectVerices[0].X; pts[0].Y = pts[0].Y > newRectVerices[0].Y ? pts[0].Y : newRectVerices[0].Y;

pts[1].X = pts[1].X < newRectVerices[1].X ? pts[1].X : newRectVerices[1].X;
pts[1].Y = pts[1].Y < newRectVerices[1].Y ? pts[1].Y : newRectVerices[1].Y; pts[2].X = pts[2].X > newRectVerices[2].X ? pts[2].X : newRectVerices[2].X;
pts[2].Y = pts[2].Y < newRectVerices[2].Y ? pts[2].Y : newRectVerices[2].Y; pts[3].X = pts[3].X > newRectVerices[3].X ? pts[3].X : newRectVerices[3].X;
pts[3].Y = pts[3].Y > newRectVerices[3].Y ? pts[3].Y : newRectVerices[3].Y;
}
}
outline = CvInvoke.MinAreaRect(pts);

Using this approach, we can get coordinates of most outer points of the image using which we can create a border.

Post processing

Once border gets rightly detected, we can accordingly crop the image along the border and thus get a processed image containing only the ID card. If ID card is skewed i.e. rotated by some angle, border will also be rotated by the same angle and since we are creating object of RotatedRectangle it also contains the angle. So, when we crop the image using the rotatedRectangle it removes that angle from the image as well.

var processed = preProcessedImg.Copy(boundary);

Conclusion

Since EmguCV is compatible to be used in Xamarin, Border detection can be used in creating Apps like CamScanner which detects the border and thus allow user to crop the unnecessary part from the image.

Thanks for taking time to read the post. Please let us know if you have any comments or get in touch for any further details you need by dropping us a message here