Please note, I am a complete beginner in computer vision and OpenCV(Java).
My objective is to identify parking signs, and to draw bounding boxes around them. My problem is that the four signs from the top (with red borders) were not identified (see last image). I am also noticing that the Canny edge detection does not capture the edges of these four signs (see second image). I have tried with other images, and got the same results. My approach is as follows:
Load the image and convert it to gray scale
Pre-process the image by applying bilateralFilter and Gaussian blur
Execute Canny edge detection
Find all contours
Calculate the perimeter with arcLength and approximate the contour with approxPolyDP
If approximated figure has 4 points, then assuming it is a rectangle hence adding the contour
Finally, draw the contours that has 4 points exactly.
Mat filtered = new Mat();
Mat edges = new Mat(src.size(), CvType.CV_8UC1);
Imgproc.cvtColor(src, edges, Imgproc.COLOR_RGB2GRAY);
Imgproc.bilateralFilter(edges, filtered, 11, 17, 17);
org.opencv.core.Size s = new Size(5, 5);
Imgproc.GaussianBlur(filtered, filtered, s, 0);
Imgproc.Canny(filtered, filtered, 170, 200);
List<MatOfPoint> contours = new ArrayList<MatOfPoint>();
Imgproc.findContours(filtered, contours, new Mat(), Imgproc.RETR_LIST, Imgproc.CHAIN_APPROX_SIMPLE);
List<MatOfPoint> rectangleContours = new ArrayList<MatOfPoint>();
for (MatOfPoint contour : contours) {
MatOfPoint2f dst = new MatOfPoint2f();
contour.convertTo(dst, CvType.CV_32F);
perimeter = Imgproc.arcLength(dst, true);
approximationAccuracy = 0.02 * perimeter;
MatOfPoint2f approx = new MatOfPoint2f();
Imgproc.approxPolyDP(dst, approx, approximationAccuracy, true);
if (approx.total() == 4) {
rectangleContours.add(contour);
Toast.makeText(reactContext.getApplicationContext(), "Rectangle detected" + approx.total(), Toast.LENGTH_SHORT).show();
}
}
Imgproc.drawContours(src, rectangleContours, -1, new Scalar(0, 255, 0), 5);
Very happy to get advice on how I could resolve this issue, even if it implies changing my stratergy.
What about starting with OCR, Tesseract, in order to recognize big "P" and other parking-related text patterns?
(Toast seems like Android: How can I use Tesseract in Android?
General Tesseract for Java: https://www.geeksforgeeks.org/tesseract-ocr-with-java-with-examples/ )
Another example, in Python, but see the preprocessing and other tricks and ideas for making the letters recognizable when the image has gradients, lower contrast, small fonts etc.: How to obtain the best result from pytesseract?
Also, there could be filtering by color, since the colors of the signs are known. The conversion to grayscale removes that valuable information, so finding the edges is OK, but the colors still can be used. E.g. split the colors to b,g,r and use each channel as grayscale and possibly boost it. The red and blue borders would stand out.
It seems the contrast around the red borders is too low, the blue signs are brighter compared to the black contour. If not splitting, before converting to grayscale, some of the color channels could be amplified anyway, like the red one.
Searching for big yellow/blue regions with low contrast, with text found, "P" etc. Tesseract has a function returning the boxes of the text that was found.
Also once you find a sign somewhere or a bar of signs and their directions, you could search there, vertically/horizontally.
You may search HoughLines as well, that may find the black border around the signs.
Calculate the perimeter with arcLength and approximate the contour
with approxPolyDP
If approximated figure has 4 points, then assuming it is a rectangle
hence adding the contour
IMO finding exactly 4 points (or after simplification of the polygon) is hard and may be not enough of an evidence, also there are round corners etc. if contours are compared directly.
The angles between the vertices and the distances matter - are the lines parallel (with some precision) etc.
The process could be iterative: gradually reducing the polygon detail, checking the area and perimeter, until the vertices reach 4 (or about that). If the area and perimeter don't change much (the ratio has to be found) after polygon aproximation (simplifying the round corners etc.), while the number of points in the contour gets reduced. I'd try also a comparison to the bounding box and the convex hull measurements etc.
If you need to only detect the parking signs, then treat this problem as a classic object detection problem (just like face detection). For the best results, you will need to use deep learning based convolutional neural network models.
To start with you can train the YOLO model which will give you a lot better results that anything you tried with OpenCV. You need at least 500 images. Then you need to annotate them. This tutorial is kick start tutorial on YOLO. Let's give a try.
Like YOLO there are so many models and all of them can be trained using similar process. So if you want to deploy your model on android, I will recommend you to choose a tensorflow based model. Train it on your PC and integrate the trained serialized model in your app.
Related
I'm trying to develop simple PC application for license plate recognition (Java + OpenCV + Tess4j). Images aren't really good (in further they will be good). I want to preprocess image for tesseract, and I'm stuck on detection of license plate (rectangle detection).
My steps:
1) Source Image
Mat img = new Mat();
img = Imgcodecs.imread("sample_photo.jpg");
Imgcodecs.imwrite("preprocess/True_Image.png", img);
2) Gray Scale
Mat imgGray = new Mat();
Imgproc.cvtColor(img, imgGray, Imgproc.COLOR_BGR2GRAY);
Imgcodecs.imwrite("preprocess/Gray.png", imgGray);
3) Gaussian Blur
Mat imgGaussianBlur = new Mat();
Imgproc.GaussianBlur(imgGray,imgGaussianBlur,new Size(3, 3),0);
Imgcodecs.imwrite("preprocess/gaussian_blur.png", imgGaussianBlur);
4) Adaptive Threshold
Mat imgAdaptiveThreshold = new Mat();
Imgproc.adaptiveThreshold(imgGaussianBlur, imgAdaptiveThreshold, 255, CV_ADAPTIVE_THRESH_MEAN_C ,CV_THRESH_BINARY, 99, 4);
Imgcodecs.imwrite("preprocess/adaptive_threshold.png", imgAdaptiveThreshold);
Here should be 5th step, which is detection of plate region (probably even without deskewing for now).
I croped needed region from image (after 4th step) with Paint, and got:
Then I did OCR (via tesseract, tess4j):
File imageFile = new File("preprocess/adaptive_threshold_AFTER_PAINT.png");
ITesseract instance = new Tesseract();
instance.setLanguage("eng");
instance.setTessVariable("tessedit_char_whitelist", "acekopxyABCEHKMOPTXY0123456789");
String result = instance.doOCR(imageFile);
System.out.println(result);
and got (good enough?) result - "Y841ox EH" (almost true)
How can I detect and crop plate region after 4th step? Have I to make some changes (improvements) in 1-4 steps? Would like to see some example implemented via Java + OpenCV (not JavaCV).
Thanks in advance.
EDIT (thanks to #Abdul Fatir's answer)
Well, I provide working (for me atleast) code sample (Netbeans+Java+OpenCV+Tess4j) for those who interested in this question. Code is not the best, but I made it just for studying.
http://pastebin.com/H46wuXWn (do not forget to put tessdata folder into your project folder)
Here's how I suggest you should do this task.
Convert to Grayscale.
Gaussian Blur with 3x3 or 5x5 filter.
Apply Sobel Filter to find vertical edges.
Sobel(gray, dst, -1, 1, 0)
Threshold the resultant image to get a binary image.
Apply a morphological close operation using suitable structuring element.
Find contours of the resulting image.
Find minAreaRect of each contour. Select rectangles based on aspect ratio and minimum and maximum area.
For each selected contour, find edge density. Set a threshold for edge density and choose the rectangles breaching that threshold as possible plate regions.
Few rectangles will remain after this. You can filter them based on orientation or any criteria you deem suitable.
Clip these detected rectangular portions from the image after adaptiveThreshold and apply OCR.
a) Result after Step 5
b) Result after Step 7. Green ones are all the minAreaRects and the Red ones are those which satisfy the following criteria: Aspect Ratio range (2,12) & Area range (300,10000)
c) Result after Step 9. Selected rectangle. Criteria: Edge Density > 0.5
EDIT
For edge-density, what I did in the above examples is the following.
Apply Canny Edge detector directly to input image. Let the cannyED image be Ic.
Multiply results of Sobel filter and Ic. Basically, take an AND of Sobel and Canny images.
Gaussian Blur the resultant image with a large filter. I used 21x21.
Threshold the resulting image using OTSU's method. You'll get a binary image
For each red rectangle, rotate the portion inside this rectangle (in the binary image) to make it upright. Loop through the pixels of the rectangle and count white pixels. (How to rotate?)
Edge Density = No. of White Pixels in the Rectangle/Total no. of Pixels in the rectangle
Choose a threshold for edge density.
NOTE: Instead of going through steps 1 to 3, you can also use the binary image from step 5 for calculating edge density.
Actually OpenCV has pre-trained model specially for Russian license plates: haarcascade_russian_plate_number
Also there is open source ANPR project for Russian license plates: plate_recognition. It is not use tesseract, but it has quite good pre-trained neural network.
You find all connected components (the white areas) and determine their outline.
If you filter them based on size (as part of the image), ratio (width-height) and white/black ratio to retrieve candidate-plates.
Undo the transformation of the rectangle
Remove the bolts
Pass in image to the OCR engine.
I'm trying to find a way to identify an archery target and all of its rings on a photo which might be made of different perspectives:
My goal is to identify the target and later on also where the arrows hit the target to automatically count their score. Presumptions are as follows:
The camera's position is not fixed and might change
The archery target might also move or rotate slightly
The target might be of different size and have different amount of circles
There might be many holes (sometimes big scratches) in the target
I have already tried OpenCV to find contours, but even with preprocessing (grayscale -> blur (-> threshold) -> edge detection) I still find a few houndred contours which are all distracted by the arrows or other obstacles (holes) on the target, so it is impossible to find a nice circular line. Using Hough to find circles doesn't work either as it will give me weired results as Hough will only find perfect circles and not ellipses.
With preprocessing the image this is my best result so far:
I was thinking about ellipse and circle fitting, but as I don't know radius, position and pose of the target this might be a very cpu consuming task. Another thought was about using recognition from a template, but the position and rotation of the target changes often.
Now I have the idea to follow every line on the image to check if it is a curve and then guess which curves belong together to form a circle/ellipse (ellipse because of the perspective). The problem is that the lines might be intersected by arrows or holes in a short distance so the line would be too short to check if it is a curve. With the smaller circles on the target the chance is high that it isn't recognised at all. Also, as you can see, circle 8, 7 and 6 have no clear line on the left side.
I think it is not neccessary to do perspective correction to achieve this task as long as I can clearly identify all the rings in the target.
I googled a long time and found some thesis which are all not exactly focussed on this specific task and also too mathematical for me to understand.
Is it by any chance possible to achieve this task? Could you share with me an idea how to solve this problem? Anything is very appreciated.
I'm doing this in Java, but the programming language is secondary. Please let me know if you need more details.
for starters see
Detecting circles and shots from paper target.
If you are using standardized target as on the image ( btw. I use these same too for my bow :) ) then do not cut off the color. You can select the regions of blue red and yellow pixels to ease up the detection. see:
footprint fitting
From that you need to fit the circles. But as you got perspective then the objects are not circles nor ellipses. You got 2 options:
Perspective correction
Use right bottom table rectangle area as marker (or the whole target). It is rectangle with known aspect ratio. so measure it on image and construct transformation that will change the image so it became rectangle again. There are tons of stuff about this: 3D scene reconstruction so google/read/implement. The basic are based just on De-skew + scaling.
Approximate circles by ellipses (not axis aligned!)
so fit ellipses to found edges instead circles. This will not be as precise but still close enough. see:
ellipse fitting
[Edit1] sorry did not have time/mood for this for a while
As you were unable to adapt my approach yourself here it is:
remove noise
you need to recolor your image to remove noise to ease up the rest... I convert it to HSV and detect your 4 colors (circles+paper) by simple tresholding and recolor the image to 4 colors (circles,paper,background) back into RGB space.
fill the gaps
in some temp image I fill the gaps in circles created by arrows and stuff. It is simple just scan pixels from opposite sides of image (in each line/row) and stop if hit selected circle color (you need to go from outer circles to inner not to overwrite the previous ones...). Now just fill the space between these two points with your selected circle color. (I start with paper, then blue,red and yellow last):
now you can use the linked approach
So find avg point of each color, that is approx circle center. Then do a histogram of radius-es and chose the biggest one. From here just cast lines out of the circle and find where the circle really stops and compute the ellipse semi-axises from it and also update the center (that handles the perspective distortions). To visually check I render cross and circle for each circle into the image from #1:
As you can see it is pretty close. If you need even better match then cast more lines (not just 90 degree H,V lines) to obtain more points and compute ellipse algebraically or fit it by approximation (second link)
C++ code (for explanations look into first link):
picture pic0,pic1,pic2;
// pic0 - source
// pic1 - output
// pic2 - temp
DWORD c0;
int x,y,i,j,n,m,r,*hist;
int x0,y0,rx,ry; // ellipse
const int colors[4]=// color sequence from center
{
0x00FFFF00, // RGB yelow
0x00FF0000, // RGB red
0x000080FF, // RGB blue
0x00FFFFFF, // RGB White
};
// init output as source image and resize temp to same size
pic1=pic0;
pic2=pic0; pic2.clear(0);
// recolor image (in HSV space -> RGB) to avoid noise and select target pixels
pic1.rgb2hsv();
for (y=0;y<pic1.ys;y++)
for (x=0;x<pic1.xs;x++)
{
color c;
int h,s,v;
c=pic1.p[y][x];
h=c.db[picture::_h];
s=c.db[picture::_s];
v=c.db[picture::_v];
if (v>100) // bright enough pixels?
{
i=25; // treshold
if (abs(h- 40)+abs(s-225)<i) c.dd=colors[0]; // RGB yelow
else if (abs(h-250)+abs(s-165)<i) c.dd=colors[1]; // RGB red
else if (abs(h-145)+abs(s-215)<i) c.dd=colors[2]; // RGB blue
else if (abs(h-145)+abs(s- 10)<i) c.dd=colors[3]; // RGB white
else c.dd=0x00000000; // RGB black means unselected pixels
} else c.dd=0x00000000; // RGB black
pic1.p[y][x]=c;
}
pic1.save("out0.png");
// fit ellipses:
pic1.bmp->Canvas->Pen->Width=3;
pic1.bmp->Canvas->Pen->Color=0x0000FF00;
pic1.bmp->Canvas->Brush->Style=bsClear;
m=(pic1.xs+pic1.ys)*2;
hist=new int[m]; if (hist==NULL) return;
for (j=3;j>=0;j--)
{
// select color per pass
c0=colors[j];
// fill the gaps with H,V lines into temp pic2
for (y=0;y<pic1.ys;y++)
{
for (x= 0;(x<pic1.xs)&&(pic1.p[y][x].dd!=c0);x++); x0=x;
for (x=pic1.xs-1;(x> x0)&&(pic1.p[y][x].dd!=c0);x--);
for (;x0<x;x0++) pic2.p[y][x0].dd=c0;
}
for (x=0;x<pic1.xs;x++)
{
for (y= 0;(y<pic1.ys)&&(pic1.p[y][x].dd!=c0);y++); y0=y;
for (y=pic1.ys-1;(y> y0)&&(pic1.p[y][x].dd!=c0);y--);
for (;y0<y;y0++) pic2.p[y0][x].dd=c0;
}
if (j==3) continue; // do not continue for border
// avg point (possible center)
x0=0; y0=0; n=0;
for (y=0;y<pic2.ys;y++)
for (x=0;x<pic2.xs;x++)
if (pic2.p[y][x].dd==c0)
{ x0+=x; y0+=y; n++; }
if (!n) continue; // no points found
x0/=n; y0/=n; // center
// histogram of radius
for (i=0;i<m;i++) hist[i]=0;
n=0;
for (y=0;y<pic2.ys;y++)
for (x=0;x<pic2.xs;x++)
if (pic2.p[y][x].dd==c0)
{
r=sqrt(((x-x0)*(x-x0))+((y-y0)*(y-y0))); n++;
hist[r]++;
}
// select most occurent radius (biggest)
for (r=0,i=0;i<m;i++)
if (hist[r]<hist[i])
r=i;
// cast lines from possible center to find edges (and recompute rx,ry)
for (x=x0-r,y=y0;(x>= 0)&&(pic2.p[y][x].dd==c0);x--); rx=x; // scan left
for (x=x0+r,y=y0;(x<pic2.xs)&&(pic2.p[y][x].dd==c0);x++); // scan right
x0=(rx+x)>>1; rx=(x-rx)>>1;
for (x=x0,y=y0-r;(y>= 0)&&(pic2.p[y][x].dd==c0);y--); ry=y; // scan up
for (x=x0,y=y0+r;(y<pic2.ys)&&(pic2.p[y][x].dd==c0);y++); // scan down
y0=(ry+y)>>1; ry=(y-ry)>>1;
i=10;
pic1.bmp->Canvas->MoveTo(x0-i,y0);
pic1.bmp->Canvas->LineTo(x0+i,y0);
pic1.bmp->Canvas->MoveTo(x0,y0-i);
pic1.bmp->Canvas->LineTo(x0,y0+i);
//rx=r; ry=r;
pic1.bmp->Canvas->Ellipse(x0-rx,y0-ry,x0+rx,y0+ry);
}
pic2.save("out1.png");
pic1.save("out2.png");
pic1.bmp->Canvas->Pen->Width=1;
pic1.bmp->Canvas->Brush->Style=bsSolid;
delete[] hist;
I want to detect the 4 corners of the document in android. Iam using opencv library.
Iam using the following approach-
(1) grayscale the image
(2) Apply median blur
(3) Apply adaptive threshold
(4) Canny Edge detection
(5) find contours
(6) find the largest contour
(7) get the edges and corners of the largest contour
My code is
srcImg = Utils.loadResource(this, R.drawable.test1, Highgui.CV_LOAD_IMAGE_COLOR);
Imgproc.cvtColor(srcImg, srcImg, Imgproc.COLOR_BGR2GRAY);
Imgproc.medianBlur(srcImg, srcImg, 9);
Imgproc.adaptiveThreshold(srcImg, srcImg, 255, Imgproc.ADAPTIVE_THRESH_MEAN_C, Imgproc.THRESH_BINARY_INV, 11, 2);
Imgproc.Canny(srcImg, srcImg, 50, 80);
Imgproc.findContours(srcImg,contours , hierarchy, Imgproc.RETR_TREE, Imgproc.CHAIN_APPROX_SIMPLE);
After canny edge detection, the outlining boundary at many points are broken due to which the largest contour detected in the image is not the actual boundary of the document. In some images the largest contour is showing some part of the boundary but in some images it is pointing to the center of the document or completely outside the document
(A)How do I proceed further in this condition ?
Another approach i applied was to apply houghes line transform and calculate the point of intersection of the lines which are intersecting at 90 degree angle but in that case am not getting exactpoints. when I put 88
(B)How to handle that much number of points
(C)or do I need to follow completely different approach ?
I'm trying to detect rectangles using OpenCV. However, sometimes this is getting pretty difficult after running the Canny method, because two of the edges are usually being erased out. I've tried many different sets of thresholds and blurring it before applying Canny, but I haven't got major positive results yet. Currently, I'm not blurring the image, so this is pretty much what I'm doing:
Mat imgSource = Highgui.imread(filepath);
Imgproc.Canny(imgSource, imgSource, 300, 600, 5, true);
Example:
original http://imagizer.imageshack.us/a/img822/8776/27i9j.jpg
Canny http://imagizer.imageshack.us/a/img841/9868/wkc95.jpg
Then, I'm trying OpenCV's findContours method to detect the rectangle, it works 80% of the time, how can I improve it?
Try with different threshold value, in this case you will get better result when using lower threshold values, like 10,100.
blur(src,src,Size(3,3));
cvtColor(src,tmp,CV_BGR2GRAY);
Canny( src, thr, 10, 100, 3 );
Or in another way you will get the contour images by applying threshold
like,
threshold(tmp,thr,50,255,THRESH_BINARY_INV);
the problem here is the image compression JPEG file type probably.
try converting image to monochrome since you only have Black/white image
and edit the threshold value. this should eliminate the noise around the edges of the lines.
then canny can be applied with any values.
i have the following picture and what i actually want to detect is the circles above the box with letter to the top left of each box. But the result is that it detects also some other circles. I have no idea why.
Image that I want to detect on:
http://imgur.com/8oKmhGp
This is what the result looks like:
http://imgur.com/qBw6YhK
As you can see it can find letters as circles sometimes and also the circles on the lego. Here is my code:
Mat source = Highgui.imread("testar.jpg", Highgui.CV_LOAD_IMAGE_COLOR);
Mat destination = new Mat(source.rows(), source.cols(), source.type());
Imgproc.cvtColor(source, destination, Imgproc.COLOR_RGB2GRAY);
Imgproc.GaussianBlur(destination, destination, new Size(3,3),0,0);
Mat circles = new Mat();
Imgproc.HoughCircles(destination, circles, Imgproc.CV_HOUGH_GRADIENT, 1, 20, 10, 20, 7, 13);
int radius;
Point pt;
for (int x = 0; x < circles.cols(); x++) {
double vCircle[] = circles.get(0,x);
if (vCircle == null)
break;
pt = new Point(Math.round(vCircle[0]), Math.round(vCircle[1]));
radius = (int)Math.round(vCircle[2]);
// draw the found circle
Core.circle(destination, pt, radius, new Scalar(0,255,255), 3);
Core.circle(destination, pt, 3, new Scalar(255,255,255), 3);
}
Highgui.imwrite("foundCircles.jpg", destination);
Well, IMHO, the Hough Circle detection algorithm is working exactly the way it is supposed to be. It IS detecting circles.
However, it seems like you do not want to detect the circles lying outside the area of the mobile phone's screen.
A simple solution can be implemented if you somehow manage to lay your hands on the exact coordinates of the four corners of the phone (or the mobile screen).
You can use the Rect class to define a rectangular block:
Rect cropRect = new Rect(topLeft_X, topLeft_Y, widthOfRectangle, heightOfRectangle);
and then use this rectangle object to reproduce a new image matrix (from the original one) that contains only the desired area:
Mat croppedImage = new Mat(inputImg, cropRect);
Now, with the freshly cropped image by your side, you can have all the fun you want with the algorithm of Mr. Paul Hough.
Now, if for some reason, it turns out that you do not have any clue about how to get the coordinates of the four corners of the phone (i.e, the phone moves around whimsically), OR you're damn irritated with the Hough circle detection reporting the O's and S's as circles, then you may try seeking the help of any good OCR implementation to help ease your pain.
Since you're on Java, you may use Tess4J. Or, you may try tweaking this project to extricate the position of the characters in the mobile screen. (There are many other OCRs which might help, please refer to this website for an exhaustive list)
Once you have the exact position of the characters, you may try running the Hough Circle detection block in the vicinity of the top left corner of the characters only.
One word of caution though, OCRs tend to be a wee bit nasty and unwieldy in Java.
If you're still unhappy with the results (or if OCRs seem to interfere with you metabolism), there's one last approach which you may try.... Hough Line detection.
Detect the lines, from the polar coordinates of the lines, estimate the grid that forms the keypad of the phone and then go around with detecting circles on the top left corner of the grids.