Develop Reference

OpenCV - find blackboard edges on video and images

UPDATE
You can find all the images I have for testing on my GitHub here:
GitHub repository with sources
There are also 2 videos, where the detection should work on as well
ORIGINAL QUESTION
I tried to use OpenCV 4.x.x to find the edges of a blackboard (image following), but somehow I cannot succeed. My code at the moment looks like this: (Android with OpenCV and live camera feed), where imgMat is a Mat from the camera feed:
Mat gray = new Mat();
Imgproc.cvtColor(imgMat, gray, Imgproc.COLOR_RGB2BGR);
Mat blurred = new Mat();
Imgproc.blur(gray, blurred, new org.opencv.core.Size(3, 3));
Mat canny = new Mat();
Imgproc.Canny(blurred, canny, 80, 230);
Mat kernel = Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new org.opencv.core.Size(2, 2));
Mat dilated = new Mat();
Imgproc.morphologyEx(canny, dilated, Imgproc.MORPH_DILATE, kernel, new Point(0, 0), 10);
Mat rectImage = new Mat();
Imgproc.morphologyEx(dilated, rectImage, Imgproc.MORPH_CLOSE, kernel, new Point(0, 0), 5);
Mat endproduct = new Mat();
Imgproc.Canny(rectImage, endproduct, 120, 230);
List<MatOfPoint> contours = new ArrayList<>();
Mat hierarchy = new Mat();
Imgproc.findContours(endproduct, contours, hierarchy, Imgproc.RETR_LIST, Imgproc.CHAIN_APPROX_SIMPLE);
double maxArea = 0;
boolean hasContour = false;
MatOfPoint2f biggestContour = new MatOfPoint2f();
Iterator<MatOfPoint> each = contours.iterator();
while (each.hasNext()) {
MatOfPoint wrapper = each.next();
double area = Imgproc.contourArea(wrapper);
if (area > maxArea) {
maxArea = area;
biggestContour = new MatOfPoint2f(wrapper.toArray());
hasContour = true;
}
}
if (hasContour) {
Mat output = imgMat.clone();
MatOfPoint2f approx = new MatOfPoint2f();
MatOfPoint poly = new MatOfPoint();
Imgproc.approxPolyDP(biggestContour, approx, Imgproc.arcLength(biggestContour, true) * .02, true);
approx.convertTo(poly, CvType.CV_32S);
Rect rect = Imgproc.boundingRect(poly);
}
Somehow I am not able to get it working, although the same code(written in python) worked on my computer with a video. I take the output from the rectangle and display it on my mobile screen, where it flickers around a lot and does not work properly.
These are my images I tried the python program on, and they worked:
What am I doing wrong? I am not able to constantly detect the edges of the blackboard.
Additional information about the blackboard:
always rectangular
may have different lighting
the text should be ignored, only the main board should be detected
the outer blackboard should be ignored as well
only the contour for the main board should be shown/returned
Thanks for any advice or code!

I used HSV because that's the easiest way to detect specific colors. I used an abundancy test to automatically select the color threshold (so this will work for green or blue boards). However, this test will fail on white or black boards since white and black count as all colors according to hue. Instead, in HSV, white and black are easiest to detect as very low saturation (white) or as very low value (black).
I did a 3-way check for each and selected the mask that had the most pixels in it (I assume that the boards are the majority of the image). I'm not sure how this will work on other images since we only have one here, so this may or may not work for other boards.
I used approxPolyDP to cut down on the number of points in the contour until I had 4 points and used that to draw the shape.
import cv2
import numpy as np
# get unique colors (to speed up search) and return the most abundant mask
def getAbundantColor(channel, margin):
# get uniques
unique_colors, counts = np.unique(channel, return_counts=True);
# check for the most abundant color
most = None;
biggest_count = -1;
for col in unique_colors:
# count number of white pixels
mask = cv2.inRange(channel, int(col - margin), int(col + margin));
count = np.count_nonzero(mask);
# if bigger, set new "most"
if count > biggest_count:
biggest_count = count;
most = mask;
return most, biggest_count;
# load image
img = cv2.imread("blackboard.jpg");
# it's huge, scale down so that we can see the whole thing
h, w = img.shape[:2];
scale = 0.25;
h = int(scale*h);
w = int(scale*w);
img = cv2.resize(img, (w,h));
# hsv
hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV);
h,s,v = cv2.split(hsv);
# median blur to get rid of most of the text
h = cv2.medianBlur(h, 5);
s = cv2.medianBlur(s, 5);
v = cv2.medianBlur(v, 5);
# get most abundant color
color_margin = 30;
hmask, hcount = getAbundantColor(h, color_margin);
# detect white and black separately
light_margin = 30;
# white
wmask = cv2.inRange(s, 0, light_margin);
wcount = np.count_nonzero(wmask);
# black
bmask = cv2.inRange(v, 0, light_margin);
bcount = np.count_nonzero(bmask);
# check which is biggest
sorter = [[hcount, hmask], [wcount, wmask], [bcount, bmask]];
sorter.sort();
mask = sorter[-1][1];
# dilate and erode to close holes
kernel = np.ones((3,3), np.uint8);
mask = cv2.dilate(mask, kernel, iterations = 2);
mask = cv2.erode(mask, kernel, iterations = 4);
mask = cv2.dilate(mask, kernel, iterations = 2);
# get contours # OpenCV 3.4, in OpenCV 2* or 4* it returns (contours, _)
_, contours, _ = cv2.findContours(mask, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE);
# for each contour, approximate a simpler shape until we have 4 points
simplified = [];
for con in contours:
# go until we have 4 points
num_points = 999999;
step_size = 0.01;
percent = step_size;
while num_points >= 4:
# get number of points
epsilon = percent * cv2.arcLength(con, True);
approx = cv2.approxPolyDP(con, epsilon, True);
num_points = len(approx);
# increment
percent += step_size;
# step back and get the points
# there could be more than 4 points if our step size misses it
percent -= step_size * 2;
epsilon = percent * cv2.arcLength(con, True);
approx = cv2.approxPolyDP(con, epsilon, True);
simplified.append(approx);
cv2.drawContours(img, simplified, -1, (0,0,200), 2);
# print out the number of points
for points in simplified:
print("Num Points: " + str(len(points)));
# show image
cv2.imshow("Image", img);
cv2.imshow("Hue", h);
cv2.imshow("Mask", mask);
cv2.waitKey(0);
Edit: In order to accommodate the uncertainty in the board's color and appearance I run the assumption that the board itself will be the majority of the picture. The lines involving the sorter are looking for the most abundant color in the image. If the white wall behind the board takes up more space in the image then that'll be the color that gets selected for the mask.
There are other ways to try and select just the board, but it's really difficult to come up with a catch-all solution. The rest of the code should do its job the same if you can come up with some way of masking the board. If you're willing to budge on the unknown color assumption and provide the original pictures of the failing cases then I can probably come up with an appropriate mask.

OpenCV remove scanning artifacts and rotate content

I was able to localize the content of the following image:
This is the current Java code:
Mat image = Imgcodecs.imread("test.png");
Mat gray = new Mat();
Imgproc.cvtColor(image, gray, Imgproc.COLOR_BGR2GRAY);
Core.absdiff(gray, new Scalar(255), gray);
Imgproc.threshold(gray, gray, 5, 255, Imgproc.THRESH_TOZERO);
Mat kernel1 = Imgproc.getStructuringElement(Imgproc.MORPH_ELLIPSE, new Size(11, 11));
Mat kernel2 = Mat.ones(3, 3, CvType.CV_8U);
Mat erosion = new Mat();
Imgproc.erode(gray, erosion, kernel2, new Point(-1, -1), 1);
Mat dilation = new Mat();
Imgproc.dilate(erosion, dilation, kernel1, new Point(-1, -1), 7);
final List<MatOfPoint> contours = new ArrayList<>();
final Mat hierarchy = new Mat();
Imgproc.findContours(dilation, contours, hierarchy,
Imgproc.RETR_TREE, Imgproc.CHAIN_APPROX_SIMPLE);
for (MatOfPoint contour : contours) {
RotatedRect rect = Imgproc.minAreaRect(new MatOfPoint2f(contour.toArray()));
Mat box = new Mat();
Imgproc.boxPoints(rect, box);
Imgproc.drawContours(image, contours, -1, new Scalar(0,0,255));
}
This is the resulting image:
As you may see - together with the useful content there are still a few scanning artifacts located with the red contours.
Is it possible to remove these scanning artifacts in some common way(that will work not only for this picture) without damage to content?
Also, how to properly rotate the content inside of this image(not the image itself) based on contours?

This problem can be treated as a Text Detection situation.
We can use some static image analysis:
Convert to Grey Scale
Apply Blurring/Smoothing
Threshold Image
Apply Morphological Dilation
Find Connected Components
Filter out components of small area
--
Gaussian Blur
Thresholding
Inverted Colors
Dilation
Detected Areas (after filtering) UPDATED
--
System.load("opencv_java320.dll");
Mat dst = new Mat();
Mat src = Imgcodecs.imread("path/to/your/image.png");
// Converting to Grey Scale
Imgproc.cvtColor(src, dst, Imgproc.COLOR_RGB2GRAY, 0);
// Blurring/Smoothing
Imgproc.GaussianBlur(dst, src, new Size(15.0,15.0),0.0,0.0);
// Thresholding / Binarization
Imgproc.threshold(src, dst, 150,255,Imgproc.THRESH_BINARY);
Mat painted = new Mat(); // UPDATED
src.copyTo(painted); // UPDATED
// Invert colors (helps with dilation)
Core.bitwise_not(dst,src);
// Image Dilation
Mat structuringElement = Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(55.0,55.0));
Imgproc.dilate(src, dst, structuringElement);
// Detect Text Areas
List<Rect> textBlocks = findTextBlocks(dst);
// Paint detected text areas
paintTextBlocks(textBlocks, painted);
static List<Rect> findTextBlocks(Mat dilated)
{
Mat labels = new Mat();
Mat stats = new Mat();
Mat centroids = new Mat();
// Find connected components
int numberOfLabels = Imgproc.connectedComponentsWithStats(dilated,labels,stats,centroids,8, CvType.CV_16U);
List<Rect> textBlocks = new ArrayList<>();
// adjust this threshold as your desire
double sizeThreshold = 0.01;
// Label 0 is considered to be the background label, so we skip it
for (int i = 1; i < numberOfLabels; i++)
{
// stats columns; [0-4] : [left top width height area}
Rect textBlock = new Rect(new Point(stats.get(i,0)[0],stats.get(i,1)[0]),new Size(stats.get(i,2)[0],
stats.get(i,3)[0]));
// stats.get(i,4)[0] is the area of the connected component / Filtering out small areas
if (Double.compare(stats.get(i,4)[0],dilated.height() * dilated.width() * sizeThreshold) > 0){
textBlocks.add(textBlock);
}
}
return textBlocks;
}
static void paintTextBlocks(List<Rect> textBlocks, Mat original)
{
for (Rect r : textBlocks)
{
Imgproc.rectangle(original, new Point(r.x,r.y), new Point(r.x+r.width,r.y+r.height),
new Scalar(100.0),2);
}
}
You can tune/adjust the following:
1) 3rd parameter of Imgproc.threshold method. Looking at the code it means that any pixel with color value higher of 150 will be replaced with 255 (white). Hence, increasing this number will result in getting fewer black/text pixels.
Decreasing the number will result in more black areas e.g. artifacts.
2) Size of Dilation structuring element (rectangle). Width and height should be the same and both odd numbers. Smaller dimensions of the structuring element means weaker dilation; smaller connected components. Larger dimensions means wider dilation with bigger connected components.
3) sizeThreshold in findTextBlocks() method. This variable controls the strength of the filtering of the connected components based on their size/area. Very small threshold will result in getting small areas e.g. artifacts and a big threshold will result in very big detected areas only.

OpenCV Java limit contours

I am using OpenCV in an Android application. I want the mobile application to automatically take a photo when a rectangle (something in the shape of a receipt for example) is in view. I am using Canny edge detection but when I am looking for contours, the array size is greater than 1500. Obviously it is not optimal to loop through all the contours and find the largest one so I was wondering is it possible to filter out the largest contour automatically through an api?
My code so far:
ArrayList contours;
#Override
public Mat onCameraFrame(final CameraBridgeViewBase.CvCameraViewFrame inputFrame) {
// Clear contours array on each frame
contours.clear();
// Get Grayscale image
final Mat gray = inputFrame.gray();
// Canny edge detection
Imgproc.Canny(gray, gray, 300, 1000, 5, true);
// New empty black matrix to store the edges captured
Mat dest = new Mat();
Core.add(dest, Scalar.all(0), dest);
// Copy the edge data over to the empty black matrix
gray.copyTo(dest);
// Is there a way to filter the size of contours so that not everything is returned? Right now this function is returning a lot of contours (1500 +)
Imgproc.findContours(gray, contours, hirearchy, Imgproc.RETR_LIST, Imgproc.CHAIN_APPROX_SIMPLE);
return dest;
}
EDIT
The user will be holding the phone and I want the application to automatically take a photo when the receipt is in view.
Example receipt

I have covered the basic techniques you may use, in the following Python code, it won't be hard to translate the code in the language of your choice, java in this case. So the technique involves:
Estimate the color of object you want to segment, which is white in your case, so safe limits for upper and lower bound can be approximated as:
RECEIPT_LOWER_BOUND = np.array([200, 200, 200])
RECEIPT_UPPER_BOUND = np.array([255, 255, 255])
Apply some Blur to input image to make the color distribution smooth, which would reduce the smaller contours in future.
img_blurred = cv2.blur(img, (5, 5))
Apply dilation to the binary image to remove the neighbouring smaller contours which surround your target largest contour
kernel = np.ones((10, 10), dtype=np.uint8)
mask = cv2.dilate(mask, kernel)
Now find contours in the mask after applying above operations and filter out the contour on the basis of contourArea.
im, contours, hierarchy = cv2.findContours(receipt_mask.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
largest_contour = max(contours, key=lambda x: cv2.contourArea(x))
Finally you may apply some threshold over the area to verify if the input was really a ticket or not.
Code:
import cv2
import numpy as np
# You may change the following ranges to define your own lower and upper BGR bounds.
RECEIPT_LOWER_BOUND = np.array([200, 200, 200])
RECEIPT_UPPER_BOUND = np.array([255, 255, 255])
def segment_receipt(img):
# Blur the input image to reduce the noise which in-turn reduces the number of contours
img_blurred = cv2.blur(img, (5, 5))
mask = cv2.inRange(img_blurred, RECEIPT_LOWER_BOUND, RECEIPT_UPPER_BOUND)
# Also dilate the binary mask which further reduces the salt and pepper noise
kernel = np.ones((10, 10), dtype=np.uint8)
mask = cv2.dilate(mask, kernel)
return mask
def get_largest_contour_rect(image):
receipt_mask = segment_receipt(image)
im, contours, hierarchy = cv2.findContours(receipt_mask.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
print "Number of contours found :", len(contours)
# Sorting the contours to get the largest one
largest_contour = max(contours, key=lambda x: cv2.contourArea(x))
# Return the last contour in sorted list as the list is sorted in increasing order.
return cv2.boundingRect(largest_contour)
image = cv2.imread("path/to/your/image.jpg")
rect = get_largest_contour_rect(image)
Output:

#J.Doe I am currently working on such a project and I have successfully being able to isolate the largest contour in the image after a whole lot of processing. The only part remaining is recognizing a rectangular contour and taking a picture.
mRgba = inputFrame.rgba();
Imgproc.Canny(mRgba,mCanny,50,200);
Imgproc.cvtColor(mRgba, mGray, Imgproc.COLOR_RGB2GRAY);
Imgproc.GaussianBlur(mGray, mGray1, new Size(3, 3), 1);
Mat kernel = Imgproc.getStructuringElement(Imgproc.MORPH_RECT,new Size(9,9));
Imgproc.dilate(mGray1, mGray2, kernel);
Imgproc.Canny(mGray2, mCanny, 50, 200);
Imgproc.findContours(mCanny,contours,hierarchy,Imgproc.RETR_TREE,Imgproc.CHAIN_APPROX_SIMPLE);
double maxVal = 0;
int maxValIdx = 0;
for(int contourIdx = 0; contourIdx < contours.size(); contourIdx++){
double contourArea = Imgproc.contourArea(contours.get(contourIdx));
if(maxVal < contourArea)
{
maxVal = contourArea;
maxValIdx = contourIdx;
}
}
Imgproc.drawContours(mRgba,contours,maxValIdx,new Scalar(0,255,255),-1);
return mRgba;
Be wary of the image names i changed them over different processes.

Want to find the Focal length first then distance of face detected in real time using opencv android

Formula for focal length is given below:
F = (P x D) / W
But I am unable to find pixel value (P) of the rectangle that appears on the detected face in real time:
Want to find the width of rectangle drawn around the mobile phone in the image:
It was done using Python and OpenCV but I am confused as to how to implement it in Java OpenCV.
http://www.pyimagesearch.com/2015/01/19/find-distance-camera-objectmarker-using-python-opencv/

In the image you added you have drawn a square around the phone so you already have the width of the square. What I understand from your question is that you want to get the real rectangle around the phone.
For this to be achieved there can be several solutions but one done through working with contours is like the following code:
// localImage would be the cropped image of the square you have drawn,
// the global image is the original image and phoneSquare is the Rect you
// have drawn
localImage = new Mat(globalImage, phoneSqure).clone();
// make the phone black and surroundings white
Imgproc.threshold(localImage, localImage, 127, 255, Imgproc.THRESH_OTSU + Imgproc.THRESH_BINARY_INV);
// get contours
ArrayList<MatOfPoint> contours = new ArrayList<>();
Mat hierarchy = new Mat();
Imgproc.findContours(canny, contours, hierarchy, Imgproc.RETR_LIST, Imgproc.CHAIN_APPROX_NONE);
// sort contours by size and get the biggest which is assumed to be the outer contour of the phone
contours.sort(new Comparator<MatOfPoint>() {
#Override
public int compare(MatOfPoint o1, MatOfPoint o2) {
return (int) Math.signum(o2.size().area() - o1.size().area());
}
});
MatOfPoints biggestContour = contours.get(contours.size() - 1);
// get the bounding rectangle of the phone, the you can get the width
Rect whatYouWant = Imgproc.boundingRect(biggestContour);

OpenCV how to find contour (Concave hull) for list of points - Java

I'm new to openCV and find it really difficult to do some even basic stuff so I'll really appreciate Your help.
My problem looks like that:
I have a list of points consisting of geo coordinates (latitude, longitude) in format as (49.074454444, 22.72638888889). Theses points form a polygon but a concave one and what I want to achieve is find concave hull of this polygon.
My idea is to achieve it with use of openCV by drawing this points than using some morphological transformations as dillatation and erosion so that the points form one solid area and than use a findContours method provided by openCV on it.
My first question would be is my approach right? I mean can it be achieved this way?
Now what is my main problem. First of all, all of my points differ only by some 6\th digit and I don't know how to properly "insert" them into Mat because as I believe Mat consists of pixels so rows and columns (ints).
I've tried to do something like this:
int matSize = 100;
Size size = new Size(matSize, matSize);
Mat src= Mat.zeros(size, CvType.CV_8UC1);
List<MatOfPoint> contours = new ArrayList<MatOfPoint>();
for(int i=0; i<listWithPoints.size(); i++){
Point point = new Point(listWithPoints.get(i).getLatitude(),listWithPoints.get(i).getLongitude());
MatOfPoint matOfPoint = new MatOfPoint(point);
contours.add(matOfPoint);
}
Imgproc.drawContours(src, contours, -1, new Scalar(255, 255, 255),1);
int dillatation_size = 5;
Mat element = Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(2*dillatation_size + 1, 2*dillatation_size +1));
Mat dst = Mat.zeros(size, CvType.CV_8UC1);
Imgproc.dilate(src, dst, element);
List<MatOfPoint> cnt = new ArrayList<>();
Mat hierarchy = new Mat();
Imgproc.findContours(dst, cnt, hierarchy,Imgproc.RETR_TREE, Imgproc.CHAIN_APPROX_SIMPLE);
But It doesn't work in fact after drawing contour only one point is marked as 255 and it's pixel (49,23) what makes sense.
My final goal is to get back geo coordinates of vertexes of my area hull.
I'll really be gratefull for any help.