Develop Reference

Image preprocessing of water meter with OpenCv Java, OCR with Tesseract

I'm trying to develop simple application (OpenCv, Tesseract and Java) where i need to get numbers from a photo of water meter. I am newbie to OpenCV and i am stuck on detection of numbers in rectangles.
So i want to achieve "00295" value as result.
Here is a example of water meter
But i am not able to achieve this result.
Steps:
Apply Gray filter
GaussianBlur filter 3x3
Sobel filter Threshold
And doing OCR with number characters allowed only
But in result i get bunch of random numbers from other labels.
Can you please give some suggestions and show way how to detect this 5 rectangles and get digits from them ?
Thanks in advance.
Here is code:
private static final int
CV_THRESH_OTSU = 8;
public static void main(String[] args) {
System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
Mat img = new Mat();
Mat imgGray = new Mat();
Mat imgGaussianBlur = new Mat();
Mat imgSobel = new Mat();
Mat imgThreshold = new Mat();
//Path to picture
String inputFilePath = "D:/OCR/test.jpg";
img = Imgcodecs.imread(inputFilePath);
Imgcodecs.imwrite("preprocess/1_True_Image.png", img);
Imgproc.cvtColor(img, imgGray, Imgproc.COLOR_BGR2GRAY);
Imgcodecs.imwrite("preprocess/2_imgGray.png", imgGray);
Imgproc.GaussianBlur(imgGray,imgGaussianBlur, new Size(3, 3),0);
Imgcodecs.imwrite("preprocess/3_imgGaussianBlur.png", imgGray);
Imgproc.Sobel(imgGaussianBlur, imgSobel, -1, 1, 0);
Imgcodecs.imwrite("preprocess/4_imgSobel.png", imgSobel);
Imgproc.threshold(imgSobel, imgThreshold, 0, 255, CV_THRESH_OTSU);
Imgcodecs.imwrite("preprocess/5_imgThreshold.png", imgThreshold);
File imageFile = new File("preprocess/5_imgThreshold.png");
Tesseract tesseract = new Tesseract();
//tessdata directory
tesseract.setDatapath("tessdata");
tesseract.setTessVariable("tessedit_char_whitelist", "0123456789");
try {
String result = tesseract.doOCR(imageFile);
System.out.println(result);
} catch (TesseractException e) {
System.err.println(e.getMessage());
}
}
}

If the position of the water filter won't change from image to image, could you just manually crop the image to your desired size? Also, after you blur the image, try using an adaptive threshold followed by canny edge detection. As a result, your image will only have the hard edges present. Then you could find contours on the image and filter through those contours till they fit the desired size that you want.

How can I detect squares using opencv4.2 (Android)

I am detecting a rectangle and comparing the color to a urine test strip.
How can i detect all of squares? I want to detect the remaining squares in the picture below. I have tried changing the brightness and contrast
Here is my code:
MainActivity.java
...
#Override
protected void onActivityResult(int requestCode, int resultCode, Intent data) {
...
Bitmap img = BitmapFactory.decodeStream(in);
in.close();
Bitmap changeImg = changeBitmapContrastBrightness(img, (float)1, 10);
Mat cMap = new Mat();
Utils.bitmapToMat(changeImg, cMap);
List<MatOfPoint> squres = processImage(cMap);
for (int i = 0; i < squres.size(); i++) {
setLabel(cMap, String.valueOf(i), squres.get(i));
}
Bitmap resultBitmap = Bitmap.createBitmap(cMap.cols(), cMap.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(cMap, resultBitmap);
imgView.setImageBitmap(resultBitmap);
...
}
...
private static List<MatOfPoint> processImage(Mat img){
ArrayList<MatOfPoint> squares = new ArrayList<>();
Mat matGray = new Mat();
Mat matCny = new Mat();
Mat matBlur = new Mat();
Mat matThresh = new Mat();
Mat close = new Mat();
// 노이즈 제거위해 다운스케일 후 업스케일
// Imgproc.pyrDown(matInit, matBase, matBase.size());
// Imgproc.pyrUp(matBase, matInit, matInit.size());
// GrayScale
Imgproc.cvtColor(img, matGray, Imgproc.COLOR_BGR2GRAY);
// Blur
Imgproc.medianBlur(matGray, matBlur, 5);
// // Canny Edge 검출
// Imgproc.Canny(matBlur, matCny, 0, 255);
// // Binary
Imgproc.threshold(matBlur, matThresh, 160, 255, Imgproc.THRESH_BINARY_INV);
Imgproc.morphologyEx(matThresh, close, Imgproc.MORPH_CLOSE, Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(3,3)));
// // 노이즈 제거
// Imgproc.erode(matCny, matCny, Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new org.opencv.core.Size(6, 6)));
// Imgproc.dilate(matCny, matCny, Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new org.opencv.core.Size(12, 12)));
List<MatOfPoint> contours = new ArrayList<>();
Mat hierarchy = new Mat();
Imgproc.findContours(close, contours, hierarchy, Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_SIMPLE);
double min_area = 0;
double max_area = 10000;
for(MatOfPoint cnt : contours){
double contourArea = Imgproc.contourArea(cnt);
if(contourArea > min_area && contourArea < max_area){
squares.add(cnt);
}
}
return squares;
}
App result Image
Original Image
Please help me..

Your code is correctly identifying the smaller boxes and ignoring the very large box which is the strip, so the basics are all in place.
It is not recognising the smaller boxes on the strip - given that your contour finding is clearly working this suggests that the threshold value in your threshold function (160 in your code above) may need to be adjusted so it includes the color boxes on the strip which do not have a black contour. The black contour will be definitely detached.
Whatever the root causes you'll probably find the easiest way to debug it is to output and look at the intermediate images generated - this will allow you check visually yourself very quickly the result of your blurring and thresholding.
You could also take a look at using adaptive thresholding if you are working with multiple images and find the threshold is not something you can reliably determine in advance. The documentation is here: https://docs.opencv.org/2.4/modules/imgproc/doc/miscellaneous_transformations.html?highlight=adaptivethreshold and there is a very nice example in this answer here: https://stackoverflow.com/a/31290735/334402
adaptiveThreshold parameters allow you fine tune its behaviour and it is worth experimenting with them see what works best for a given type if image:

Java and haarcascade face and mouth detection - mouth as the nose

Today I begin to test the project which detects a smile in Java and OpenCv. To recognition face and mouth project used haarcascade_frontalface_alt and haarcascade_mcs_mouth But i don't understand why in some reasons project detect nose as a mouth.
I have two methods:
private ArrayList<Mat> detectMouth(String filename) {
int i = 0;
ArrayList<Mat> mouths = new ArrayList<Mat>();
// reading image in grayscale from the given path
image = Highgui.imread(filename, Highgui.CV_LOAD_IMAGE_GRAYSCALE);
MatOfRect faceDetections = new MatOfRect();
// detecting face(s) on given image and saving them to MatofRect object
faceDetector.detectMultiScale(image, faceDetections);
System.out.println(String.format("Detected %s faces", faceDetections.toArray().length));
MatOfRect mouthDetections = new MatOfRect();
// detecting mouth(s) on given image and saving them to MatOfRect object
mouthDetector.detectMultiScale(image, mouthDetections);
System.out.println(String.format("Detected %s mouths", mouthDetections.toArray().length));
for (Rect face : faceDetections.toArray()) {
Mat outFace = image.submat(face);
// saving cropped face to picture
Highgui.imwrite("face" + i + ".png", outFace);
for (Rect mouth : mouthDetections.toArray()) {
// trying to find right mouth
// if the mouth is in the lower 2/5 of the face
// and the lower edge of mouth is above of the face
// and the horizontal center of the mouth is the enter of the face
if (mouth.y > face.y + face.height * 3 / 5 && mouth.y + mouth.height < face.y + face.height
&& Math.abs((mouth.x + mouth.width / 2)) - (face.x + face.width / 2) < face.width / 10) {
Mat outMouth = image.submat(mouth);
// resizing mouth to the unified size of trainSize
Imgproc.resize(outMouth, outMouth, trainSize);
mouths.add(outMouth);
// saving mouth to picture
Highgui.imwrite("mouth" + i + ".png", outMouth);
i++;
}
}
}
return mouths;
}
and detect smile
private void detectSmile(ArrayList<Mat> mouths) {
trainSVM();
CvSVMParams params = new CvSVMParams();
// set linear kernel (no mapping, regression is done in the original feature space)
params.set_kernel_type(CvSVM.LINEAR);
// train SVM with images in trainingImages, labels in trainingLabels, given params with empty samples
clasificador = new CvSVM(trainingImages, trainingLabels, new Mat(), new Mat(), params);
// save generated SVM to file, so we can see what it generated
clasificador.save("svm.xml");
// loading previously saved file
clasificador.load("svm.xml");
// returnin, if there aren't any samples
if (mouths.isEmpty()) {
System.out.println("No mouth detected");
return;
}
for (Mat mouth : mouths) {
Mat out = new Mat();
// converting to 32 bit floating point in gray scale
mouth.convertTo(out, CvType.CV_32FC1);
if (clasificador.predict(out.reshape(1, 1)) == 1.0) {
System.out.println("Detected happy face");
} else {
System.out.println("Detected not a happy face");
}
}
}
Examples:
For that picture
correctly detects this mounth:
but in other picture
nose is detected
What's the problem in your opinion ?

Most likely it detects it wrong on your picture, because of proportion of face (too long distance from eyes to mouth compared to distance between eyes). Detection of mouth and nose using haar detector isn't very stable, so algorithms usually use geometry model of face, to choose best combination of feature candidates for each facial feature. Some implementations can even try to predict mouth position based on eyes, if no mouth candidates was found.
Haar detector isn't the newest and best known at this time for feature detection. Try to use deformable parts model implementations. Try this, they have matlab code with efficient c++ optimized functions:
https://www.ics.uci.edu/~xzhu/face/

OpenCV Object Detection and Splitting, clustering?

I'm currently working on an application that will split a scanned image (that contains multiple receipts) into individual receipt images.
Below is the sample image:
sample image
I was able to detect the edges of each receipts in the scanned image using canny function of OpenCV.
Below is the sample image with detected edges:
sample image with detected edges
... and the sample code is
Mat src = Highgui.imread(filename);
Mat gray = new Mat();
int threshold = 12;
Imgproc.cvtColor(src, gray, Imgproc.COLOR_BGR2GRAY);
Imgproc.blur(gray, gray, new Size(3, 3));
Imgproc.Canny(gray, gray, threshold, threshold * 3, 3, true);
List<MatOfPoint> contours = new ArrayList<>();
Mat hierarchy = new Mat();
Imgproc.findContours(gray, contours, hierarchy,
Imgproc.RETR_CCOMP,
Imgproc.CHAIN_APPROX_SIMPLE);
if (hierarchy.size().height > 0 && hierarchy.size().width > 0) {
for (int idx = 0; idx >= 0; idx = (int) hierarchy.get(0, idx)[0]) {
Rect rect = Imgproc.boundingRect(contours.get(idx));
Core.rectangle(src, new Point(rect.x, rect.y),
new Point(rect.x + rect.width, rect.y + rect.height),
new Scalar(255, 0, 0));
}
}
Now my problem is, I don't know how am I going to identify the 3rd receipt since unlike with the first 2 it is not enclosed in one rectangular shape which I will use as the basis for splitting the image.
I've heard that for me to extract the 3rd image, I must use a clustering algorithm like DBSCAN, unfortunately I can't find one.
Anyone knows how am I going to identify the 3rd image?
Thank you in advance!

Why does my program terminate after the first frame of the video?

I'm currently working on a program which takes the video from a webcam as input and then detects movement within this video, drawing lines around objects to show where they've moved to and from.
However, when I run this program, all it does it display one still image from my webcam. I have a pretty good idea why this is happening - the if-statement if (!(matFrame.empty())) is being evaluated as false, so the else statement runs, changing keepProcessing to false. This then terminates the while-loop, leaving nothing but ims.showImage(matFrame); as an output.
I can't find why this might be happening though, so I was hoping someone here might be able to help me. I've posted the code below so you can check for problems. I've also tried running it with a video to make sure this wasn't the fault of my webcam, and I found the same problem. Thanks for your time.
public class CaptureVideo {
public static void main(String[] args) throws InterruptedException {
// load the Core OpenCV library by name
System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
// create video capture device object
VideoCapture cap = new VideoCapture();
// try to use the hardware device if present
int CAM_TO_USE = 0;
// create a new image object
Mat matFrame = new Mat();
Mat previousFrame = new Mat();
Mat diffFrame = new Mat();
// try to open first capture device (0)
try {
cap.open(CAM_TO_USE);
} catch (Exception e1) {
System.out.println("No webcam attached");
// otherwise try opening a video file
try{
cap.open("files/video.mp4");
} catch (Exception e2) {
System.out.println("No video file found");
}
}
// if the a video capture source is now open
if (cap.isOpened())
{
// create a new window object
Imshow ims = new Imshow("From video source ... ");
boolean keepProcessing = true;
// add a flag to check whether the first frame has been read
boolean firstFrame = true;
while (keepProcessing)
{
// save previous frame before getting next one, but
// only do this if the first frame has passed
if (!firstFrame)
previousFrame = matFrame.clone();
// grab the next frame from video source
cap.grab();
// decode and return the grabbed video frame
cap.retrieve(matFrame);
// if the frame is valid (not end of video for example)
if (!(matFrame.empty()))
{
// if we are on the first frame, only show that and
// set the flag to false
if (firstFrame) {
ims.showImage(matFrame);
firstFrame = false;
}
// now show absolute difference after first frame
else {
Core.absdiff(matFrame, previousFrame, diffFrame);
ims.showImage(diffFrame);
}
// now convert it to grey and threshold it
Mat grey = new Mat();
Imgproc.cvtColor(diffFrame, grey, Imgproc.COLOR_BGR2GRAY);
Imgproc.adaptiveThreshold(grey, diffFrame, 255, Imgproc.ADAPTIVE_THRESH_MEAN_C,
Imgproc.THRESH_BINARY_INV, 7, 10);
// now clean it up using some morphological operations
Size ksize = new Size(15,15);
Mat kernel = Imgproc.getStructuringElement(Imgproc.MORPH_ELLIPSE, ksize);
Imgproc.morphologyEx(diffFrame, diffFrame, Imgproc.MORPH_CLOSE, kernel);
// find the all the contours from the binary image using the edge to contour
// stuff we looked at in lectures
List<MatOfPoint> contours = new Vector<MatOfPoint>();
Imgproc.findContours(diffFrame, contours, new Mat(), Imgproc.RETR_LIST, Imgproc.CHAIN_APPROX_SIMPLE);
// draw the contours on image 2 in red
Imgproc.drawContours(matFrame, contours, -1, new Scalar(0,0,255));
// find the largest contour by area
double maxArea = 0;
int maxAreaIndex = 0;
for (int i = 0; i < contours.size(); i++) {
double area = Imgproc.contourArea(contours.get(i), false);
if ( area > maxArea )
{
maxArea = area;
maxAreaIndex = i;
}
}
// draw the largest contour in red
Imgproc.drawContours(matFrame, contours, maxAreaIndex, new Scalar(0,255,0));
// create a new window objects
Imshow ims_diff = new Imshow("Difference");
// display images
ims_diff.showImage(diffFrame);
// display image with a delay of 40ms (i.e. 1000 ms / 25 = 25 fps)
Thread.sleep(40);
} else {
keepProcessing = false;
}
}
}
}
}

You should be seeing an exception on your console or output window:
OpenCV Error: Assertion failed (scn == 3 || scn == 4) in cv::cvtColor, file ..\..\..\..\opencv\modules\imgproc\src\color.cpp, line 3739
Exception in thread "main" CvException [org.opencv.core.CvException: cv::Exception: ..\..\..\..\opencv\modules\imgproc\src\color.cpp:3739: error: (-215) scn == 3 || scn == 4 in function cv::cvtColor
]
at org.opencv.imgproc.Imgproc.cvtColor_1(Native Method)
at org.opencv.imgproc.Imgproc.cvtColor(Imgproc.java:4598)
at CaptureVideo.main(CaptureVideo.java:87)
Which references line 87 (in my source file) which is:
Imgproc.cvtColor(diffFrame, grey, Imgproc.COLOR_BGR2GRAY);
The problem is that diffFrame hasn't been initialized so it's bombing out. I was able to get it to work locally by adding this block:
// decode and return the grabbed video frame
cap.retrieve(matFrame);
// *** START
if (firstFrame) {
firstFrame = false;
continue;
}
// *** End
// if the frame is valid (not end of video for example)
if (!(matFrame.empty()))
The effect of this is that the first frame will not be painted, but subsequent ones will. Also, code later on will open a new JFrame (Imshow) for every "diff" frame, which will quickly kill your machine, so be ready to kill the process.