Develop Reference

How can I detect squares using opencv4.2 (Android)

I am detecting a rectangle and comparing the color to a urine test strip.
How can i detect all of squares? I want to detect the remaining squares in the picture below. I have tried changing the brightness and contrast
Here is my code:
MainActivity.java
...
#Override
protected void onActivityResult(int requestCode, int resultCode, Intent data) {
...
Bitmap img = BitmapFactory.decodeStream(in);
in.close();
Bitmap changeImg = changeBitmapContrastBrightness(img, (float)1, 10);
Mat cMap = new Mat();
Utils.bitmapToMat(changeImg, cMap);
List<MatOfPoint> squres = processImage(cMap);
for (int i = 0; i < squres.size(); i++) {
setLabel(cMap, String.valueOf(i), squres.get(i));
}
Bitmap resultBitmap = Bitmap.createBitmap(cMap.cols(), cMap.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(cMap, resultBitmap);
imgView.setImageBitmap(resultBitmap);
...
}
...
private static List<MatOfPoint> processImage(Mat img){
ArrayList<MatOfPoint> squares = new ArrayList<>();
Mat matGray = new Mat();
Mat matCny = new Mat();
Mat matBlur = new Mat();
Mat matThresh = new Mat();
Mat close = new Mat();
// 노이즈 제거위해 다운스케일 후 업스케일
// Imgproc.pyrDown(matInit, matBase, matBase.size());
// Imgproc.pyrUp(matBase, matInit, matInit.size());
// GrayScale
Imgproc.cvtColor(img, matGray, Imgproc.COLOR_BGR2GRAY);
// Blur
Imgproc.medianBlur(matGray, matBlur, 5);
// // Canny Edge 검출
// Imgproc.Canny(matBlur, matCny, 0, 255);
// // Binary
Imgproc.threshold(matBlur, matThresh, 160, 255, Imgproc.THRESH_BINARY_INV);
Imgproc.morphologyEx(matThresh, close, Imgproc.MORPH_CLOSE, Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(3,3)));
// // 노이즈 제거
// Imgproc.erode(matCny, matCny, Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new org.opencv.core.Size(6, 6)));
// Imgproc.dilate(matCny, matCny, Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new org.opencv.core.Size(12, 12)));
List<MatOfPoint> contours = new ArrayList<>();
Mat hierarchy = new Mat();
Imgproc.findContours(close, contours, hierarchy, Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_SIMPLE);
double min_area = 0;
double max_area = 10000;
for(MatOfPoint cnt : contours){
double contourArea = Imgproc.contourArea(cnt);
if(contourArea > min_area && contourArea < max_area){
squares.add(cnt);
}
}
return squares;
}
App result Image
Original Image
Please help me..

Your code is correctly identifying the smaller boxes and ignoring the very large box which is the strip, so the basics are all in place.
It is not recognising the smaller boxes on the strip - given that your contour finding is clearly working this suggests that the threshold value in your threshold function (160 in your code above) may need to be adjusted so it includes the color boxes on the strip which do not have a black contour. The black contour will be definitely detached.
Whatever the root causes you'll probably find the easiest way to debug it is to output and look at the intermediate images generated - this will allow you check visually yourself very quickly the result of your blurring and thresholding.
You could also take a look at using adaptive thresholding if you are working with multiple images and find the threshold is not something you can reliably determine in advance. The documentation is here: https://docs.opencv.org/2.4/modules/imgproc/doc/miscellaneous_transformations.html?highlight=adaptivethreshold and there is a very nice example in this answer here: https://stackoverflow.com/a/31290735/334402
adaptiveThreshold parameters allow you fine tune its behaviour and it is worth experimenting with them see what works best for a given type if image:

OpenCV on Android: net.forward yields "215 Assertion failed"

Following this tutorial from openCV, and it should be straight forward. However, it crashes with an assertion fail on the net.forward, that I cannot resolve/find anywhere else.
Thought this problem seemed similar and tried to go through the fix/problem finding. However, restarting the discussion and trials showed it is likely not the same. I used initially 3.4.3, which did not support the same Mat type somehow. Updated to 3.4.7 now, and can confirm the blob size is okay (generated from image). Tried also various other prototxt and caffemodels, but doubt by now that the problem lies there (works if the files are okay, otherwise the net loading fails). The key code should be this:
// Load a network.
public void onCameraViewStarted(int width, int height) {
String proto = getPath("deploy.prototxt", this);
String weights = getPath("MobileNetSSD_deploy.caffemodel", this);
net = Dnn.readNetFromCaffe(proto, weights);
Log.i(TAG, "Network loaded successfully");
}
public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
// Get a new frame
Mat frame = inputFrame.rgba();
Imgproc.cvtColor(frame, frame, Imgproc.COLOR_RGBA2RGB);
// Forward image through network.
Mat blob = Dnn.blobFromImage(frame, 0.007843,
new Size(300, 300),
new Scalar(127.5, 127.5, 127.5));
net.setInput(blob);
Mat detections = net.forward(); //***215 ASSERTION FAILED occurs***
int cols = frame.cols();
int rows = frame.rows();
detections = detections.reshape(1, (int)detections.total() / 7);
for (int i = 0; i < detections.rows(); ++i) {
double confidence = detections.get(i, 2)[0];
if (confidence > 0.2) {
int classId = (int)detections.get(i, 1)[0];
int left = (int)(detections.get(i, 3)[0] * cols);
int top = (int)(detections.get(i, 4)[0] * rows);
int right = (int)(detections.get(i, 5)[0] * cols);
int bottom = (int)(detections.get(i, 6)[0] * rows);
// Draw rectangle around detected object.
Imgproc.rectangle(frame, new Point(left, top), new Point(right, bottom),
new Scalar(0, 255, 0));
String label = classNames[classId] + ": " + confidence;
int[] baseLine = new int[1];
Size labelSize = Imgproc.getTextSize(label, Core.FONT_HERSHEY_SIMPLEX, 0.5, 1, baseLine);
// Draw background for label.
Imgproc.rectangle(frame, new Point(left, top - labelSize.height),
new Point(left + labelSize.width, top + baseLine[0]),
new Scalar(255, 255, 255), Core.FILLED);
// Write class name and confidence.
Imgproc.putText(frame, label, new Point(left, top),
Core.FONT_HERSHEY_SIMPLEX, 0.5, new Scalar(0, 0, 0));
}
}
return frame;
}
public void onCameraViewStopped() {}
// Upload file to storage and return a path.
private static String getPath(String file, Context context) {
AssetManager assetManager = context.getAssets();
BufferedInputStream inputStream = null;
try {
// Read data from assets.
inputStream = new BufferedInputStream(assetManager.open(file));
byte[] data = new byte[inputStream.available()];
inputStream.read(data);
inputStream.close();
// Create copy file in storage.
File outFile = new File(context.getFilesDir(), file);
FileOutputStream os = new FileOutputStream(outFile);
os.write(data);
os.close();
// Return a path to file which may be read in common way.
return outFile.getAbsolutePath();
} catch (IOException ex) {
Log.i(TAG, "Failed to upload a file");
}
return "";
}
The full error message is
cv::Exception: OpenCV(3.4.7) /build/3_4_pack-android/opencv/modules/dnn/src/layers/batch_norm_layer.cpp:39: error: (-215:Assertion failed) blobs.size() >= 2 in function 'cv::dnn::BatchNormLayerImpl::BatchNormLayerImpl(const cv::dnn::experimental_dnn_34_v13::LayerParams&)'
I expect it to not crash. The frame should be okay (image loaded), the net is not empty, and the layers in the net seem fine too (checked since there are some differences using caffe in java). Any help is appreciated!

After some days of research in different directions, I found the problem: the frame format should be BGR, not RGB! That means
Imgproc.cvtColor(frame, frame, Imgproc.COLOR_RGBA2BGR);

Java char recognition with OpenCV

i am trying to build an application that with a camera can recognize some numbers and letters from a board in front of the camera.
At moment i can detect faces, counters but i want to use ROI and Tess4j to recognize this live video.
Do you know any example of something like this?
My idea is that i have to analyze frame by frame and when i detect a char like '*' i make the full verification of the values
public class Demo {
public static void main(String[] args) throws Exception {
String classifierName = null;
if (args.length > 0) {
classifierName = args[0];
} else {
URL url = new URL("https://raw.github.com/Itseez/opencv/2.4.0/data/haarcascades/haarcascade_frontalface_alt.xml");
File file = Loader.extractResource(url, null, "classifier", ".xml");
file.deleteOnExit();
classifierName = file.getAbsolutePath();
}
// Preload the opencv_objdetect module to work around a known bug.
Loader.load(opencv_objdetect.class);
// We can "cast" Pointer objects by instantiating a new object of the desired class.
CvHaarClassifierCascade classifier = new CvHaarClassifierCascade(cvLoad(classifierName));
if (classifier.isNull()) {
System.err.println("Error loading classifier file \"" + classifierName + "\".");
System.exit(1);
}
// The available FrameGrabber classes include OpenCVFrameGrabber (opencv_videoio),
// DC1394FrameGrabber, FlyCaptureFrameGrabber, OpenKinectFrameGrabber, OpenKinect2FrameGrabber,
// RealSenseFrameGrabber, PS3EyeFrameGrabber, VideoInputFrameGrabber, and FFmpegFrameGrabber.
FrameGrabber grabber = FrameGrabber.createDefault(0);
grabber.start();
// CanvasFrame, FrameGrabber, and FrameRecorder use Frame objects to communicate image data.
// We need a FrameConverter to interface with other APIs (Android, Java 2D, or OpenCV).
OpenCVFrameConverter.ToIplImage converter = new OpenCVFrameConverter.ToIplImage();
// FAQ about IplImage and Mat objects from OpenCV:
// - For custom raw processing of data, createBuffer() returns an NIO direct
// buffer wrapped around the memory pointed by imageData, and under Android we can
// also use that Buffer with Bitmap.copyPixelsFromBuffer() and copyPixelsToBuffer().
// - To get a BufferedImage from an IplImage, or vice versa, we can chain calls to
// Java2DFrameConverter and OpenCVFrameConverter, one after the other.
// - Java2DFrameConverter also has static copy() methods that we can use to transfer
// data more directly between BufferedImage and IplImage or Mat via Frame objects.
IplImage grabbedImage = converter.convert(grabber.grab());
int width = grabbedImage.width();
int height = grabbedImage.height();
IplImage grayImage = IplImage.create(width, height, IPL_DEPTH_8U, 1);
IplImage rotatedImage = grabbedImage.clone();
// Objects allocated with a create*() or clone() factory method are automatically released
// by the garbage collector, but may still be explicitly released by calling release().
// You shall NOT call cvReleaseImage(), cvReleaseMemStorage(), etc. on objects allocated this way.
CvMemStorage storage = CvMemStorage.create();
// The OpenCVFrameRecorder class simply uses the CvVideoWriter of opencv_videoio,
// but FFmpegFrameRecorder also exists as a more versatile alternative.
FrameRecorder recorder = FrameRecorder.createDefault("output.avi", width, height);
recorder.start();
// CanvasFrame is a JFrame containing a Canvas component, which is hardware accelerated.
// It can also switch into full-screen mode when called with a screenNumber.
// We should also specify the relative monitor/camera response for proper gamma correction.
CanvasFrame frame = new CanvasFrame("Some Title", CanvasFrame.getDefaultGamma()/grabber.getGamma());
// Let's create some random 3D rotation...
CvMat randomR = CvMat.create(3, 3), randomAxis = CvMat.create(3, 1);
// We can easily and efficiently access the elements of matrices and images
// through an Indexer object with the set of get() and put() methods.
DoubleIndexer Ridx = randomR.createIndexer(), axisIdx = randomAxis.createIndexer();
axisIdx.put(0, (Math.random()-0.5)/4, (Math.random()-0.5)/4, (Math.random()-0.5)/4);
cvRodrigues2(randomAxis, randomR, null);
double f = (width + height)/2.0; Ridx.put(0, 2, Ridx.get(0, 2)*f);
Ridx.put(1, 2, Ridx.get(1, 2)*f);
Ridx.put(2, 0, Ridx.get(2, 0)/f); Ridx.put(2, 1, Ridx.get(2, 1)/f);
System.out.println(Ridx);
// We can allocate native arrays using constructors taking an integer as argument.
CvPoint hatPoints = new CvPoint(3);
while (frame.isVisible() && (grabbedImage = converter.convert(grabber.grab())) != null) {
cvClearMemStorage(storage);
// Let's try to detect some faces! but we need a grayscale image...
cvCvtColor(grabbedImage, grayImage, CV_BGR2GRAY);
CvSeq faces = cvHaarDetectObjects(grayImage, classifier, storage,
1.1, 3, CV_HAAR_FIND_BIGGEST_OBJECT | CV_HAAR_DO_ROUGH_SEARCH);
int total = faces.total();
for (int i = 0; i < total; i++) {
CvRect r = new CvRect(cvGetSeqElem(faces, i));
int x = r.x(), y = r.y(), w = r.width(), h = r.height();
cvRectangle(grabbedImage, cvPoint(x, y), cvPoint(x+w, y+h), CvScalar.RED, 1, CV_AA, 0);
// To access or pass as argument the elements of a native array, call position() before.
hatPoints.position(0).x(x-w/10) .y(y-h/10);
hatPoints.position(1).x(x+w*11/10).y(y-h/10);
hatPoints.position(2).x(x+w/2) .y(y-h/2);
cvFillConvexPoly(grabbedImage, hatPoints.position(0), 3, CvScalar.GREEN, CV_AA, 0);
}
// Let's find some contours! but first some thresholding...
cvThreshold(grayImage, grayImage, 64, 255, CV_THRESH_BINARY);
// To check if an output argument is null we may call either isNull() or equals(null).
CvSeq contour = new CvSeq(null);
cvFindContours(grayImage, storage, contour, Loader.sizeof(CvContour.class),
CV_RETR_LIST, CV_CHAIN_APPROX_SIMPLE);
while (contour != null && !contour.isNull()) {
if (contour.elem_size() > 0) {
CvSeq points = cvApproxPoly(contour, Loader.sizeof(CvContour.class),
storage, CV_POLY_APPROX_DP, cvContourPerimeter(contour)*0.02, 0);
cvDrawContours(grabbedImage, points, CvScalar.BLUE, CvScalar.BLUE, -1, 1, CV_AA);
}
contour = contour.h_next();
}
//TESS4J - se ahou os contornos, analisar conteúdo
if(contour != null && !contour.isNull()) {
IplImageToBufferedImage(grayImage);
ITesseract instance = new Tesseract(); // JNA Interface Mapping
instance.setTessVariable("tessedit_char_whitelist", "0123456789");
try {
String result = instance.doOCR(IplImageToBufferedImage(grayImage));
System.out.println(result);
} catch (TesseractException e) {
System.err.println(e.getMessage());
}
}
cvWarpPerspective(grabbedImage, rotatedImage, randomR);
Frame rotatedFrame = converter.convert(rotatedImage);
frame.showImage(rotatedFrame);
recorder.record(rotatedFrame);
}
frame.dispose();
recorder.stop();
grabber.stop();
}
public static BufferedImage IplImageToBufferedImage(IplImage src) {
OpenCVFrameConverter.ToIplImage grabberConverter = new OpenCVFrameConverter.ToIplImage();
Java2DFrameConverter paintConverter = new Java2DFrameConverter();
Frame frame = grabberConverter.convert(src);
return paintConverter.getBufferedImage(frame,1);
}
}

After using a LineSegmentDetector on an image, how can I retrieve the data, and can I choose how small of a line segment to detect?

I'm trying to design a program that will recognize the rectangle in the image below. After running the Line Segment Detector from OpenCV, I'm left with the second image.
I can clearly see the rectangle, and I want to start filtering out garbage line segments, but I can't seem to find any way to retrieve the data of the line segments created.
It would also be helpful if, with the initial .detect method, I could explicity say "no lines smaller than 20 pixels". Is that possible?
A full working sample of my code is below:
public class VinDetector {
public static void main(String args[]) throws Exception {
try {
System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
File input = new File("image.jpg");
BufferedImage image = ImageIO.read(input);
byte[] data = ((DataBufferByte) image.getRaster().getDataBuffer()).getData();
Mat sourceImg = new Mat(image.getHeight(), image.getWidth(), CvType.CV_8UC3);
sourceImg.put(0, 0, data);
Mat destImage = new Mat(image.getHeight(),image.getWidth(),CvType.CV_8UC1);
Imgproc.cvtColor(sourceImg, destImage, Imgproc.COLOR_RGB2GRAY);
MatOfFloat4 lines = new MatOfFloat4();
LineSegmentDetector ls = Imgproc.createLineSegmentDetector();
ls.detect(destImage, lines);
System.out.println(ls.compareSegments(new Size(destImage.width(), destImage.height()), lines, lines));
Mat lineImage = Imgcodecs.imread("image black.jpg");
ls.drawSegments(lineImage, lines);
Imgcodecs.imwrite("image TEST.jpg", lineImage);
} catch (Exception e) {
System.out.println("error: " + e.getMessage());
}
}
}

Converting IplImage to BufferedImage to integrate

I'm making my own Image processing application that completely operates in BufferedImage.
Now i have stumbled upon a code on Face detection in a blog of [OpenShift.com]
Now i want to integrate that code into my own GUI application.But facing problems as the Face Detector code the image is an instance of iplImage object and for that i need to first convert the buffered image to IplImage so that the method accepts the now converted image.
Please help..
i am leaving below the Face detector code.
public class FaceDetection{
//Load haar classifier XML file
public static final String XML_FILE =
"C:\\opencv\\sources\\data\\haarcascades\\haarcascade_frontalface_alt2.xml";
public static void main(String[] args){
//Load image
IplImage img = cvLoadImage("C:\\Users\\The Blue Light\\Desktop\\13.jpg");
detect(img);
}
//Detect for face using classifier XML file
public static void detect(IplImage src){
//Define classifier
CvHaarClassifierCascade cascade = new CvHaarClassifierCascade(cvLoad(XML_FILE));
CvMemStorage storage = CvMemStorage.create();
//Detect objects
CvSeq sign = cvHaarDetectObjects(
src,
cascade,
storage,
1.5,
3,
CV_HAAR_DO_CANNY_PRUNING);
cvClearMemStorage(storage);
int total_Faces = sign.total();
//Draw rectangles around detected objects
for(int i = 0; i < total_Faces; i++){
CvRect r = new CvRect(cvGetSeqElem(sign, i));
cvRectangle (
src,
cvPoint(r.x(), r.y()),
cvPoint(r.width() + r.x(), r.height() + r.y()),
CvScalar.CYAN,
2,
CV_AA,
0);
}
//Display result
cvShowImage("Result", src);
cvWaitKey(0);
}
}

IplImage image = IplImage.createFrom(yourBufferedImage);
Thanks #Marco13
exactly what i needed..