Develop Reference

OpenCV on Android: net.forward yields "215 Assertion failed"

Following this tutorial from openCV, and it should be straight forward. However, it crashes with an assertion fail on the net.forward, that I cannot resolve/find anywhere else.
Thought this problem seemed similar and tried to go through the fix/problem finding. However, restarting the discussion and trials showed it is likely not the same. I used initially 3.4.3, which did not support the same Mat type somehow. Updated to 3.4.7 now, and can confirm the blob size is okay (generated from image). Tried also various other prototxt and caffemodels, but doubt by now that the problem lies there (works if the files are okay, otherwise the net loading fails). The key code should be this:
// Load a network.
public void onCameraViewStarted(int width, int height) {
String proto = getPath("deploy.prototxt", this);
String weights = getPath("MobileNetSSD_deploy.caffemodel", this);
net = Dnn.readNetFromCaffe(proto, weights);
Log.i(TAG, "Network loaded successfully");
}
public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
// Get a new frame
Mat frame = inputFrame.rgba();
Imgproc.cvtColor(frame, frame, Imgproc.COLOR_RGBA2RGB);
// Forward image through network.
Mat blob = Dnn.blobFromImage(frame, 0.007843,
new Size(300, 300),
new Scalar(127.5, 127.5, 127.5));
net.setInput(blob);
Mat detections = net.forward(); //***215 ASSERTION FAILED occurs***
int cols = frame.cols();
int rows = frame.rows();
detections = detections.reshape(1, (int)detections.total() / 7);
for (int i = 0; i < detections.rows(); ++i) {
double confidence = detections.get(i, 2)[0];
if (confidence > 0.2) {
int classId = (int)detections.get(i, 1)[0];
int left = (int)(detections.get(i, 3)[0] * cols);
int top = (int)(detections.get(i, 4)[0] * rows);
int right = (int)(detections.get(i, 5)[0] * cols);
int bottom = (int)(detections.get(i, 6)[0] * rows);
// Draw rectangle around detected object.
Imgproc.rectangle(frame, new Point(left, top), new Point(right, bottom),
new Scalar(0, 255, 0));
String label = classNames[classId] + ": " + confidence;
int[] baseLine = new int[1];
Size labelSize = Imgproc.getTextSize(label, Core.FONT_HERSHEY_SIMPLEX, 0.5, 1, baseLine);
// Draw background for label.
Imgproc.rectangle(frame, new Point(left, top - labelSize.height),
new Point(left + labelSize.width, top + baseLine[0]),
new Scalar(255, 255, 255), Core.FILLED);
// Write class name and confidence.
Imgproc.putText(frame, label, new Point(left, top),
Core.FONT_HERSHEY_SIMPLEX, 0.5, new Scalar(0, 0, 0));
}
}
return frame;
}
public void onCameraViewStopped() {}
// Upload file to storage and return a path.
private static String getPath(String file, Context context) {
AssetManager assetManager = context.getAssets();
BufferedInputStream inputStream = null;
try {
// Read data from assets.
inputStream = new BufferedInputStream(assetManager.open(file));
byte[] data = new byte[inputStream.available()];
inputStream.read(data);
inputStream.close();
// Create copy file in storage.
File outFile = new File(context.getFilesDir(), file);
FileOutputStream os = new FileOutputStream(outFile);
os.write(data);
os.close();
// Return a path to file which may be read in common way.
return outFile.getAbsolutePath();
} catch (IOException ex) {
Log.i(TAG, "Failed to upload a file");
}
return "";
}
The full error message is
cv::Exception: OpenCV(3.4.7) /build/3_4_pack-android/opencv/modules/dnn/src/layers/batch_norm_layer.cpp:39: error: (-215:Assertion failed) blobs.size() >= 2 in function 'cv::dnn::BatchNormLayerImpl::BatchNormLayerImpl(const cv::dnn::experimental_dnn_34_v13::LayerParams&)'
I expect it to not crash. The frame should be okay (image loaded), the net is not empty, and the layers in the net seem fine too (checked since there are some differences using caffe in java). Any help is appreciated!

After some days of research in different directions, I found the problem: the frame format should be BGR, not RGB! That means
Imgproc.cvtColor(frame, frame, Imgproc.COLOR_RGBA2BGR);

How to blur a portion of an image with JAVA

How to blur a portion of an image, to hide some privates parts like credit card informations.
I try to use ConvolveOp.class like :
float[] matrix = new float[400];
for (int i = 0; i < 400; i++)
matrix[i] = 1.0f/500.0f;
BufferedImage sourceImage = (BufferedImage) image; ;
BufferedImage destImage = null ;
BufferedImageOp op = new ConvolveOp( new Kernel(20, 20, matrix), ConvolveOp.EDGE_NO_OP, null );
BufferedImage blurredImage = op.filter(sourceImage, destImage);
it seems to work, except that the image is completely blurred.

In the case you want to focus on the application and not on the specifics of image processing, you can use an image processing framework like Marvin. Thus, you can do more with less code.
Input image:
Output image:
Source code:
import static marvin.MarvinPluginCollection.*;
public class PortionBlur {
public PortionBlur(){
// 1. Load image
MarvinImage image = MarvinImageIO.loadImage("./res/credit_card.jpg");
// 2. Create masks for each blurred region
MarvinImageMask mask1 = new MarvinImageMask(image.getWidth(), image.getHeight(), 38,170,345,24);
MarvinImageMask mask2 = new MarvinImageMask(image.getWidth(), image.getHeight(), 52,212,65,24);
MarvinImageMask mask3 = new MarvinImageMask(image.getWidth(), image.getHeight(), 196,212,65,20);
MarvinImageMask mask4 = new MarvinImageMask(image.getWidth(), image.getHeight(), 38,240,200,20);
// 3. Process Image with each mask
GaussianBlur gaussianBlur = new GaussianBlur();
gaussianBlur.load();
gaussianBlur.attributes.set("radius",15);
gaussianBlur.process(image.clone(), image, mask1);
gaussianBlur.process(image.clone(), image, mask2);
gaussianBlur.process(image.clone(), image, mask3);
gaussianBlur.process(image.clone(), image, mask4);
// 4. Save the final image
MarvinImageIO.saveImage(image, "./res/credit_card_out.jpg");
}
public static void main(String[] args) {
new PortionBlur();
System.exit(0);
}
}
Gaussian blur algorithm source code:
https://github.com/gabrielarchanjo/marvinproject/blob/master/marvinproject/dev/MarvinPlugins/src/org/marvinproject/image/blur/gaussianBlur/GaussianBlur.java

I don't know whether this can be done by changing the matrix values, but this should definitely be possible by filtering a subimage, since, according to the BufferedImage.getSubimage() documentation:
The returned BufferedImage shares the same data array as the original image.
So the original BufferedImage should change with code like this:
BufferedImage image = /* ... */;
BufferedImage subImage = image.getSubimage(10, 20, 30, 40); // x, y, width, height
new ConvolveOp(new Kernel(20, 20, matrix), ConvolveOp.EDGE_NO_OP, null).filter(subImage, subImage);
I didn't test this though, and I can imagine that filter doesn't work as expected if source and destination are the same, in which case you could use a copy of the subimage, using the solution from this question:
BufferedImage image = /* ... */;
BufferedImage dest = image.getSubimage(10, 20, 30, 40); // x, y, width, height
ColorModel cm = dest.getColorModel();
BufferedImage src = new BufferedImage(cm, dest.copyData(dest.getRaster().createCompatibleWritableRaster()), cm.isAlphaPremultiplied(), null).getSubimage(0, 0, dest.getWidth(), dest.getHeight());
new ConvolveOp(new Kernel(20, 20, matrix), ConvolveOp.EDGE_NO_OP, null).filter(src, dest);
After that, continue working with image (not subImage, src or dest!)

How to compress and resize image from a Rich Text

I am inserting image in using this
var readFileIntoDataUrl = function (fileInfo) {
var loader = $.Deferred(),
fReader = new FileReader();
fReader.onload = function (e) {
loader.resolve(e.target.result);
};
fReader.onerror = loader.reject;
fReader.onprogress = loader.notify;
fReader.readAsDataURL(fileInfo);
return loader.promise();
};
$.when(readFileIntoDataUrl(fileInfo)).done(function (dataUrl) {
execCommand('insertimage', dataUrl);
}).fail(function (e) {
options.fileUploadError("file-reader", e);
});
Let say i added a text Hello World and added a image. Now when i take $("#editor").html() it shows something like below
Here is a sample source of a image+text
Hello World!
img src="data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEAYABgAAD/4QBoRXhpZgAATU0AKgAAAAgA
BAEaAAUAAAABAAAAPgEbAAUAAAABAAAARgEoAAMAAAABAAIAAAExAAIAAAARAAAATgAAAAAAAABgAAAA
AQAAAGAAAAABUGFpbnQuTkVUIHYzLjUuOAAA/9sAQwAHBQUGBQQHBgUGCAcHCAoRCwoJCQoVDxAMERgV
[... more base64 data here....]
Now here i have both text+image So both on server side and client side i want to resize & compress image
So that no one can insert image > 5MP and also keep a rich text with resize image in my db

This is how you can do it on the server side (Java):
String imageData = "Hello World! img src=\"data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEAYABgAAD/4QBoR...";
//Image data starting point
int startIndex = imageData.indexOf(";base64,") + ";base64,".length();
//keep only the image data
imageData = imageData.substring(startIndex, imageData.length());
//convert the image data String to a byte[]
byte[] dta = DatatypeConverter.parseBase64Binary(imageData);
try (InputStream in = new ByteArrayInputStream(dta);) {
BufferedImage fullSize = ImageIO.read(in);
// Create a new image half the size of the original image
BufferedImage resized = new BufferedImage(fullSize.getWidth() / 2, fullSize.getHeight() / 2, BufferedImage.TYPE_4BYTE_ABGR);
Graphics2D g2 = (Graphics2D) resized.getGraphics();
g2.drawImage(fullSize, 0, 0, resized.getWidth(), resized.getHeight(), 0, 0, fullSize.getWidth(), fullSize.getHeight(), null);
g2.dispose();
} catch (IOException e) {
e.printStackTrace();
}
As for the JavaScript part you can use a canvas, size the canvas to the dimension that you want, draw your image on it and use the toDataURL method to convert the image to a String

OpenCV output Using Mat object in Jpanel

I know we can output an Image using Jpanel using the following code:
JFrame frame = new JFrame("IMG");
frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
frame.setResizable(true);
frame.setLocationRelativeTo(null);
ImageIcon image = new ImageIcon(".../img.jpg");
frame.setSize(image.getIconWidth()+10,image.getIconHeight()+35);
JLabel label1 = new JLabel(" ", image, JLabel.CENTER);
frame.getContentPane().add(label1);
frame.validate();
frame.setVisible(true);
This code would assume that I have created or already have an image file in my directory. But what I want to do is output the image in the Jpanel directly without creating the image file. How do I do that using OpenCV Mat object in JAVA?

You can convert Mat object to BufferedImage object using following method:
public static BufferedImage createAwtImage(Mat mat) {
int type = 0;
if (mat.channels() == 1) {
type = BufferedImage.TYPE_BYTE_GRAY;
} else if (mat.channels() == 3) {
type = BufferedImage.TYPE_3BYTE_BGR;
} else {
return null;
}
BufferedImage image = new BufferedImage(mat.width(), mat.height(), type);
WritableRaster raster = image.getRaster();
DataBufferByte dataBuffer = (DataBufferByte) raster.getDataBuffer();
byte[] data = dataBuffer.getData();
mat.get(0, 0, data);
return image;
}
Then you can simply display image using ImageIcon:
// Load image using Highgui or create Mat object other way you want
Mat mat = Highgui.imread(".../img.jpg");
ImageIcon image = new ImageIcon(createAwtImage(mat));

Opencv java - Load image to GUI

I'm developing an application using Java Opencv-2.4.4 and swing GUI. Problem is that I'm unable to find any solution, that shows efficient way how to print processed image (saved in Mat object) to java swing GUI. For this moment I'm using this clumsy solution:
javax.swing.JLabel outputImage;
outputImage.setIcon(new javax.swing.ImageIcon("/home/username/Output.png"));
private void sliderStateChanged(javax.swing.event.ChangeEvent evt) {
.
.
Mat canny; // Here is saved what I want to plot
String filename = "/home/username/Output.png";
Highgui.imwrite(filename, canny); // write to disk
outputImage.setIcon(new ImageIcon(ImageIO.read(new File(filename)))); //update Icon
.
.
}
When user changes some values, inputs etc ., in GUI I have to overwrite Output.png on disk and update jLabel with new image from disk.
Is there any more elegant / efficient solution to this ? Is it posible to plot or convert Mat object directly to Canvas or Image or anything that is printable as image in swing ?

Yes there is more elegant way to do it. You can concert Mat to BufferedImage type and then just Load it with swing. The code to convert it to Buffered image is:
Mat image_tmp = your image
MatOfByte matOfByte = new MatOfByte();
Highgui.imencode(".jpg", image_tmp, matOfByte);
byte[] byteArray = matOfByte.toArray();
BufferedImage bufImage = null;
try {
InputStream in = new ByteArrayInputStream(byteArray);
bufImage = ImageIO.read(in);
} catch (Exception e) {
e.printStackTrace();
}
And then you just can paint it in your GUI object:
g.drawImage(bufImage , 0, 0, null);
where g is of type Graphics
Hope this helps.

jpeg encoding is interesting, but there are a couple problems:
it is not a lossless format, you will lose image data when compressing
it takes quite a while (around 6 to 10 times longer than the suggested one below)
public Image toBufferedImage(Mat m){
int type = BufferedImage.TYPE_BYTE_GRAY;
if ( m.channels() > 1 ) {
type = BufferedImage.TYPE_3BYTE_BGR;
}
int bufferSize = m.channels()*m.cols()*m.rows();
byte [] b = new byte[bufferSize];
m.get(0,0,b); // get all the pixels
BufferedImage image = new BufferedImage(m.cols(),m.rows(), type);
final byte[] targetPixels = ((DataBufferByte) image.getRaster().getDataBuffer()).getData();
System.arraycopy(b, 0, targetPixels, 0, b.length);
return image;
}

This is a readymade solution for Imshow() equivalent in Java OpenCV Its simple to use. API will look like:
Imshow im = new Imshow("Title");
im.showImage(matimage);
Visit here https://github.com/master-atul/ImShow-Java-OpenCV
This is a better solution as you don't store the image into disk and read again. Hence it reduces the overhead of reading from a disk and thus is faster.

Using #andriy's anwser. I came up with this solution. I used JFrame instead of Graphics. Hope this helps.
public void imshow(Mat src){
BufferedImage bufImage = null;
try {
MatOfByte matOfByte = new MatOfByte();
Highgui.imencode(".jpg", src, matOfByte);
byte[] byteArray = matOfByte.toArray();
InputStream in = new ByteArrayInputStream(byteArray);
bufImage = ImageIO.read(in);
JFrame frame = new JFrame("Image");
frame.getContentPane().setLayout(new FlowLayout());
frame.getContentPane().add(new JLabel(new ImageIcon(bufImage)));
frame.pack();
frame.setVisible(true);
frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
} catch (Exception e) {
e.printStackTrace();
}
}