Develop Reference

Java OpenCV opening video file issues

I have a grayscale .mkv video, which i want to open with OpenCV in Java, but i get the following errors:
With return new VideoCapture(path, Videoio.CAP_FFMPEG);
Errors:
[ERROR:0#0.004] global /build/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp (1108) open Could not find decoder for codec_id=61
[ERROR:0#0.004] global /build/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp (1140) open VIDEOIO/FFMPEG: Failed to initialize VideoCapture
With return new VideoCapture(path, Videoio.CAP_DSHOW); No errors, but
video.isOpened() is false
With return new VideoCapture(path);
Errors:
[ERROR:0#0.005] global /build/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp (1108) open Could not find decoder for codec_id=61
[ERROR:0#0.005] global /build/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp (1140) open VIDEOIO/FFMPEG: Failed to initialize VideoCapture
[ WARN:0#0.122] global C:\build\master_winpack-bindings-win64-vc14-static\opencv\modules\videoio\src\cap_msmf.cpp (923) CvCapture_MSMF::initStream Failed to set mediaType (stream 0, (480x360 # 1) MFVideoFormat_RGB24(codec not found)
I have installed OpenCV and added it as a dependency using this video.
I have also tried adding ...\opencv\build\bin\opencv_videoio_ffmpeg455_64.dll to the native libraries, and also tried using this: System.load("path\\to\\opencv\\build\\bin\\opencv_videoio_ffmpeg455_64.dll");.
Full code:
public class Test {
static {
System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
System.load("path\\to\\opencv\\build\\bin\\opencv_videoio_ffmpeg455_64.dll");
}
public static void main(String[] args) {
List<Mat> frames = getVideoFrames(openVideoFile(args[0]));
System.out.println(frames.size());
}
}
//... different class
public static VideoCapture openVideoFile(String path) {
return new VideoCapture(path);
}
public static List<Mat> getVideoFrames(VideoCapture video) {
List<Mat> frames = new ArrayList<>();
Mat frame = new Mat();
if (video.isOpened()) {
while (video.read(frame)) {
frames.add(frame);
}
video.release();
}
return frames;
}
ffprobe result:
Metadata:
MAJOR_BRAND : qt
MINOR_VERSION : 512
COMPATIBLE_BRANDS: qt
ENCODER : Lavf56.40.101
Duration: 00:01:05.83, start: 0.000000, bitrate: 1511 kb/s
Stream #0:0(eng): Video: png (MPNG / 0x474E504D), rgb24(pc), 480x360 [SAR 1:1 DAR 4:3], 6 fps, 6 tbr, 1k tbn (default)
Metadata:
LANGUAGE : eng
HANDLER_NAME : DataHandler
ENCODER : Lavc56.60.100 png
DURATION : 00:01:05.834000000

The error message is indicating that OpenCV's FFmpeg plugin is not built with MPNG codec support. So, you are essentially SOL to get this task done only with OpenCV (you can request OpenCV to support the codec, but it won't be a quick adaption even if you succeed to convince their devs to do so). Here are a couple things I could think of as a non-Java/non-OpenCV person (I typically deal with Python/FFmpeg):
1a) If you have a control of the upstream of your data, change the video codec from MPNG to one with OpenCV support.
1b) Transcode the MKV file to re-encode the video stream with a supported codec within your program
Create a thread and call ffmpeg from Java as a subprocess (I think ProcessBuilder is the class you are interested in) and load the video data via stdout pipe. FFmpeg can be called in the following manner:
ffmpeg -i <video_path> -f rawvideo -pix_fmt gray -an -
The stdout pipe will receive 480x360 bytes per frame. Read as many frames as you need at a time. If you need to limit the frames, you need to do this in seconds using -ss, -t, and/or -to options.
I'm assuming this video is grayscale as you mentioned (ffprobe is indicating the video is saved in RGB format). If you need to get RGB, use -pix_fmt rgb24 and the video frame data.
Once you have the image data in memory, there should be an OpenCV function to create an image object from in-memory data.

Java auto vectorization example

I'm trying to find a concise example which shows auto vectorization in java on a x86-64 system.
I've implemented the below code using y[i] = y[i] + x[i] in a for loop. This code can benefit from auto vectorization, so I think java should compile it at runtime using SSE or AVX instructions to speed it up.
However, I couldn't find the vectorized instructions in the resulting native machine code.
VecOpMicroBenchmark.java should benefit from auto vectorization:
/**
* Run with this command to show native assembly:<br/>
* java -XX:+UnlockDiagnosticVMOptions
* -XX:CompileCommand=print,VecOpMicroBenchmark.profile VecOpMicroBenchmark
*/
public class VecOpMicroBenchmark {
private static final int LENGTH = 1024;
private static long profile(float[] x, float[] y) {
long t = System.nanoTime();
for (int i = 0; i < LENGTH; i++) {
y[i] = y[i] + x[i]; // line 14
}
t = System.nanoTime() - t;
return t;
}
public static void main(String[] args) throws Exception {
float[] x = new float[LENGTH];
float[] y = new float[LENGTH];
// to let the JIT compiler do its work, repeatedly invoke
// the method under test and then do a little nap
long minDuration = Long.MAX_VALUE;
for (int i = 0; i < 1000; i++) {
long duration = profile(x, y);
minDuration = Math.min(minDuration, duration);
}
Thread.sleep(10);
System.out.println("\n\nduration: " + minDuration + "ns");
}
}
To find out if it gets vectorized, I did the following:
open eclipse and create the above file
right-click the file and from the dropdown menu, choose Run > Java Application (ignore the output for now)
in the eclipse menu, click Run > Run Configurations...
in the opened window, find VecOpMicroBenchmark, click it and choose the Arguments tab
in the Arguments tab, under VM arguments: put in this: -XX:+UnlockDiagnosticVMOptions -XX:CompileCommand=print,VecOpMicroBenchmark.profile
get libhsdis and copy (possibly rename) the file hsdis-amd64.so (.dll for windows) to java/lib directory. In my case, this was /usr/lib/jvm/java-11-openjdk-amd64/lib .
run VecOpMicroBenchmark again
It should now print lots of information to the console, part of it being the disassembled native machine code, which was produced by the JIT compiler. If you see lots of messages, but no assembly instructions like mov, push, add, etc, then maybe you can somewhere find the following message:
Could not load hsdis-amd64.so; library not loadable; PrintAssembly is disabled
This means that java couldn't find the file hsdis-amd64.so - it's not in the right directory or it doesn't have the right name.
hsdis-amd64.so is the disassembler which is required for showing the resulting native machine code. After the JIT compiler compiles the java bytecode to native machine code, hsdis-amd64.so is used to disassemble the native machine code to make it human readable. You can find more infos on how to get/install it at How to see JIT-compiled code in JVM? .
After finding assembly instructions in the output, I skimmed through it (too much to post all of it here) and looked for line 14. I found this:
0x00007fac90ee9859: nopl 0x0(%rax)
0x00007fac90ee9860: cmp 0xc(%rdx),%esi ; implicit exception: dispatches to 0x00007fac90ee997f
0x00007fac90ee9863: jnb 0x7fac90ee9989
0x00007fac90ee9869: movsxd %esi,%rbx
0x00007fac90ee986c: vmovss 0x10(%rdx,%rbx,4),%xmm0 ;*faload {reexecute=0 rethrow=0 return_oop=0}
; - VecOpMicroBenchmark::profile#16 (line 14)
0x00007fac90ee9872: cmp 0xc(%rdi),%esi ; implicit exception: dispatches to 0x00007fac90ee9997
0x00007fac90ee9875: jnb 0x7fac90ee99a1
0x00007fac90ee987b: movsxd %esi,%rbx
0x00007fac90ee987e: vmovss 0x10(%rdi,%rbx,4),%xmm1 ;*faload {reexecute=0 rethrow=0 return_oop=0}
; - VecOpMicroBenchmark::profile#20 (line 14)
0x00007fac90ee9884: vaddss %xmm1,%xmm0,%xmm0
0x00007fac90ee9888: movsxd %esi,%rbx
0x00007fac90ee988b: vmovss %xmm0,0x10(%rdx,%rbx,4) ;*fastore {reexecute=0 rethrow=0 return_oop=0}
; - VecOpMicroBenchmark::profile#22 (line 14)
So it's using the AVX instruction vaddss. But, if I'm correct here, vaddss means
add scalar single-precision floating-point values and this only adds one float value to another one (here, scalar means just one, whereas here single means 32 bit, i.e. float and not double).
What I expect here is vaddps, which means add packed single-precision floating-point values and which is a true SIMD instruction (SIMD = single instruction, multiple data = vectorized instruction). Here, packed means multiple floats packed together in one register.
About the ..ss and ..ps, see http://www.songho.ca/misc/sse/sse.html :
SSE defines two types of operations; scalar and packed. Scalar operation only operates on the least-significant data element (bit 0~31), and packed operation computes all four elements in parallel. SSE instructions have a suffix -ss for scalar operations (Single Scalar) and -ps for packed operations (Parallel Scalar).
Queston:
Is my java example incorrect, or why is there no SIMD instruction in the output?

In the main() method, put in i < 1000000 instead of just i < 1000. Then the JIT also produces AVX vector instructions like below, and the code runs faster:
0x00007f20c83da588: vmovdqu 0x10(%rbx,%r11,4),%ymm0
0x00007f20c83da58f: vaddps 0x10(%r13,%r11,4),%ymm0,%ymm0
0x00007f20c83da596: vmovdqu %ymm0,0x10(%rbx,%r11,4) ;*fastore {reexecute=0 rethrow=0 return_oop=0}
; - VecOpMicroBenchmark::profile#22 (line 14)
The code from the question is actually optimizable by the JIT compiler using auto-vectorization. However, as Peter Cordes pointed out in a comment, the JIT needs quite some processing, thus it is rather reluctant to decide that it should fully optimize some code.
The solution is simply to execute the code more often during one execution of the program, not just 1000 times, but 100000 times or a million times.
When executing the profile() method this many times, the JIT compiler is convinced that the code is very important and the overall runtime will benefit from full optimization, thus it optimizes the code again and then it also uses true vector instructions like vaddps.
More details in Auto Vectorization in Java

R code in Java working in Linux but not in Windows

What am I doing?
I am writing a data analysis program in Java which relies on R´s arulesViz library to mine association rules.
What do I want?
My purpose is to store the rules in a String variable in Java so that I can process them later.
How does it work?
The code works using a combination of String.format and eval Java and RJava instructions respectively, being its behavior summarized as:
Given properly formatted Java data structures, creates a data frame in R.
Formats the recently created data frame into a transaction list using the arules library.
Runs the apriori algorithm with the transaction list and some necessary values passed as parameter.
Reorders the generated association rules.
Given that the association rules cannot be printed, they are written to the standard output with R´s write method, capture the output and store it in a variable. We have converted the association rules into a string variable.
We return the string.
The code is the following:
// Step 1
Rutils.rengine.eval("dataFrame <- data.frame(as.factor(c(\"Red\", \"Blue\", \"Yellow\", \"Blue\", \"Yellow\")), as.factor(c(\"Big\", \"Small\", \"Small\", \"Big\", \"Tiny\")), as.factor(c(\"Heavy\", \"Light\", \"Light\", \"Heavy\", \"Heavy\")))");
//Step 2
Rutils.rengine.eval("transList <- as(dataFrame, 'transactions')");
//Step 3
Rutils.rengine.eval(String.format("info <- apriori(transList, parameter = list(supp = %f, conf = %f, maxlen = 2))", supportThreshold, confidenceThreshold));
// Step 4
Rutils.rengine.eval("orderedRules <- sort(info, by = c('count', 'lift'), order = FALSE)");
// Step 5
REXP res = Rutils.rengine.eval("rulesAsString <- paste(capture.output(write(orderedRules, file = stdout(), sep = ',', quote = TRUE, row.names = FALSE, col.names = FALSE)), collapse='\n')");
// Step 6
return res.asString().replaceAll("'", "");
What´s wrong?
Running the code in Linux Will work perfectly, but when I try to run it in Windows, I get the following error referring to the return line:
Exception in thread "main" java.lang.NullPointerException
This is a common error I have whenever the R code generates a null result and passes it to Java. There´s no way to syntax check the R code inside Java, so whenever it´s wrong, this error message appears.
However, when I run the R code in brackets in the R command line in Windows, it works flawlessly, so both the syntax and the data flow are OK.
Technical information
In Linux, I am using R with OpenJDK 10.
In Windows, I am currently using Oracle´s latest JDK release, but trying to run the program with OpenJDK 12 for Windows does not solve anything.
Everything is 64 bits.
The IDE used in both operating systems is IntelliJ IDEA 2019.
Screenshots
Linux run configuration:
Windows run configuration:

Apply corretion formula to each channel for 5D image

I'm completely new to groovy and have only limited experience in writing ImageJ macros, so this might be a really easy problem to fix, but here goes:
I have a 5D hyperstack (3 channels, 3 slices, ~100 stage positions) which suffer from a highly uneven illumination. I have found a rather straightforward formula for correcting for this. The formula is
where C is the corrected image, R is the raw image, D is the dark-field image, F is the flat-field image and m is the image-averaged value of (F-D).
For single-channel 3D images (x, y, p) this is relatively straight-forward and does not need any macro scripting, but for multi-channel, multi-slice 5D images, I would have to at the very least split the image into individual channels before I can apply the correction formula to each channel and then recombine them. I have been trying to write a macro using groovy to handle this for me (I chose groovy because I was told it was more user-friendly, so I'm open to other suggestions), but I can't seem to get it running. Currently, I have the code below (I left out the dark-field because my images are already corrected for this during acquisition):
import ij.*
import ij.plugin.filter.ImageMath
import ij.process.*
import ij.gui.*
import java.awt.*
import ij.plugin.*
class My_Plugin implements PlugIn {
void run(java.lang.String arg) {
ImagePlus flatfield = WindowManager.getImage("flatfield.tif")
ImagePlus rawstack = WindowManager.getImage("Untitled.tif")
ImagePlus correctedstack = IJ.createImage("HyperStack", "32-bit composite-mode", 512, 512, 3, rawstack.z, 1)
float m;
for (c in flatfield.c) {
flatfield.setC(c)
rawstack.setC(c)
correctedstack.setC(c)
m = flatfield.getStatistics().mean
rawstack.z.each { z ->
rawstack.setZ(z)
correctedstack.setZ(z)
if (m > 0) {
rawstack.processor.multiply(m)
correctedstack.processor.divide(flatfield)
}
}
}
correctedstack.show()
}
}
new My_Plugin().run()
This code currently fails with the following exception (but I suspect the code itself is generally poorly written):
groovy.lang.MissingMethodException: No signature of method: ij.process.FloatProcessor.divide() is applicable for argument types: (ij.CompositeImage) values: [img["flatfield.tif" (-274), 8-bit, 512x512x3x3x1]] Possible solutions: dilate(), dilate(), erode(), erode(), find(), noise(double)
Any help is greatly appreciated!

I can just explain the Groovy error you encounter, not how to fix your implementation of the correction formula. It says that FloatProcessor doesn't have a method with the below signature. Neither its own nor inherited nor through other Groovy mechanics, e.g. MOP, AST or Extensions
divide(ij.CompositeImage img)
The ImageJ's source code confirms that neither FloatProcessor nor its superclass ImageProcessor don't have such a method.

cannot allocate vector of size n MB in R while executing through Java

I am facing an error while executing R code through java using Rengine. I don't know how to set max memory in java code using Rengine. Is there a way to set max memory for Rengine? May be like below
Rengine re = getREngine();
re.eval("options(java.parameters = '-Xmx10g')", false);
I know this is wrong syntax. Kindly guide me how to set this.

Have you seen this: http://www.bramschoenmakers.nl/en/node/726
> options( java.parameters = "-Xmx4g" )
> library( "rJava" )
but I think this options set max memory for java being executed from R. However, you would like to exceed R memory, where R is executed from Java so maybe you should extend R memory: memory.limit(memory.limit()*2)