I have once again run out of ideas for an error and returned to the helpful Stack Overflow community for guidance. I understand that this is a very long post to wade through, but I believe that it could be helpful to others in the community, and provide a challenge for veterans.
Essentially, I am building an app that processes a video frame by frame. Initially, this process was done linearly, but speed is a major concern for this application. The natural thought was that seperate threads could be implemented to process the frames. Each thread would simply be a single iteration of the previous loop; retrieving the frame from the video, processing it, and placing the resulting data into the correct index of an array.
The issue is that during the transition from a loop to threads, a NullPointerException is sometimes thrown (Roughly once every 200 frames), when attempting to access the frame returned by MediaMetadataRetriever.getFrameAtTime()
Here are some sections of the code that may be of use:
Excerpt from the onClickListener attached to the button that starts the processing. The onClickListener starts a ThreadPoolExecutor which manages the threads for frame processing.
long videoLength = getVideoLength(videoUri);
coords = new coordinate[(int)(videoLength/frameIntervalInMicroSeconds)];
ThreadPoolExecutor executor = new ThreadPoolExecutor(
Runtime.getRuntime().availableProcessors()+1,
Runtime.getRuntime().availableProcessors()+1,
1,
TimeUnit.SECONDS,
new LinkedBlockingQueue<Runnable>());
for (long a = 0; a < videoLength; a += frameIntervalInMicroSeconds) {
new ProcessFrame().executeOnExecutor(executor, (long) a);
getVideoFrame(), a helper method that takes in a Uri for a video, and a time in microseconds, and returns the frame of the video as a Bitmap
private Bitmap getVideoFrame(Uri uri, long timeInUSeconds) {
MediaMetadataRetriever retriever = new MediaMetadataRetriever();
retriever.setDataSource(this, uri);
Bitmap temp = retriever.getFrameAtTime(timeInUSeconds, MediaMetadataRetriever.OPTION_CLOSEST);
retriever.release();
return temp;
}
Excerpt from ProcessFrame, the thread (as an ASyncTask) that is run by the ThreadPoolExecutor. As mentioned above, the thread simply gets the frame, processes the frame, and places it into the correct location in the array.
protected Void doInBackground(Long... frameTime){
/* Excluded variable declarations */
Bitmap bMap;
synchronized (this) {
bMap = getVideoFrame(videoUri, frameTime[0]);
}
try {
//This line below is where the NullPointerException is thrown
bMap = Bitmap.createScaledBitmap(bMap, bMap.getWidth() / RESIZE_FACTOR, bMap.getHeight() / RESIZE_FACTOR, true);
}catch (Exception e){
System.out.println("Caught NullPointerException");
return null;
}
/* I excluded the frame processing done here */
//add coordinate to the list
try {
synchronized (this){
coords[(int) (frameTime[0] / frameIntervalInMicroSeconds)] = temp;
}
}catch (Exception e){
System.out.println("Caught out of bounds coordinate");
}
/* excluded clean up */
}
Error message produced when a null frame is accessed:
7043-7856/com.example.tyler.laserphysicsappb E/MediaMetadataRetrieverJNI﹕ getFrameAtTime: videoFrame is a NULL pointer
General comments, observations, and things I have tried:
The line where the exception is thrown is marked with a comment in the last code block.
Without the try/catch block, my phone does not not display the normal app crashed message on this exception, it flashes a black screen, and then quickly returns to the home screen. (I discovered which line it was by quickly taking a screenshot of logcat while the black screen flashed)
Without the try/catch block, another phone that I tried simply ignores the error and continues, but this ruins the results.
Adding the synchronized block around bMap = getVideoFrame(videoUri, frameTime[0]) seems to have made the error less common, but it still happens.
I have tried the ffmpegMediaMetadataRetriever alternative. This package did not eliminate the error, and slowed down the processing time.
Once the nullPointerException has been thrown, all subsequent threads seem to be more likely to throw the exception again.
Running the ThreadPoolExecutor with a maximum of one thread does not produce the error.
Even if I enclose the entire body of doInBackground() in a synchronize block, the error still appears
I have been able to fix the bug, but with lackluster results. The only way I have found to stop the exception from appearing is by putting a synchronize block around the MediaMetadataRetriever getFrameAtTime() function call. The new getVideoFrame method is below.
private Bitmap getVideoFrame(Uri uri, long timeInUSeconds) {
MediaMetadataRetriever retriever = new MediaMetadataRetriever();
retriever.setDataSource(this, uri);
synchronize (this) {
Bitmap temp = retriever.getFrameAtTime(timeInUSeconds, MediaMetadataRetriever.OPTION_CLOSEST);
}
retriever.release();
return temp;
}
Sadly, as this fixes the bug, the speed concern is not aleviated, as getting the frame still takes a substantial amount of time, far more so than actually processing the frames. Nevertheless, the pr
Related
There I have a loop:
public void updateDrawee(View view) {
if (begin) {
begin = false;
for (int i = 0; i < 5; i++) {
CloseableReference<CloseableImage> reference = createBitmapRefer(i);
Log.i("reference", reference+"");
imgList.add(reference);
}Log.i("imgList", imgList.toString());Log.i("imgList.0", imgList.get(0)+"");
}
//...some code
}
and the method createBitmapRefer(int count) follow:
public CloseableReference<CloseableImage> createBitmapRefer(int count) {
ImagePipeline pipeline = Fresco.getImagePipeline();
int[] drawableIds = {R.drawable.alpha1, R.drawable.alpha2,
R.drawable.alpha3, R.drawable.alpha4, R.drawable.alpha5};
ImageRequest levelRequest
= ImageRequestBuilder.newBuilderWithResourceId(drawableIds[count])//++
.setProgressiveRenderingEnabled(true)//逐行加载
.build();
CloseableReference<CloseableImage> bmpReference = null;
DataSource<CloseableReference<CloseableImage>> dataSource
= pipeline.fetchImageFromBitmapCache(levelRequest, this);
try {
if (!dataSource.hasResult()) {
dataSource = pipeline.fetchDecodedImage(levelRequest, this);
}
//count %= 5;
Log.i("dataSource has result", dataSource.hasResult() +"");
Log.i("dataSource fail?", dataSource.hasFailed() + "");
bmpReference = dataSource.getResult();
Log.i("bmpRefer", bmpReference+"");
if (bmpReference != null) {
CloseableReference<CloseableImage> returnRef;
returnRef = bmpReference.clone();
return returnRef;
}else {
return null;
}
}finally {
dataSource.close();
CloseableReference.closeSafely(bmpReference);
}
}
when I debug, if i click step into and see the code step by step, it will return a CloseableReference just as I want, and the imgList(its a ArrayList) can get the element too.BUT if I step over the for loop, it return nothing!
Is there any different between keep looking at it or not???
the watches show elements in imgList, when index=1 and 4, I clicked step into.
and the logcat show what Log.i() print.
Or because I have not use this classCloseableReference in Standardized way?
Let me try to explain what happens here.
You are not using Fresco in the intended way. I will step back for a moment and strongly suggest that you use SimpleDraweView if you just need to display images. If however you really need the underlying bitmap, you can get it from the ImagePipeline in way similar to what you already doing, but with one key difference. Fetching images happens asynchronously. What that means is that you can't just do dataSource = pipeline.fetchDecodedImage(...) and then immediately dataSource.getResult. If the image was not found in the memory cache, getResult will just return null. What you need to do instead is to subscribe to the DataSource as explained in the Fresco documentation. I strongly suggest that you read those few chapters about ImagePipeline if you intend to use it directly. Otherwise you may cause your app to leak the memory or to crash because of rendering recycled bitmap.
What you see in debugger is exactly what I described above. The array of size 5 looks like this:
0 = null,
1 = CloseableReference#4908,
2 = null,
3 = null,
4 = CloseableReference#5231
The AndroidStudio UI just hides the null entries for brevity. You can turn this off if you don't like it by right-clicking there and opening options. The reason you get something for 1 and 4 is because the image has been found in the bitmap memory cache and was retrieved from it immediately. The reason you get null for 0, 2 and 3 is because the image has not been loaded yet. Fresco might need to download the image, and even if it is already downloaded and in the disk cache, it may need to decode the image. All of this takes some time and is not instantaneous. That's why you need to subscribe your callback and ImagePipeline will notify you when the image is ready.
I'm building a barcode scanner which, different from other implementations, does the scanning part continuously in the background rather than waiting the user to trigger the process.
Now, the most (or what I think is the most) obvious way to achieve this is to process the scanning part in another thread to make sure that the main thread won't be interrupted. So that the user won't be bothered with UI lags, stutters, and whatnot.
I'm not the brightest guy when it comes to concurrency. But I've did my homework and done some research about it which, in turn has lead me to write this:
...
mScannerExecutor = Executors.newFixedThreadPool(3);
...
Camera.PreviewCallback previewCallback = new Camera.PreviewCallback() {
public void onPreviewFrame(byte[] data, Camera camera) {
Camera.Parameters parameters = camera.getParameters();
Camera.Size size = parameters.getPreviewSize();
final Image barcode = new Image(size.width, size.height, "Y800");
barcode.setData(data);
Runnable scan = new Runnable() {
#Override
public void run() {
int result = mBarcodeScanner.scanImage(barcode);
if (result != 0) {
if(isInPreview) {
isInPreview = false;
mCamera.stopPreview();
}
SymbolSet symbolSet = mBarcodeScanner.getResults();
mListener.onBarcodeScanned(symbolSet.iterator().next());
if (enableRepeatedScanning) {
new Handler().postDelayed(restartPreview, mRescanIntervalMillis);
}
}
}
};
mScannerExecutor.execute(scan);
}
};
But the above code has been causing a lot of error in its execution. I can't even keep the app running for more than a mere couple of seconds. The error message varies from time to time, but this below was shown the most:
Fatal signal 8 (SIGFPE), code -6, fault addr 0x17b8 in tid 6410 (pool-1-thread-1)
I have a strong feeling that this design in general is heavily flawed. Thus the constant crashing.
What can I do to make this right? Did I miss something really important here?
p.s., The previewCallback defined above will be called very frequently; once every 2000ms (2 secs).
I'm trying to visualize various graph-algorithms. I want to make it so, that after every 2 seconds the graph get updated and gets repainted. I have tried using the Thread.sleep() method but it just freezes the GUI and then after a while is done with the complete algorithm.
(I am fairly new to Java so don't be to harsh with the code)
The Code in question:
else if(ae.getSource() == fordFulkersonButton){
dinicButton.setEnabled(false);
edmondsKarpButton.setEnabled(false);
gtButton.setEnabled(false);
if(checkbox.isEnabled()){
fordFulkersonButton.setEnabled(false);
while(!fordFulkerson.getIsDone()){
flowNetwork = fordFulkerson.algoFF(flowNetwork);
popupText.setVisible(true);
Integer i = new Integer(flowNetwork.getCurrentFlow());
String s = i.toString();
popupText.setText("Aktueller Fluß: "+s);
graphDrawer.setFlowNetwork(flowNetwork);
this.showFrame();
try {
Thread.sleep(2 * 1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
Integer in = new Integer(flowNetwork.getCurrentFlow());
String st = in.toString();
popupText.setText("Algorithmus is beendet mit Fluss: "+st);
}
flowNetwork = fordFulkerson.algoFF(flowNetwork);
popupText.setVisible(true);
Integer i = new Integer(flowNetwork.getCurrentFlow());
String s = i.toString();
if(fordFulkerson.getIsDone()){
popupText.setText("Algorithmuss beednet mit maximalen Fluß: "+s);
}else{
popupText.setText("Aktueller Fluß: "+s);
}
graphDrawer.setFlowNetwork(flowNetwork);
this.showFrame();
}
Doing huge amounts of work in a UI thread freezes the UI. If you want to do animations or complex work, use a worker thread or a Swing timer.
You must not call Thread.sleep() in Swing UI code. Instead, you need to think like an animator: Show one frame of the animation at a time. Some external source will call your code to show the next frame. The external source is the Swing timer. Your code draws the next frame and returns. That way, you never block the UI thread for long.
Google for "swing animation". Interesting results are:
http://www.java2s.com/Tutorial/Java/0240__Swing/Timerbasedanimation.htm
http://zetcode.com/tutorials/javagamestutorial/animation/
I have a problem when a JButton on my interface, when pressed the button triggers this event listener:
if (event.getSource() == goButton)
{
MainLauncher.doProgram(assetsField.getText(), FileFinder.completePath(String.format("%s\\%s", modDirectoryPath, modList.getSelectedValue())), databaseField.getText());
}
which runs the following code:
public static void doProgram(final String baseGamePath, final String modPath, final String databasePath)
{
new Thread()
{
public void run()
{
System.out.println("Running: " + modPath + "\n");
reader = new MetaDataReader(databasePath);
reader.formConnection();
long start = System.currentTimeMillis();
long temp;
File cache = new File("RegisterCache.SC");
if (!cache.exists())
{
temp = System.currentTimeMillis();
start += System.currentTimeMillis() - temp;
System.out.println("Calculating number of files");
FileFinder baseGame = new FileFinder();
baseGame.calculateNumFiles(baseGamePath);
System.out.println(String.format("Loading %s base game files", baseGame.getNumFiles()));
gui.doProgressBar();
baseGame.findFiles(baseGamePath, true);
gui.killProgressBar();
System.out.println();
//load the base game assets, we want to move this to metadata later
System.out.println("Checking for base game reference errors");
new ReferenceDetector(Table.getRegister()).findReferences(true);
Table.serializeTable(cache); //write the database to cache, we can use it next time
}
else
{
System.out.println("Loading cache");
Table.unserializeTable(cache);
}
gui.doCommands(); //calls reset()!!! be careful with the temporary excludes!
System.out.println("Eliminating base game requires errors");
RequiresWhere.executeRequires(true);
temp = System.currentTimeMillis();
start += System.currentTimeMillis() - temp;
System.out.println("Calculating number of files");
FileFinder mod = new FileFinder();
mod.calculateNumFiles(modPath);
System.out.println(String.format("Loading %s mod files", mod.getNumFiles()));
gui.doProgressBar();
mod.findFiles(modPath, false);
gui.killProgressBar();
System.out.println();
System.out.println("Finding reference errors");
new ReferenceDetector(Table.getRegister()).findReferences(false);
System.out.println("Finding requires errors");
RequiresWhere.executeRequires(false);
System.out.println("Checks complete");
reader.killConnection();
System.out.println(String.format("Execution Time: %dms", System.currentTimeMillis() - start));
Table.reset();
}
}.run();
}
The odd thing is this:
When testing I didn't have my button added to the interface, and so on main() I was calling goButton.doClick(); which triggered the doProgram(), and gives me the results I expected which is this:
a couple of things are printed out (to a JTextArea on the interface that receives from System.out.println()), a progress bar pops up above the console, and goes from 0% to 100% in around 10 seconds, disappears, a few more things are printed etc etc.
However when I added the button to the interface I clicked it and the behavior was not the same at all:
Nothing is printed. The progress Bar comes up, and stays at 0% for 10 seconds, then disappears, a few seconds later the doProgram finishes executing and all the information that should have been printed during runtime is suddenly printed all at once.
Can anybody tell me what is causing this strange behavior and how it can be fixed?
I would post a SSCCE, but it is a little difficult to do for this program as there are so many tasks and subprocesses. Note that, the progress bar I use it run on the GUI thread, the main code is done in it's own thread (as you can see)
EDIT: I added setEnabled(false); and true inside the listener surrounding the call to doProgram, and when the button is programatically pressed, the button is greyed out and this renabled as I expect, however if I click the button by hand it does not disable, and I can press it again
The code looks a bit confusing.
However, first of all: You are calling the run() method of the new Thread that you are creating there. This will cause the code from the run() method to be executed by the thread that calls doProgram, and not by the new Thread that you are creating there. In order to execute the code really in this new Thread, you have to call start() instead of run().
This explains probably most of the differences that you described. However, keep in mind that modifications to Swing components always have to be done on the Event Dispatch Thread. So whatever you are doing, for example, with gui.doCommands(), make sure that you don't violate this Single Thread Rule.
Edit: If anyone also has any other recommendations for increasing performance of screen capture please feel free to share as it might fully address my problem!
Hello Fellow Developers,
I'm working on some basic screen capture software for myself. As of right now I've got some proof of concept/tinkering code that uses java.awt.Robot to capture the screen as a BufferedImage. Then I do this capture for a specified amount of time and afterwards dump all of the pictures to disk. From my tests I'm getting about 17 frames per second.
Trial #1
Length: 15 seconds
Images Captured: 255
Trial #2
Length: 15 seconds
Images Captured: 229
Obviously this isn't nearly good enough for a real screen capture application. Especially since these capture were me just selecting some text in my IDE and nothing that was graphically intensive.
I have two classes right now a Main class and a "Monitor" class. The Monitor class contains the method for capturing the screen. My Main class has a loop based on time that calls the Monitor class and stores the BufferedImage it returns into an ArrayList of BufferedImages.
If I modify my main class to spawn several threads that each execute that loop and also collect information about the system time of when the image was captured could I increase performance? My idea is to use a shared data structure that will automatically sort the frames based on capture time as I insert them, instead of a single loop that inserts successive images into an arraylist.
Code:
Monitor
public class Monitor {
/**
* Returns a BufferedImage
* #return
*/
public BufferedImage captureScreen() {
Rectangle screenRect = new Rectangle(Toolkit.getDefaultToolkit().getScreenSize());
BufferedImage capture = null;
try {
capture = new Robot().createScreenCapture(screenRect);
} catch (AWTException e) {
e.printStackTrace();
}
return capture;
}
}
Main
public class Main {
public static void main(String[] args) throws InterruptedException {
String outputLocation = "C:\\Users\\ewillis\\Pictures\\screenstreamer\\";
String namingScheme = "image";
String mediaFormat = "jpeg";
DiscreteOutput output = DiscreteOutputFactory.createOutputObject(outputLocation, namingScheme, mediaFormat);
ArrayList<BufferedImage> images = new ArrayList<BufferedImage>();
Monitor m1 = new Monitor();
long startTimeMillis = System.currentTimeMillis();
long recordTimeMillis = 15000;
while( (System.currentTimeMillis() - startTimeMillis) <= recordTimeMillis ) {
images.add( m1.captureScreen() );
}
output.saveImages(images);
}
}
Re-using the screen rectangle and robot class instances will save you a little overhead. The real bottleneck is storing all your BufferedImage's into an array list.
I would first benchmark how fast your robot.createScreenCapture(screenRect); call is without any IO (no saving or storing the buffered image). This will give you an ideal throughput for the robot class.
long frameCount = 0;
while( (System.currentTimeMillis() - startTimeMillis) <= recordTimeMillis ) {
image = m1.captureScreen();
if(image !== null) {
frameCount++;
}
try {
Thread.yield();
} catch (Exception ex) {
}
}
If it turns out that captureScreen can reach the FPS you want there is no need to multi-thread robot instances.
Rather than having an array list of buffered images I'd have an array list of Futures from the AsynchronousFileChannel.write.
Capture loop
Get BufferedImage
Convert BufferedImage to byte array containing JPEG data
Create an async channel to the output file
Start a write and add the immediate return value (the future) to your ArrayList
Wait loop
Go through your ArrayList of Futures and make sure they all finished
I guess that the intensive memory usage is an issue here. You are capturing in your tests about 250 screenshots. Depending on the screen resolution, this is:
1280x800 : 250 * 1280*800 * 3/1024/1024 == 732 MB data
1920x1080: 250 * 1920*1080 * 3/1024/1024 == 1483 MB data
Try caputuring without keeping all those images in memory.
As #Obicere said, it is a good idea to keep the Robot instance alive.