Develop Reference

Slow Multithreading in Java - Air Percussion project

I am creating "Air Percussion" using IMU sensors and Arduino to communicate with computer (3 separate IMUs and Arduinos). They are connected to the computer through USBs. I am gathering data on separate Threads (each thread for each sensor). When I connect only one "set" my program is working really fast. I can get even 5 plays of sound per second. Unfortunatelly when i am trying to connect 3 sensors and run them on separate Threads at the same time my program slows down horribly. Even when im moving only one of sensors, I can get like 1 "hit" per second and sometimes it's even losing some of the sounds it should play. I'll show only important parts of the code below.
In the main i've got ActionListener for button, where it should start gathering the data. I run there 3 separate Threads for each USB Port.
connectButton.addActionListener(new ActionListener(){
#Override public void actionPerformed(ActionEvent arg0) {
int dialogButton = 1;
if(!flagaKalibracjiLewa || !flagaKalibracjiPrawa){ //some unimportant flags
dialogButton = JOptionPane.showConfirmDialog(null, "Rozpoczynając program bez kalibracji będziesz miał do dyspozycji mniejszą ilość dzwięków. Czy chcesz kontynuować?","Warning",JOptionPane.YES_NO_OPTION);
}else{
dialogButton = JOptionPane.YES_OPTION;
}
if(dialogButton == JOptionPane.YES_OPTION){
if(connectButton.getText().equals("Connect")) {
if(!flagaKalibracjiLewa && !flagaKalibracjiPrawa) podlaczPorty();
Thread thread = new Thread(){
#Override public void run() {
Scanner data = new Scanner(chosenPort.getInputStream());
dataIncoming(data, "lewa");
data.close();
}
};
Thread thread2 = new Thread(){
#Override public void run() {
Scanner data = new Scanner(chosenPort2.getInputStream());
dataIncoming(data, "prawa");
data.close();
}
};
Thread thread3 = new Thread(){
#Override public void run() {
Scanner data = new Scanner(chosenPort3.getInputStream());
dataIncoming(data, "stopa");
data.close();
}
};
thread.start();
thread2.start();
thread3.start();
connectButton.setText("Disconnect");
} else {
// disconnect from the serial port
chosenPort.closePort();
chosenPort2.closePort();
chosenPort3.closePort();
portList.setEnabled(true);
portList2.setEnabled(true);
portList3.setEnabled(true);
connectButton.setText("Connect");
}
}
}
});
in "dataIncoming" method there is bunch of not important things (like picking, which sound should be played etc.). The important part is in the while loop. In the "while" im gathering next lines of data from sensor. When one of the values is higher than something it should play a sound but only if some time has passed and the sensor has moved a certain way. (when the drumstick is going down the "imuValues[4]" is increasing, when its going up its decreasing, so when its past 160 it means that the player has taken the drumstick up so its ready for the next hit)
while(data.hasNextLine()) {
try{
imuValues = data.nextLine().split(",");
if(Double.parseDouble(imuValues[4])>200 && flagaThreada) {
flagaThreada = false;
playSound(sound1);
}
if(Double.parseDouble(imuValues[4])<160 && System.currentTimeMillis()-startTime>100) {
flagaThreada = true;
startTime=System.currentTimeMillis();
}
}catch(Exception e){
System.out.println("ERROR");
}
}
and finally the method for playing the sound is :
public static synchronized void playSound(String sound) {
try {
String url = "/sounds/"+sound+".wav";
Clip clip = AudioSystem.getClip();
AudioInputStream inputStream = AudioSystem.getAudioInputStream(
Main.class.getResourceAsStream(url));
clip.open(inputStream);
clip.start();
} catch (Exception e) {
System.err.println("ERROR IN OPENING");
}
}
Is my computer to slow to compute and play sounds for 3 sensors at the same time? Or is there a way to create those Threads in a better fashion?

I wrote a version of Clip, called AudioCue, which allows multi-threading on the play commands. It is open source, BSD license (free), consists of three files which you can cut and paste into your program. There is also an API link for it. More info at AudioCue. The site has code examples as well as link to API and source code. There is also some dialogue about its use at Java-gaming.org, under the "Sound" topic thread.
The basic principle behind the code is to make the audio data available in a float array, and send multiple, independent "cursors" through it (one per play command). The setup lets us also do real time volume fading, pitch changes and panning. The audio is output via a SourceDataLine which you can configure (set thread priority, buffer size).
I'm maybe a week or two away from sharing a more advanced version that allows all AudioCues to be mixed through a single output line. This version has five classes/interfaces instead of three, and is being set up for release on github. I'm also hoping to get a donate button and the like set up for this next iteration. The next version might be more useful for Arduino in that I believe you are only allowed up to 8 audio outputs on that system.
Other than that, the steps you have taken (separating the open from the play, using setFramePosition for restarts) are correct. I can't think of anything else to add to help out besides writing your own mixer/cue player (as I have done and am willing to share).

For Loop delay not working Android

I am a newbie to android development so bear with me. I have scoured the site and implemented several suggested answers but I am still running into a problem. I want to iterate through an object array and display the contents of the array with delay to allow user interaction(The user gets to say if the object and the text displayed is what they were looking for by clicking on a yes or no button). My objects however display last first and then start zooming real fast in a seemingly endless loop. Here is the method I call to load the images:
private void displayInstructions() {
for (Emergency_Instructions instruction : instructions) {
final Emergency_Instructions instruction2 = instruction;
final Handler handler = new Handler();
handler.postDelayed(new Runnable()
{
public void run() {
displayimages(instruction2.getStep(), instruction2.getStepImage());
handler.postDelayed(this, 5000);
}
}, 5000);
}
void displayimages(String text, Bitmap image)
{
instructiontext.setText(text);
instructionbmp.setImageBitmap(image);
}
Any help will be greatly appreciated

You create multiple Handlers where you need only one. You tell each one to post a Runnable to the current thread's message queue, to be run after the same delay. That delay expires at pretty much the same time for all the tasks, so they are then executed one right after another, as fast as the queue can go. Each of those tasks also posts a message via the handler, to be run after the same delay. Once that delay expires, that will produce a second group of messages posted rapidly one right after the other, as fast as the device can go.
If you just want to schedule what is effectively a slide show, you might do it like this:
private void displayInstructions() {
final Handler handler = new Handler();
int delay = 0;
int step = 5000;
for (Emergency_Instructions instruction : instructions) {
final Emergency_Instructions instruction2 = instruction;
handler.postDelayed(new Runnable() {
public void run() {
displayimages(instruction2.getStep(), instruction2.getStepImage());
// handler.postDelayed(this, 5000);
}
}, delay);
delay += step;
}
}
Note there that each task is posted with a larger delay than the last -- this is a delay with respect to when the message it posted, not with respect to when the preceding message was displayed.
With that said, I'm doubtful that this is really what you want to do, or at least that it is all you want to do. As it stands, this approach will cause all the messages to be displayed (eventually) regardless of any user interaction. At minimum, you will probably want to provide for subsequent messages to be canceled in the event that the user accepts one, or cancels the overall operation.

NullPointerException when threading MediaMetadataRetriever [Android]

I have once again run out of ideas for an error and returned to the helpful Stack Overflow community for guidance. I understand that this is a very long post to wade through, but I believe that it could be helpful to others in the community, and provide a challenge for veterans.
Essentially, I am building an app that processes a video frame by frame. Initially, this process was done linearly, but speed is a major concern for this application. The natural thought was that seperate threads could be implemented to process the frames. Each thread would simply be a single iteration of the previous loop; retrieving the frame from the video, processing it, and placing the resulting data into the correct index of an array.
The issue is that during the transition from a loop to threads, a NullPointerException is sometimes thrown (Roughly once every 200 frames), when attempting to access the frame returned by MediaMetadataRetriever.getFrameAtTime()
Here are some sections of the code that may be of use:
Excerpt from the onClickListener attached to the button that starts the processing. The onClickListener starts a ThreadPoolExecutor which manages the threads for frame processing.
long videoLength = getVideoLength(videoUri);
coords = new coordinate[(int)(videoLength/frameIntervalInMicroSeconds)];
ThreadPoolExecutor executor = new ThreadPoolExecutor(
Runtime.getRuntime().availableProcessors()+1,
Runtime.getRuntime().availableProcessors()+1,
1,
TimeUnit.SECONDS,
new LinkedBlockingQueue<Runnable>());
for (long a = 0; a < videoLength; a += frameIntervalInMicroSeconds) {
new ProcessFrame().executeOnExecutor(executor, (long) a);
getVideoFrame(), a helper method that takes in a Uri for a video, and a time in microseconds, and returns the frame of the video as a Bitmap
private Bitmap getVideoFrame(Uri uri, long timeInUSeconds) {
MediaMetadataRetriever retriever = new MediaMetadataRetriever();
retriever.setDataSource(this, uri);
Bitmap temp = retriever.getFrameAtTime(timeInUSeconds, MediaMetadataRetriever.OPTION_CLOSEST);
retriever.release();
return temp;
}
Excerpt from ProcessFrame, the thread (as an ASyncTask) that is run by the ThreadPoolExecutor. As mentioned above, the thread simply gets the frame, processes the frame, and places it into the correct location in the array.
protected Void doInBackground(Long... frameTime){
/* Excluded variable declarations */
Bitmap bMap;
synchronized (this) {
bMap = getVideoFrame(videoUri, frameTime[0]);
}
try {
//This line below is where the NullPointerException is thrown
bMap = Bitmap.createScaledBitmap(bMap, bMap.getWidth() / RESIZE_FACTOR, bMap.getHeight() / RESIZE_FACTOR, true);
}catch (Exception e){
System.out.println("Caught NullPointerException");
return null;
}
/* I excluded the frame processing done here */
//add coordinate to the list
try {
synchronized (this){
coords[(int) (frameTime[0] / frameIntervalInMicroSeconds)] = temp;
}
}catch (Exception e){
System.out.println("Caught out of bounds coordinate");
}
/* excluded clean up */
}
Error message produced when a null frame is accessed:
7043-7856/com.example.tyler.laserphysicsappb E/MediaMetadataRetrieverJNI﹕ getFrameAtTime: videoFrame is a NULL pointer
General comments, observations, and things I have tried:
The line where the exception is thrown is marked with a comment in the last code block.
Without the try/catch block, my phone does not not display the normal app crashed message on this exception, it flashes a black screen, and then quickly returns to the home screen. (I discovered which line it was by quickly taking a screenshot of logcat while the black screen flashed)
Without the try/catch block, another phone that I tried simply ignores the error and continues, but this ruins the results.
Adding the synchronized block around bMap = getVideoFrame(videoUri, frameTime[0]) seems to have made the error less common, but it still happens.
I have tried the ffmpegMediaMetadataRetriever alternative. This package did not eliminate the error, and slowed down the processing time.
Once the nullPointerException has been thrown, all subsequent threads seem to be more likely to throw the exception again.
Running the ThreadPoolExecutor with a maximum of one thread does not produce the error.
Even if I enclose the entire body of doInBackground() in a synchronize block, the error still appears

I have been able to fix the bug, but with lackluster results. The only way I have found to stop the exception from appearing is by putting a synchronize block around the MediaMetadataRetriever getFrameAtTime() function call. The new getVideoFrame method is below.
private Bitmap getVideoFrame(Uri uri, long timeInUSeconds) {
MediaMetadataRetriever retriever = new MediaMetadataRetriever();
retriever.setDataSource(this, uri);
synchronize (this) {
Bitmap temp = retriever.getFrameAtTime(timeInUSeconds, MediaMetadataRetriever.OPTION_CLOSEST);
}
retriever.release();
return temp;
}
Sadly, as this fixes the bug, the speed concern is not aleviated, as getting the frame still takes a substantial amount of time, far more so than actually processing the frames. Nevertheless, the pr

android: taking pictures in task or thread at regular interval?

I'm writing an android app which should take pictures in a user-defined interval (20 sec - 1 min). It should take the pictures even while it is running in background or while the device is sleeping. The app will run for a very long time period. If it is necessary to wake up the device, it should put back to sleep as soon as possible to save batterie life. After taking a picture the app will process some additional work (comparison of two pictures).
I read some stuff about sheduling alarms (http://developer.android.com/training/scheduling/alarms.htm), creating Services (also # android training) and Android AsyncTasks, Java threads (http://www.mergeconflict.net/2012/05/java-threads-vs-android-asynctask-which.html)
... but I'm still not sure what is the best way to achieve this.
My questions are:
Should I use thread or a task to take the pictures in background?
(the comparison of the two pictures might take longer than a few
milliseconds but i don't know anything about the cpu load of this
operation)
Should I use an alarm to wake the device up or are there any alternative solutions?
How can both (alarms and thread/task) work together? (Include the Alarm in the Task/Thread?)
Many thanks for your help in advance.

As to our question I know I can help get started with the aspect of repeating the picture taking task at a user defined time interval. For such a task you can user a Timer to achieve this. The code would look something like this:
mTmr = new Timer();
mTsk = new TimerTask() {
#Override
public void run() {
//Take picture or do whatever you want
}
};
mTmr.schedule(mTsk, 0, USER_DEFINED_EXECUTION_INTERVAL);
schedule begins the timer. The first parameter of schedule used here is the task to run which is mTsk. The second parameter is the delay until the first execution (in milliseconds), in this case no delay. The third parameter is what you'll want to manipulate which is the interval of executions. The parameter is the time between executions so if it were 20 seconds you'd pass in 20,000. If it were a minute it would be 60,000. You can get this value from the user using any method you'd like.
To keep the timer running make sure you don't call mTmr.cancel in onPause because for your case you want to keep the timer running while the user isn't on the app. Not calling cancel means the timer will hold it's resources until the app is closed by the user.
OR you can look at this How to schedule a periodic task in Java? If you'd like to use ScheduledExecutorService instead of a Timer.

I have made this app - Lenx. It uses Camera extensively and I am processing image in the background. I have used AsyncTask to process the image and it has never given any problems. The app also has a timer which starts the process after certain interval. The logic that I have used is very simple.
I have not used Camera2 API yet, so the code might be deprecated. I created CameraPreview class which implements Camera.PreivewCallback.
#Override
public void onPreviewFrame(byte[] data, Camera camera) {
if (data == null) {
return;
}
int expectedBytes = previewWidth * previewHeight *
ImageFormat.getBitsPerPixel(ImageFormat.NV21) / 8;
if (expectedBytes != data.length) {
Log.e(TAG, "Mismatched size of buffer! Expected ");
mState = STATE_NO_CALLBACKS;
mCamera.setPreviewCallbackWithBuffer(null);
return;
}
if (mProcessInProgress || mState == STATE_PROCESS_IN_PROGRESS) {
mCamera.addCallbackBuffer(data);
return;
}
if (mState == STATE_PROCESS) {
mProcessInProgress = true;
processDataTask = new ProcessDataTask();
processDataTask.execute(data);
}
}
public void startProcessing() {
mState = STATE_PROCESS;
}
And my AsyncTask is something like this
private class ProcessDataTask
extends
AsyncTask<byte[], Void, Boolean> {
#Override
protected Boolean doInBackground(byte[]... datas) {
mState = STATE_PROCESS_IN_PROGRESS;
Log.i(TAG, "background process started");
byte[] data = datas[0];
long t1 = java.lang.System.currentTimeMillis();
// process your data
long t2 = java.lang.System.currentTimeMillis();
Log.i(TAG, "processing time = " + String.valueOf(t2 - t1));
mCamera.addCallbackBuffer(data);
mProcessInProgress = false;
return true;
}
#Override
protected void onPostExecute(Boolean result) {
mState = STATE_PROCESS_WAIT;
}
}
onPreviewFrame() will always get called as long as the camera preview is running. You need to take the data and process it only when you trigger something. So simply change the state of a variable, in this case, mState, and based on the state, call your AsyncTask.

Android postDelayed works but can't put it in a loop?

Sorry I am a noob I've read countless tutorials about making a simple timer and was wondering why it doesn't work until I noticed it is the while-loop causing the issue o.O I have removed it and then it works but only 1 time I need to use the loop though so the movement finishes :C
Heres the code:
old_x is the coordinate from an ImageView and new_x from the onTouch Event, maybe the problem because I am casting them as an int? I don't know what I need to do so it works please help O:
while(old_x != new_x)
{
timedMoveIV(100);
old_x = (int)img.getX();
}
It calls this method which works if I do it without the loop.
public void timedMoveIV(int time_ms)
{
//sleep for time_ms milliseconds
Handler handler = new Handler();
handler.postDelayed(new Runnable() {
public void run() {
if(new_x > img.getX())
{
img.setX(img.getX() + 1);
}
else
{
img.setX(img.getX() - 1);
}
}
}, time_ms);
}

Your main problem is that your loop doesn't take a break, so it's constantly running the function, posting a gazillion runnables.
What you want to do is make the runnable call itself after another 100 ms. Take a look at this example:
if(old_x != new_x)
timedMoveIV(100);
Here you simply call the function once. After that, you let the posted runnable decide whether or not it needs to move again:
public void timedMoveIV(final int time_ms)
{
Handler handler = new Handler();
handler.postDelayed(new Runnable() {
public void run()
{
if(new_x > img.getX())
img.setX(img.getX() + 1);
else
img.setX(img.getX() - 1);
// if not in position, call again
if((int)img.getX() != new_x)
timedMoveIV(time_ms);
}
}, time_ms);
}
It should stop once img.getX() == new_x. Notice the cast to int, though, because if you leave it out, you might get some oscillation as it gets within a pixel of the destination.
This assumes that new_x is an int. If it's a float as well, you should either cast both to int to compare, or compare them to a minimum threshold. For instance, if there's less than 0.5 difference, treat it as "done".