Develop Reference

Slow Multithreading in Java - Air Percussion project

I am creating "Air Percussion" using IMU sensors and Arduino to communicate with computer (3 separate IMUs and Arduinos). They are connected to the computer through USBs. I am gathering data on separate Threads (each thread for each sensor). When I connect only one "set" my program is working really fast. I can get even 5 plays of sound per second. Unfortunatelly when i am trying to connect 3 sensors and run them on separate Threads at the same time my program slows down horribly. Even when im moving only one of sensors, I can get like 1 "hit" per second and sometimes it's even losing some of the sounds it should play. I'll show only important parts of the code below.
In the main i've got ActionListener for button, where it should start gathering the data. I run there 3 separate Threads for each USB Port.
connectButton.addActionListener(new ActionListener(){
#Override public void actionPerformed(ActionEvent arg0) {
int dialogButton = 1;
if(!flagaKalibracjiLewa || !flagaKalibracjiPrawa){ //some unimportant flags
dialogButton = JOptionPane.showConfirmDialog(null, "Rozpoczynając program bez kalibracji będziesz miał do dyspozycji mniejszą ilość dzwięków. Czy chcesz kontynuować?","Warning",JOptionPane.YES_NO_OPTION);
}else{
dialogButton = JOptionPane.YES_OPTION;
}
if(dialogButton == JOptionPane.YES_OPTION){
if(connectButton.getText().equals("Connect")) {
if(!flagaKalibracjiLewa && !flagaKalibracjiPrawa) podlaczPorty();
Thread thread = new Thread(){
#Override public void run() {
Scanner data = new Scanner(chosenPort.getInputStream());
dataIncoming(data, "lewa");
data.close();
}
};
Thread thread2 = new Thread(){
#Override public void run() {
Scanner data = new Scanner(chosenPort2.getInputStream());
dataIncoming(data, "prawa");
data.close();
}
};
Thread thread3 = new Thread(){
#Override public void run() {
Scanner data = new Scanner(chosenPort3.getInputStream());
dataIncoming(data, "stopa");
data.close();
}
};
thread.start();
thread2.start();
thread3.start();
connectButton.setText("Disconnect");
} else {
// disconnect from the serial port
chosenPort.closePort();
chosenPort2.closePort();
chosenPort3.closePort();
portList.setEnabled(true);
portList2.setEnabled(true);
portList3.setEnabled(true);
connectButton.setText("Connect");
}
}
}
});
in "dataIncoming" method there is bunch of not important things (like picking, which sound should be played etc.). The important part is in the while loop. In the "while" im gathering next lines of data from sensor. When one of the values is higher than something it should play a sound but only if some time has passed and the sensor has moved a certain way. (when the drumstick is going down the "imuValues[4]" is increasing, when its going up its decreasing, so when its past 160 it means that the player has taken the drumstick up so its ready for the next hit)
while(data.hasNextLine()) {
try{
imuValues = data.nextLine().split(",");
if(Double.parseDouble(imuValues[4])>200 && flagaThreada) {
flagaThreada = false;
playSound(sound1);
}
if(Double.parseDouble(imuValues[4])<160 && System.currentTimeMillis()-startTime>100) {
flagaThreada = true;
startTime=System.currentTimeMillis();
}
}catch(Exception e){
System.out.println("ERROR");
}
}
and finally the method for playing the sound is :
public static synchronized void playSound(String sound) {
try {
String url = "/sounds/"+sound+".wav";
Clip clip = AudioSystem.getClip();
AudioInputStream inputStream = AudioSystem.getAudioInputStream(
Main.class.getResourceAsStream(url));
clip.open(inputStream);
clip.start();
} catch (Exception e) {
System.err.println("ERROR IN OPENING");
}
}
Is my computer to slow to compute and play sounds for 3 sensors at the same time? Or is there a way to create those Threads in a better fashion?

I wrote a version of Clip, called AudioCue, which allows multi-threading on the play commands. It is open source, BSD license (free), consists of three files which you can cut and paste into your program. There is also an API link for it. More info at AudioCue. The site has code examples as well as link to API and source code. There is also some dialogue about its use at Java-gaming.org, under the "Sound" topic thread.
The basic principle behind the code is to make the audio data available in a float array, and send multiple, independent "cursors" through it (one per play command). The setup lets us also do real time volume fading, pitch changes and panning. The audio is output via a SourceDataLine which you can configure (set thread priority, buffer size).
I'm maybe a week or two away from sharing a more advanced version that allows all AudioCues to be mixed through a single output line. This version has five classes/interfaces instead of three, and is being set up for release on github. I'm also hoping to get a donate button and the like set up for this next iteration. The next version might be more useful for Arduino in that I believe you are only allowed up to 8 audio outputs on that system.
Other than that, the steps you have taken (separating the open from the play, using setFramePosition for restarts) are correct. I can't think of anything else to add to help out besides writing your own mixer/cue player (as I have done and am willing to share).

Class Threading Consistency

As part of a project I am currently working on we (a few fellow coding enthusiasts friends) we are trying to make a relatively simple game. The game was to consist of several smaller games that we would program separately and my job was to combine them together into a sorta frankensteined super game.
One of these mini games consists of a "transportation Simulator". Very similar to something like frogger; just a lane swerving minigame. My friend who programmed this has it running 14 swing workers to complete tasks like spawning oncoming cars, road animations and your car. The problem I am having is when I call this class all of the threads don't launch. Some don't activate till after others finish, and others will run behind others. I moved the swingworkers into an Execution Service, but still get similar results. Is there a cap on number of threads running simultaneously? Are swingworkers the wrong thing for the job? It is very confusing for me because I cannot see any consistency in the failed threads. There seems to be no reason why sometimes they launch and other times don't. Any suggestions?
This is the Constructor
travelGUI(Player playerIn) throws InterruptedException {
...
//player is a object that contains a variable for each GUI
//it is also a Parameter
player = playerIn;
player.mapGUI.setVisible(false);
player.mapGUI.dispose();
methodNewThread();//road display
carThread();//Used to update the car in the game
copShootThread();
copShootThread();//Used to set up the bullet shoot thread that shoots at the car
healthThread();
bulletThread();
copThread("1", 1, "1" + "p");
copThread("2", 2, "2" + "p");
copThread("3", 3, "3" + "p");
copThread("4", 4, "4" + "p");
//Creates the background animation for the road and the end conditions for the level
intComponents();
}
an example of one of the threads is
public static void healthThread() {//This thread is used to update the players health
SwingWorker<Void, String> he;
he = new SwingWorker<Void, String>() {
#Override
protected Void doInBackground() throws Exception {
//code to be run in the new thread
...
}
return null;
}
};
// he.execute();
executorService.submit(he);
}

Processing 2 - Will threading speed up rendering many objects?

I'm working on a processing sketch here: https://github.com/davidcool/processing/tree/master/polyhedrons/polyhedrons_4
It's not very elegant code-wise but works fine. It renders rotating complex (meaning many faces) polyhedrons to the screen. Each time you click it adds 5 new rotating polyhedron objects... Once you have 20-25 objects it starts to bog down, meaning the frames/sec drop and it looks jumpy.
I've been reading about threading in Processing/Java. So I started to think maybe I could split the total number of objects out to each processing core. I saw this example in particular: http://www.camnewnham.com/threading-in-processing/
Before I dive into this goose chase, does anyone know if threading would help in terms of animation speed? When I run a sketch normally does it always just use one core for the draw loop? Can threading spread out the object animation rendering over "idle" cores?
Thanks!

The draw() function is always called by the same Thread. This same Thread also calls the mousePressed() and similar functions. In Processing this is called the Animation Thread- Java has a similar idea, called the EDT.
So you can't simply move your drawing to other threads. This will cause problems with the rendering- for example, the Animation Thread might be trying to draw the next frame, while your drawing threads are still trying to draw the previous frame. It's not going to work.
You could try to do your own multi-threaded off-screen buffering by having all your helper threads draw to a PGraphics instead of calling the Processing drawing methods directly. You'd then have to synchronize all of your drawing threads and only draw your PGraphics to the screen (using the Animation Thread) after all of those threads have finished.
This is not a particularly difficult job, but it does involve a pretty decent understanding of how threading works, which goes beyond the scope of most Processing sketches.
Also note that JavaScript doesn't have multiple threads, so anything you do with threading will not work in JavaScript mode.
Here's an example sketch that uses just the Animation Thread to draw 10000 random points every frame. I get about 7 FPS with it:
PGraphics sharedGraphics;
void setup(){
size(500, 500);
sharedGraphics = createGraphics(500, 500);
}
void draw(){
sharedGraphics.beginDraw();
sharedGraphics.background(0);
for(int i = 0; i < 10000; i++){
sharedGraphics.ellipse(random(500), random(500), 5, 5);
}
sharedGraphics.endDraw();
image(sharedGraphics, 0, 0);
println(frameRate);
}
And here is how you might use multiple Threads to render to a PGraphics:
PGraphics sharedGraphics;
void setup() {
size(500, 500);
sharedGraphics = createGraphics(500, 500);
}
void draw() {
ArrayList<Thread> threads = new ArrayList<Thread>();
sharedGraphics.beginDraw();
sharedGraphics.background(0);
for (int i = 0; i < 100; i++) {
Thread t = new DrawThread();
threads.add(t);
t.start();
}
for (Thread t : threads) {
try {
t.join();
}
catch(InterruptedException e) {
e.printStackTrace();
}
}
sharedGraphics.endDraw();
image(sharedGraphics, 0, 0);
println(frameRate);
}
class DrawThread extends Thread {
public void run() {
for (int i = 0; i < 100; i++) {
sharedGraphics.ellipse(random(500), random(500), 5, 5);
}
}
}
However, the performance of that is even worse than the single-threaded model, and I get strange artifacts (some of the ellipses are filled, others are not) which suggests that PGraphics is not thread-safe. That might depend on the type of renderer you're using. You might then add some synchronization:
class DrawThread extends Thread {
public void run() {
for (int i = 0; i < 100; i++) {
synchronized(sharedGraphics) {
sharedGraphics.ellipse(random(500), random(500), 5, 5);
}
}
}
}
That works, but the performance of it is even worse, since you're still only accessing the PGraphics with one thread at a time, and you're doing a bunch of extra work every time draw() is called.
You might be able to fiddle around with it to make it work, but the end result is that it's probably not worth it.
"Some people, when confronted with a problem, think “I know, I'll use multithreading”. Nothhw tpe yawrve o oblems."

Windows hangs during (or after) execution of Java code generating many images

I am developing a code which sometimes has to generate a lot of images. The program works perfectly fine when I have relatively small number of images to generate, however when I have to generate tens of thousands of images something strange could happen.
At some random point Windows appears to hung during execution of the code for up to few minutes. Task manager claims that Java application uses 0% of processor at that time. In fact every application that tries to use a resource from hard drive hangs, but applications that are already opened and don't require access to the hard drive seems to work.
What is more strange this behavior could happen even few seconds/minutes after my program is finished. But sometimes it doesn't happen at all.
Here is the simplified example:
public static void main(String[] args) {
try {
for (int i = 1; i <= 100; i++) {
String dirName = "tmp/" + i + "/";
File dir = new File(dirName);
dir.mkdirs();
//put only 200 file into one directory
for (int j = 0; j < 200; j++) {
drawImage(dirName, j);
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
private static void drawImage(String dirName, int j)
throws FileNotFoundException, IOException {
BufferedImage bi = new BufferedImage(512, 512, BufferedImage.TYPE_INT_ARGB);
Graphics2D graphics = bi.createGraphics();
//draw something on the image
for (int k = 0; k < 10; k++)
graphics.drawLine(k, 0, k*2, 512);
BufferedImage tmpBI=new BufferedImage(512, 512, BufferedImage.TYPE_INT_ARGB);
Graphics2D tmpGraphics = tmpBI.createGraphics();
tmpGraphics.drawImage(bi, 0, 0, 512, 512, 0, 0, 512, 512, null);
//write image to png
FileOutputStream fos;
fos = new FileOutputStream(new File(dirName + "img" + j + ".PNG"));
ImageIO.write(tmpBI, "PNG", fos);
fos.close();
}
My first guess is that there are some problems with file handlers in OS or that my Java code improperly handles files.
The second guess is that garbage collector does some magic things that I don't understand.
But to be honest I have no idea how to find out what the real problem is and how to fix it.
I run the code on Windows 7 64bit and jdk1.7 64-bit with NTFS file system.
UPDATE
Few responses proposed some workarounds. I tested all of them with the same effect:
change the output directory to USB memory stick
additional thread for computations
single ZIP file as the output stream for all files
The last try suprised me. I expected that in this case it wouldn't hang. So, I performed another test and instead of writing to file I used NullOutputStream. The result was the same...
My conclusion: either there is something wrong in swing library (very unprobable) or maybe there is something wrong with my computer/OS. I will check it on other computers/OS. If the problem persists I will get back to it.

The code you write access some kernell functions.
The open and close functions for OutputStream are partially done in native, so is createGraphics().
When accessing native functions, keep in mind that they will have their own synchronization implemented.
I believe you are in the scenario where the quantity of information sent to the native thread comes at a much higher pace then the native thread writes to disk (when you think about it, RAM memory access is way quicker then HDD memory).
Even if your Java machine quits, the "tasks" were already given to the native thread to write.
That's probably why all HDD operations are getting hanged.
Try adding a Thread Sleep to your thread in order to put the Java thread in a lower priority and see if this helps.
As a general rule of thumb, file creation is a very expensive operation.20000 files is hard to even be listed in a folder :).

My feeling is that you should invoke the drawings on the UI thread, because you make use of awt.Graphics and BufferedImage. These are UI specific components and you always want to make UI operations on the UI thread.
Try somethig like:
EventQueue.invokeLater(new Runnable() {
public void run() {
try {
// do your graphics related operations here
} catch (Exception e) {
e.printStackTrace();
}
}
});

Try going multithreaded, so you don't use up your main thread. I guess this code is originally ran in a swing form other than console, if you do massive operations on the main thread the program won't redraw until the operation is over.
public static void main(String[] args) {
Thread operation = new Thread(new Runnable(ThreadOperation));
operation.start();
}
public static void ThreadOperation() {
try {
for (int i = 1; i <= 100; i++) {
String dirName = "tmp/" + i + "/";
File dir = new File(dirName);
dir.mkdirs();
//put only 200 file into one directory
for (int j = 0; j < 200; j++) {
drawImage(dirName, j);
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
Also you may track the progress with a taskbar with a little more coding.
Now, you should really expect other programs to hang while your program saves a massive amount of data into the disk, since hard disks aren't as fast as your processor can be. Unless (or even if) you have an SSD HD you will have this issue.
Other programs hanging while your process writes your data, only means they're trying to write data into the disk using their main thread and won't redraw until they finish writing their data, then windows thinks: Wait this guy isn't redrawing... must be dead.
PS: This code might be wrong somewhere, i've been coding way too much in c# lately.

Increasing screen capture speed when using Java and awt.Robot

Edit: If anyone also has any other recommendations for increasing performance of screen capture please feel free to share as it might fully address my problem!
Hello Fellow Developers,
I'm working on some basic screen capture software for myself. As of right now I've got some proof of concept/tinkering code that uses java.awt.Robot to capture the screen as a BufferedImage. Then I do this capture for a specified amount of time and afterwards dump all of the pictures to disk. From my tests I'm getting about 17 frames per second.
Trial #1
Length: 15 seconds
Images Captured: 255
Trial #2
Length: 15 seconds
Images Captured: 229
Obviously this isn't nearly good enough for a real screen capture application. Especially since these capture were me just selecting some text in my IDE and nothing that was graphically intensive.
I have two classes right now a Main class and a "Monitor" class. The Monitor class contains the method for capturing the screen. My Main class has a loop based on time that calls the Monitor class and stores the BufferedImage it returns into an ArrayList of BufferedImages.
If I modify my main class to spawn several threads that each execute that loop and also collect information about the system time of when the image was captured could I increase performance? My idea is to use a shared data structure that will automatically sort the frames based on capture time as I insert them, instead of a single loop that inserts successive images into an arraylist.
Code:
Monitor
public class Monitor {
/**
* Returns a BufferedImage
* #return
*/
public BufferedImage captureScreen() {
Rectangle screenRect = new Rectangle(Toolkit.getDefaultToolkit().getScreenSize());
BufferedImage capture = null;
try {
capture = new Robot().createScreenCapture(screenRect);
} catch (AWTException e) {
e.printStackTrace();
}
return capture;
}
}
Main
public class Main {
public static void main(String[] args) throws InterruptedException {
String outputLocation = "C:\\Users\\ewillis\\Pictures\\screenstreamer\\";
String namingScheme = "image";
String mediaFormat = "jpeg";
DiscreteOutput output = DiscreteOutputFactory.createOutputObject(outputLocation, namingScheme, mediaFormat);
ArrayList<BufferedImage> images = new ArrayList<BufferedImage>();
Monitor m1 = new Monitor();
long startTimeMillis = System.currentTimeMillis();
long recordTimeMillis = 15000;
while( (System.currentTimeMillis() - startTimeMillis) <= recordTimeMillis ) {
images.add( m1.captureScreen() );
}
output.saveImages(images);
}
}

Re-using the screen rectangle and robot class instances will save you a little overhead. The real bottleneck is storing all your BufferedImage's into an array list.
I would first benchmark how fast your robot.createScreenCapture(screenRect); call is without any IO (no saving or storing the buffered image). This will give you an ideal throughput for the robot class.
long frameCount = 0;
while( (System.currentTimeMillis() - startTimeMillis) <= recordTimeMillis ) {
image = m1.captureScreen();
if(image !== null) {
frameCount++;
}
try {
Thread.yield();
} catch (Exception ex) {
}
}
If it turns out that captureScreen can reach the FPS you want there is no need to multi-thread robot instances.
Rather than having an array list of buffered images I'd have an array list of Futures from the AsynchronousFileChannel.write.
Capture loop
Get BufferedImage
Convert BufferedImage to byte array containing JPEG data
Create an async channel to the output file
Start a write and add the immediate return value (the future) to your ArrayList
Wait loop
Go through your ArrayList of Futures and make sure they all finished

I guess that the intensive memory usage is an issue here. You are capturing in your tests about 250 screenshots. Depending on the screen resolution, this is:
1280x800 : 250 * 1280*800 * 3/1024/1024 == 732 MB data
1920x1080: 250 * 1920*1080 * 3/1024/1024 == 1483 MB data
Try caputuring without keeping all those images in memory.
As #Obicere said, it is a good idea to keep the Robot instance alive.