My application generates heatmap images as fast as the CPU can (around 30-60 per second) and I want to display them in a single "live heatmap". In AWT/Swing, I could just paint them into a JPanel which worked like a charm.
Recently, I switched to JavaFX and want to achieve the same here; at first, I tried with a Canvas, which was slow but okay-ish but had a severe memory leak problem, causing the application to crash. Now, I tried the ImageView component - which apparently is way too slow as the image gets quite laggy (using ImageView.setImage on every new iteration). As far as I understand, setImage does not guarantee that the image is actually displayed when the function finishes.
I am getting the impression that I am on the wrong track, using those components in a manner they are not made to. How can I display my 30-60 Images per second?
EDIT: A very simple test application. You will need the JHeatChart library.
Note that on a desktop machine, I get around 70-80 FPS and the visualization is okay and fluid, but on a smaller raspberry pi (my target machine), I get around 30 FPS but an extremly stucking visualization.
package sample;
import javafx.application.Application;
import javafx.embed.swing.SwingFXUtils;
import javafx.scene.Scene;
import javafx.scene.image.ImageView;
import javafx.scene.layout.VBox;
import javafx.stage.Stage;
import org.tc33.jheatchart.HeatChart;
import java.awt.*;
import java.awt.geom.AffineTransform;
import java.awt.image.AffineTransformOp;
import java.awt.image.BufferedImage;
import java.util.LinkedList;
public class Main extends Application {
ImageView imageView = new ImageView();
final int scale = 15;
#Override
public void start(Stage primaryStage) {
Thread generator = new Thread(() -> {
int col = 0;
LinkedList<Long> fps = new LinkedList<>();
while (true) {
fps.add(System.currentTimeMillis());
double[][] matrix = new double[48][128];
for (int i = 0; i < 48; i++) {
for (int j = 0; j < 128; j++) {
matrix[i][j] = col == j ? Math.random() : 0;
}
}
col = (col + 1) % 128;
HeatChart heatChart = new HeatChart(matrix, 0, 1);
heatChart.setShowXAxisValues(false);
heatChart.setShowYAxisValues(false);
heatChart.setLowValueColour(java.awt.Color.black);
heatChart.setHighValueColour(java.awt.Color.white);
heatChart.setAxisThickness(0);
heatChart.setChartMargin(0);
heatChart.setCellSize(new Dimension(1, 1));
long currentTime = System.currentTimeMillis();
fps.removeIf(elem -> currentTime - elem > 1000);
System.out.println(fps.size());
imageView.setImage(SwingFXUtils.toFXImage((BufferedImage) scale(heatChart.getChartImage(), scale), null));
}
});
VBox box = new VBox();
box.getChildren().add(imageView);
Scene scene = new Scene(box, 1920, 720);
primaryStage.setScene(scene);
primaryStage.show();
generator.start();
}
public static void main(String[] args) {
launch(args);
}
private static Image scale(Image image, int scale) {
BufferedImage res = new BufferedImage(image.getWidth(null) * scale, image.getHeight(null) * scale,
BufferedImage.TYPE_INT_ARGB);
AffineTransform at = new AffineTransform();
at.scale(scale, scale);
AffineTransformOp scaleOp =
new AffineTransformOp(at, AffineTransformOp.TYPE_NEAREST_NEIGHBOR);
return scaleOp.filter((BufferedImage) image, res);
}
}
Your code updates the UI from a background thread, which is definitely not allowed. You need to ensure you update from the FX Application Thread. You also want to try to "throttle" the actual UI updates to occur no more than once per JavaFX frame rendering. The easiest way to do this is with an AnimationTimer, whose handle() method is invoked each time a frame is rendered.
Here's a version of your code which does that:
import java.awt.Dimension;
import java.awt.Image;
import java.awt.geom.AffineTransform;
import java.awt.image.AffineTransformOp;
import java.awt.image.BufferedImage;
import java.util.LinkedList;
import java.util.concurrent.atomic.AtomicReference;
import org.tc33.jheatchart.HeatChart;
import javafx.animation.AnimationTimer;
import javafx.application.Application;
import javafx.embed.swing.SwingFXUtils;
import javafx.scene.Scene;
import javafx.scene.image.ImageView;
import javafx.scene.layout.VBox;
import javafx.stage.Stage;
public class Main extends Application {
ImageView imageView = new ImageView();
final int scale = 15;
#Override
public void start(Stage primaryStage) {
AtomicReference<BufferedImage> image = new AtomicReference<>();
Thread generator = new Thread(() -> {
int col = 0;
LinkedList<Long> fps = new LinkedList<>();
while (true) {
fps.add(System.currentTimeMillis());
double[][] matrix = new double[48][128];
for (int i = 0; i < 48; i++) {
for (int j = 0; j < 128; j++) {
matrix[i][j] = col == j ? Math.random() : 0;
}
}
col = (col + 1) % 128;
HeatChart heatChart = new HeatChart(matrix, 0, 1);
heatChart.setShowXAxisValues(false);
heatChart.setShowYAxisValues(false);
heatChart.setLowValueColour(java.awt.Color.black);
heatChart.setHighValueColour(java.awt.Color.white);
heatChart.setAxisThickness(0);
heatChart.setChartMargin(0);
heatChart.setCellSize(new Dimension(1, 1));
long currentTime = System.currentTimeMillis();
fps.removeIf(elem -> currentTime - elem > 1000);
System.out.println(fps.size());
image.set((BufferedImage) scale(heatChart.getChartImage(), scale));
}
});
VBox box = new VBox();
box.getChildren().add(imageView);
Scene scene = new Scene(box, 1920, 720);
primaryStage.setScene(scene);
primaryStage.show();
generator.setDaemon(true);
generator.start();
AnimationTimer animation = new AnimationTimer() {
#Override
public void handle(long now) {
BufferedImage img = image.getAndSet(null);
if (img != null) {
imageView.setImage(SwingFXUtils.toFXImage(img, null));
}
}
};
animation.start();
}
public static void main(String[] args) {
launch(args);
}
private static Image scale(Image image, int scale) {
BufferedImage res = new BufferedImage(image.getWidth(null) * scale, image.getHeight(null) * scale,
BufferedImage.TYPE_INT_ARGB);
AffineTransform at = new AffineTransform();
at.scale(scale, scale);
AffineTransformOp scaleOp = new AffineTransformOp(at, AffineTransformOp.TYPE_NEAREST_NEIGHBOR);
return scaleOp.filter((BufferedImage) image, res);
}
}
Using the AtomicReference to wrap the buffered image ensures that it is safely shared between the two threads.
On my machine, this generates about 130 images per second; note that not all are displayed, as only the latest one is shown each time the JavaFX graphics framework displays a frame (which is typically throttled at 60fps).
If you want to ensure you show all images that are generated, i.e. you throttle the image generation by the JavaFX framerate, then you can use a BlockingQueue to store the images:
// AtomicReference<BufferedImage> image = new AtomicReference<>();
// Size of the queue is a trade-off between memory consumption
// and smoothness (essentially works as a buffer size)
BlockingQueue<BufferedImage> image = new ArrayBlockingQueue<>(5);
// ...
// image.set((BufferedImage) scale(heatChart.getChartImage(), scale));
try {
image.put((BufferedImage) scale(heatChart.getChartImage(), scale));
} catch (InterruptedException exc) {
Thread.currentThread.interrupt();
}
and
#Override
public void handle(long now) {
BufferedImage img = image.poll();
if (img != null) {
imageView.setImage(SwingFXUtils.toFXImage(img, null));
}
}
The code is pretty inefficient, as you generate a new matrix, new HeatChart, etc, on every iteration. This causes many objects to be created on the heap and quickly discarded, which can cause the GC to be run too often, particularly on a small-memory machine. That said, I ran this with the maximum heap size set at 64MB, (-Xmx64m), and it still performed fine. You may be able to optimize the code, but using the AnimationTimer as shown above, generating images more quickly will not cause any additional stress on the JavaFX framework. I would recommend investigating using the mutability of HeatChart (i.e. setZValues()) to avoid creating too many objects, and/or using PixelBuffer to directly write data to the image view (this would need to be done on the FX Application Thread).
Here's a different example, which (almost) completely minimizes object creation, using one off-screen int[] array to compute data, and one on-screen int[] array to display it. There's a little low-level threading details to ensure the on-screen array is only seen in a consistent state. The on-screen array is used to underly a PixelBuffer, which in turn is used for a WritableImage.
This class generates the image data:
import java.util.concurrent.atomic.AtomicLong;
import java.util.concurrent.locks.ReentrantLock;
import java.util.function.Consumer;
public class ImageGenerator {
private final int width;
private final int height;
// Keep two copies of the data: one which is not exposed
// that we modify on the fly during computation;
// another which we expose publicly.
// The publicly exposed one can be viewed only in a complete
// state if operations on it are synchronized on this object.
private final int[] privateData ;
private final int[] publicData ;
private final long[] frameTimes ;
private int currentFrameIndex ;
private final AtomicLong averageGenerationTime ;
private final ReentrantLock lock ;
private static final double TWO_PI = 2 * Math.PI;
private static final double PI_BY_TWELVE = Math.PI / 12; // 15 degrees
public ImageGenerator(int width, int height) {
super();
this.width = width;
this.height = height;
privateData = new int[width * height];
publicData = new int[width * height];
lock = new ReentrantLock();
this.frameTimes = new long[100];
this.averageGenerationTime = new AtomicLong();
}
public void generateImage(double angle) {
// compute in private data copy:
int minDim = Math.min(width, height);
int minR2 = minDim * minDim / 4;
for (int x = 0; x < width; x++) {
int xOff = x - width / 2;
int xOff2 = xOff * xOff;
for (int y = 0; y < height; y++) {
int index = x + y * width;
int yOff = y - height / 2;
int yOff2 = yOff * yOff;
int r2 = xOff2 + yOff2;
if (r2 > minR2) {
privateData[index] = 0xffffffff; // white
} else {
double theta = Math.atan2(yOff, xOff);
double delta = Math.abs(theta - angle);
if (delta > TWO_PI - PI_BY_TWELVE) {
delta = TWO_PI - delta;
}
if (delta < PI_BY_TWELVE) {
int green = (int) (255 * (1 - delta / PI_BY_TWELVE));
privateData[index] = (0xff << 24) | (green << 8); // green, fading away from center
} else {
privateData[index] = 0xff << 24; // black
}
}
}
}
// copy computed data to public data copy:
lock.lock();
try {
System.arraycopy(privateData, 0, publicData, 0, privateData.length);
} finally {
lock.unlock();
}
frameTimes[currentFrameIndex] = System.nanoTime() ;
int nextIndex = (currentFrameIndex + 1) % frameTimes.length ;
if (frameTimes[nextIndex] > 0) {
averageGenerationTime.set((frameTimes[currentFrameIndex] - frameTimes[nextIndex]) / frameTimes.length);
}
currentFrameIndex = nextIndex ;
}
public void consumeData(Consumer<int[]> consumer) {
lock.lock();
try {
consumer.accept(publicData);
} finally {
lock.unlock();
}
}
public long getAverageGenerationTime() {
return averageGenerationTime.get() ;
}
}
And here's the UI:
import java.nio.IntBuffer;
import javafx.animation.AnimationTimer;
import javafx.application.Application;
import javafx.scene.Scene;
import javafx.scene.control.Label;
import javafx.scene.image.ImageView;
import javafx.scene.image.PixelFormat;
import javafx.scene.image.PixelWriter;
import javafx.scene.image.WritableImage;
import javafx.scene.layout.BorderPane;
import javafx.stage.Stage;
public class AnimationApp extends Application {
private final int size = 400 ;
private IntBuffer buffer ;
#Override
public void start(Stage primaryStage) throws Exception {
// background image data generation:
ImageGenerator generator = new ImageGenerator(size, size);
// Generate new image data as fast as possible:
Thread thread = new Thread(() -> {
while( true ) {
long now = System.currentTimeMillis() ;
double angle = 2 * Math.PI * (now % 10000) / 10000 - Math.PI;
generator.generateImage(angle);
}
});
thread.setDaemon(true);
thread.start();
generator.consumeData(data -> buffer = IntBuffer.wrap(data));
PixelFormat<IntBuffer> format = PixelFormat.getIntArgbPreInstance() ;
PixelBuffer<IntBuffer> pixelBuffer = new PixelBuffer<>(size, size, buffer, format);
WritableImage image = new WritableImage(pixelBuffer);
BorderPane root = new BorderPane(new ImageView(image));
Label fps = new Label("FPS: ");
root.setTop(fps);
Scene scene = new Scene(root);
primaryStage.setScene(scene);
primaryStage.setTitle("Give me a ping, Vasili. ");
primaryStage.show();
AnimationTimer animation = new AnimationTimer() {
#Override
public void handle(long now) {
// Update image, ensuring we only see the underlying
// data in a consistent state:
generator.consumeData(data -> {
pixelBuffer.updateBuffer(pb -> null);
});
long aveGenTime = generator.getAverageGenerationTime() ;
if (aveGenTime > 0) {
double aveFPS = 1.0 / (aveGenTime / 1_000_000_000.0);
fps.setText(String.format("FPS: %.2f", aveFPS));
}
}
};
animation.start();
}
public static void main(String[] args) {
Application.launch(args);
}
}
For a version that doesn't rely on the JavaFX 13 PixelBuffer, you can just modify this class to use a PixelWriter (AIUI this won't be quite as efficient, but works just as smoothly in this example):
// generator.consumeData(data -> buffer = IntBuffer.wrap(data));
PixelFormat<IntBuffer> format = PixelFormat.getIntArgbPreInstance() ;
// PixelBuffer<IntBuffer> pixelBuffer = new PixelBuffer<>(size, size, buffer, format);
// WritableImage image = new WritableImage(pixelBuffer);
WritableImage image = new WritableImage(size, size);
PixelWriter pixelWriter = image.getPixelWriter() ;
and
AnimationTimer animation = new AnimationTimer() {
#Override
public void handle(long now) {
// Update image, ensuring we only see the underlying
// data in a consistent state:
generator.consumeData(data -> {
// pixelBuffer.updateBuffer(pb -> null);
pixelWriter.setPixels(0, 0, size, size, format, data, 0, size);
});
long aveGenTime = generator.getAverageGenerationTime() ;
if (aveGenTime > 0) {
double aveFPS = 1.0 / (aveGenTime / 1_000_000_000.0);
fps.setText(String.format("FPS: %.2f", aveFPS));
}
}
};
Related
I am currently making a side scroller, and I've been working on the moving background. Here is the class that I use to manage it's movement:
package game;
import java.awt.Graphics2D;
import java.awt.Rectangle;
import java.awt.image.BufferedImage;
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import javax.imageio.ImageIO;
import framework.GClock;
public class BackgroundManager {
public BufferedImage background;
public BufferedImage[] movingImages;
private final int y=4,x=2;
public final int scale = 3;
private float[] speed;
private Rectangle[][] rects;
private float speedScale =150f;
public BackgroundManager() {
try {
URL bg = BackgroundManager.class
.getResource("/resources/background.png");
background = ImageIO.read(bg);
URL[] urls = new URL[4];
urls[0] = BackgroundManager.class
.getResource("/resources/mountainBackground.png");
urls[1] = BackgroundManager.class
.getResource("/resources/mountainForeground.png");
urls[2] = BackgroundManager.class
.getResource("/resources/treesBackground.png");
urls[3] = BackgroundManager.class
.getResource("/resources/treesForeground.png");
movingImages = new BufferedImage[4];
for (int i = 0; i < y; i++) {
movingImages[i] = ImageIO.read(urls[i]);
}
int bgw = background.getWidth()*scale;
speed = new float[4];
for (int i = 0; i < y; i++) {
speed[i] = speedScale * (movingImages[i].getWidth()*scale
/ bgw-1);
System.out.println(speed[i]);
// maybe use
// Image.getScaledInstance(scaledWidth,scaledHeight,Image.SCALE_DEFAULT);
}
rects = new Rectangle[x][y];
for (int i = 0; i < y; i++) {
rects[0][i] = new Rectangle(0, 0,
movingImages[i].getWidth()*scale,
movingImages[i].getHeight()*scale);
rects[1][i] = new Rectangle(rects[0][i].width,
0, movingImages[i].getWidth()*scale,
movingImages[i].getHeight()*scale);
}
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
public void update(GClock clock) {
float dt = clock.getElapsedTime();
for (int i = 0; i < y; i++) {
if (rects[0][i].x + rects[0][i].width < -10)
rects[0][i].x = rects[1][i].x + rects[1][i].width;
if (rects[1][i].x + rects[1][i].width < -10)
rects[1][i].x = rects[0][i].x + rects[0][i].width;
}
for (int i = 0; i < y; i++) {
rects[0][i].x -= speed[i] * dt;
rects[1][i].x -= speed[i] * dt;
}
}
public void drawBackground(Graphics2D g) {
//static background
g.drawImage(background, 0, 0, background.getWidth() * scale,
background.getHeight() * scale, null);
//other layers
for (int i = 0; i < y; i++) {
g.drawImage(movingImages[i], rects[0][i].x, 0,
rects[0][i].width,
rects[0][i].height, null);
g.drawImage(movingImages[i], rects[1][i].x, 0,
rects[1][i].width,
rects[1][i].height, null);
}
/*
// background
g.drawImage(background, 0, 0, background.getWidth() * scale,
background.getHeight() * scale, null);
// mountainBackground
g.drawImage(mountainBackground, (int) (-speed[0] * dt), 0,
mountainBackground.getWidth() * scale,
mountainBackground.getHeight() * scale, null);
// mountainForeground
g.drawImage(mountainForeground, (int) (-speed[1] * dt), 0,
mountainForeground.getWidth() * scale,
mountainForeground.getHeight() * scale, null);
// treesBackground
g.drawImage(treesBackground, (int) (-speed[2] * dt), 0,
treesBackground.getWidth() * scale, treesBackground.getHeight()
* scale, null);
// treesForeground
g.drawImage(treesForeground, (int) (-speed[3] * dt), 0,
treesForeground.getWidth() * scale, treesForeground.getHeight()
* scale, null);
*/
}
}
GClock is a class I made to manage time, and clock.getElapsedTime() is the amount of time the last update->draw->repaint->sleep cycle took.
For some reason, instead of moving the first image to the right side of the next image to simulate a continuous background, it does this. In the clip, the next layer speeds up until it reaches the left side of the screen, and then matches the speed of the last layer. I've tried removing the block of code that switches the position of the bounding rectangles, and I've spent about an hour on top of that trying to figure this out, but no luck. What am I doing wrong?
EDIT: Here are the relevant fields for GClock:
private long startTime;
private long dt;
private long lastTime;
private float targetTime;
private boolean isFpsCapped;
public GClock(){
startTime=lastTime=System.currentTimeMillis();
targetTime=1f/60f;
isFpsCapped=true;
}
protected void update(){
this.dt=System.currentTimeMillis()-lastTime;
this.lastTime=System.currentTimeMillis();
}
public float getElapsedTime(){
return (float)dt/1000f;
}
Here's the gameloop:
while(isRunning){
clock.update();//calculates the amount of time that the last game loop took and prepares for the next clock.update() call.
if(!isPaused){
this.update(clock);//update is an abstract method that is implemented into child classes. For this instance, it only contains BackgroundManager.update(clock);
this.render(clock);//render is a method that manages the game's Graphics2D object, and calls this.draw(), which, like update(), is an abstract method called only by implementations of this class.
this.paintScreen();//repains the screen
}
if(clock.isFpsCapped()){
//thread.sleep(millis);
try {
Thread.sleep(clock.calculateSleepTime());//makes the thread sleep for a certain amount of time, so that the amount of time the gameloop took + calculateSleepTime() = some target time (in this case, it tries to keep the game at about 60 fps.)
} catch (InterruptedException e) {
e.printStackTrace();
}
}//end of if statement
}
I'm experimenting with JavaFX and animations, especially PathTransition. I'm creating a simple program that makes a ball "bounce," without using the QuadCurveTo class. Here is my code so far:
Ellipse ball = new Ellipse(375, 250, 10, 10);
root.getChildren().add(ball);
Path path = new Path();
path.getElements().add(new MoveTo(375, 500));
int posX = 375;
int posY = 500;
int changeX = 10;
int changeY = 50;
int gravity = 10; // approximate in m/s^2
int sec = 0;
for(; posY<=500; sec++, posX-=changeX, posY-=changeY, changeY-=gravity)
path.getElements().add(new LineTo(posX, posY));
// How do I equally space these elements?
PathTransition pathTransition = new PathTransition();
pathTransition.setDuration(Duration.millis(sec*1000));
pathTransition.setNode(ball);
pathTransition.setAutoReverse(true);
pathTransition.setCycleCount(Timeline.INDEFINITE);
pathTransition.setInterpolator(Interpolator.LINEAR);
pathTransition.setPath(path);
pathTransition.play();
I have the for loop running through a quadratic sequence, and the ball moves in the correct motion (a curved path).
However, I want it to move slower at the top of the curve (vertex) because it is moving less distance (as changeY, the vertical increment variable, is decreasing as the loop goes on) to simulate a more realistic curve. However, it is traveling in a linear speed throughout the full time.
Is there any way to make each of the elements equally spaced (throughout) time, so that this "bounce" would show correctly? Thanks.
I wouldn't use a timeline or transition at all for this. Use an AnimationTimer and compute the new coordinates based on the elapsed time since the last frame. The AnimationTimer has a handle method which is invoked once per rendering frame, taking a value that represents a timestamp in nanoseconds.
SSCCE (with elasticity added to the physics):
import javafx.animation.AnimationTimer;
import javafx.application.Application;
import javafx.scene.Scene;
import javafx.scene.layout.Pane;
import javafx.scene.paint.Color;
import javafx.scene.shape.Circle;
import javafx.stage.Stage;
public class BouncingBall extends Application {
#Override
public void start(Stage primaryStage) {
Circle ball = new Circle(20, 80, 10);
ball.setFill(Color.DARKBLUE);
Pane pane = new Pane(ball);
AnimationTimer timer = new AnimationTimer() {
long lastUpdate = 0 ;
double changeX = 0.1 ;
double changeY = 0 ;
double gravity = 10 ;
double elasticity = 0.95 ;
#Override
public void handle(long now) {
if (lastUpdate == 0) {
lastUpdate = now ;
return ;
}
long elapsedNanos = now - lastUpdate;
double elapsedSeconds = elapsedNanos / 1_000_000_000.0 ;
lastUpdate = now ;
ball.setCenterX(ball.getCenterX() + changeX);
if (ball.getCenterY() + changeY + ball.getRadius() >= pane.getHeight()) {
changeY = - changeY * elasticity;
} else {
changeY = changeY + gravity * elapsedSeconds ;
}
ball.setCenterY(Math.min(ball.getCenterY() + changeY, pane.getHeight() - ball.getRadius()));
}
};
primaryStage.setScene(new Scene(pane, 400, 400));
primaryStage.show();
timer.start();
}
public static void main(String[] args) {
launch(args);
}
}
I want to create a audio level meter in java for the microphone to check how loud the input is. It should look like the one of the OS. I'm not asking about the gui. It is just about calculating the audio level out of the bytestream produced by
n = targetDataLine.read( tempBuffer , 0 , tempBuffer.length );
So I already have something that is running, but it is not even close to the levelmeter of my OS (windows) It stucks in the middle. I have values between 0 and 100 that is good but in the middle volume it stucks around 60 no matter how loud the input is.
This is how I calculate it now:
amplitude = 0;
for (int j = 0; j < tempBuffer.length; j = j +2 ){
if (tempBuffer[j] > tempBuffer[j+1])
amplitude = amplitude + tempBuffer[j] - tempBuffer[j+1];
else amplitude = amplitude + tempBuffer[j + 1] - tempBuffer[j];
}
amplitude = amplitude / tempBuffer.length * 2;
Is there a better/more precise way to calculate the audio level to monitor it? Or did I maybe do a major mistake?
That is my Audioformat:
public static AudioFormat getAudioFormat(){
float sampleRate = 20000.0F;
//8000,11025,16000,22050,44100
int sampleSizeInBits = 16;
//8,16
int channels = 1;
//1,2
boolean signed = true;
//true,false
boolean bigEndian = false;
//true,false
return new AudioFormat( sampleRate, sampleSizeInBits, channels, signed, bigEndian );
//return new AudioFormat(AudioFormat.Encoding.PCM_SIGNED, 8000.0F, 8, 1, 1, 8000.0F, false);
}
Principally the problem seems to be that you are reading the audio data incorrectly.
Specifically I'm not really sure what this excerpt is supposed to mean:
if (tempBuffer[j] > tempBuffer[j+1])
... tempBuffer[j] - tempBuffer[j+1];
else
... tempBuffer[j + 1] - tempBuffer[j];
But anyhow since you are recording 16-bit data the bytes in the byte array aren't meaningful on their own. Each byte only represents 1/2 of the bits in each sample. You need to 'unpack' them to int, float, whatever, before you can do anything with them. For raw LPCM, concatenating the bytes is done by shifting them and ORing them together.
Here is an MCVE to demonstrate a rudimentary level meter (both RMS and simple peak hold) in Java.
import javax.swing.SwingUtilities;
import javax.swing.JFrame;
import javax.swing.JPanel;
import javax.swing.JComponent;
import java.awt.BorderLayout;
import java.awt.Graphics;
import java.awt.Color;
import java.awt.Dimension;
import javax.swing.border.EmptyBorder;
import javax.sound.sampled.AudioFormat;
import javax.sound.sampled.TargetDataLine;
import javax.sound.sampled.AudioSystem;
import javax.sound.sampled.LineUnavailableException;
public class LevelMeter extends JComponent {
private int meterWidth = 10;
private float amp = 0f;
private float peak = 0f;
public void setAmplitude(float amp) {
this.amp = Math.abs(amp);
repaint();
}
public void setPeak(float peak) {
this.peak = Math.abs(peak);
repaint();
}
public void setMeterWidth(int meterWidth) {
this.meterWidth = meterWidth;
}
#Override
protected void paintComponent(Graphics g) {
int w = Math.min(meterWidth, getWidth());
int h = getHeight();
int x = getWidth() / 2 - w / 2;
int y = 0;
g.setColor(Color.LIGHT_GRAY);
g.fillRect(x, y, w, h);
g.setColor(Color.BLACK);
g.drawRect(x, y, w - 1, h - 1);
int a = Math.round(amp * (h - 2));
g.setColor(Color.GREEN);
g.fillRect(x + 1, y + h - 1 - a, w - 2, a);
int p = Math.round(peak * (h - 2));
g.setColor(Color.RED);
g.drawLine(x + 1, y + h - 1 - p, x + w - 1, y + h - 1 - p);
}
#Override
public Dimension getMinimumSize() {
Dimension min = super.getMinimumSize();
if(min.width < meterWidth)
min.width = meterWidth;
if(min.height < meterWidth)
min.height = meterWidth;
return min;
}
#Override
public Dimension getPreferredSize() {
Dimension pref = super.getPreferredSize();
pref.width = meterWidth;
return pref;
}
#Override
public void setPreferredSize(Dimension pref) {
super.setPreferredSize(pref);
setMeterWidth(pref.width);
}
public static void main(String[] args) {
SwingUtilities.invokeLater(new Runnable() {
#Override
public void run() {
JFrame frame = new JFrame("Meter");
frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
JPanel content = new JPanel(new BorderLayout());
content.setBorder(new EmptyBorder(25, 50, 25, 50));
LevelMeter meter = new LevelMeter();
meter.setPreferredSize(new Dimension(9, 100));
content.add(meter, BorderLayout.CENTER);
frame.setContentPane(content);
frame.pack();
frame.setLocationRelativeTo(null);
frame.setVisible(true);
new Thread(new Recorder(meter)).start();
}
});
}
static class Recorder implements Runnable {
final LevelMeter meter;
Recorder(final LevelMeter meter) {
this.meter = meter;
}
#Override
public void run() {
AudioFormat fmt = new AudioFormat(44100f, 16, 1, true, false);
final int bufferByteSize = 2048;
TargetDataLine line;
try {
line = AudioSystem.getTargetDataLine(fmt);
line.open(fmt, bufferByteSize);
} catch(LineUnavailableException e) {
System.err.println(e);
return;
}
byte[] buf = new byte[bufferByteSize];
float[] samples = new float[bufferByteSize / 2];
float lastPeak = 0f;
line.start();
for(int b; (b = line.read(buf, 0, buf.length)) > -1;) {
// convert bytes to samples here
for(int i = 0, s = 0; i < b;) {
int sample = 0;
sample |= buf[i++] & 0xFF; // (reverse these two lines
sample |= buf[i++] << 8; // if the format is big endian)
// normalize to range of +/-1.0f
samples[s++] = sample / 32768f;
}
float rms = 0f;
float peak = 0f;
for(float sample : samples) {
float abs = Math.abs(sample);
if(abs > peak) {
peak = abs;
}
rms += sample * sample;
}
rms = (float)Math.sqrt(rms / samples.length);
if(lastPeak > peak) {
peak = lastPeak * 0.875f;
}
lastPeak = peak;
setMeterOnEDT(rms, peak);
}
}
void setMeterOnEDT(final float rms, final float peak) {
SwingUtilities.invokeLater(new Runnable() {
#Override
public void run() {
meter.setAmplitude(rms);
meter.setPeak(peak);
}
});
}
}
}
Note the format conversion is hard-coded there.
You may also see "How do I use audio sample data from Java Sound?" for my detailed explanation of how to unpack audio data from the raw bytes.
Related:
How to keep track of audio playback position?
How to make waveform rendering more interesting?
The above code will find the data point with highest value but cannot determine the peak value of the reconstructed data samples. To find the reconstructed peak you would have to pass the data samples through a low pass filter. or use a DFT/FFT algorithm.
There is something in the following code that I am unable to understand. After digging through google for a while, I decided it would be better to ask someone.
I am following a game programming tutorial on youtube, and I feel I understand (to some degree) everything I have written, except for some lines which concern the rendering part of the program.
package com.thomas.game;
import java.awt.Canvas;
import java.awt.Dimension;
import java.awt.Graphics;
import java.awt.image.BufferStrategy;
import java.awt.image.BufferedImage;
import java.awt.image.DataBufferInt;
import javax.swing.JFrame;
import com.thomas.game.graphics.Screen;
import com.thomas.game.input.Keyboard;
public class Game extends Canvas implements Runnable {
private static final int WIDTH = 300;
private static final int HEIGHT = (WIDTH / 16) * 9;
private static final int SCALE = 3;
private static final String TITLE = "Game";
private JFrame frame;
private Thread thread;
private Screen screen;
private BufferedImage image;
private Keyboard key;
private int[] pixels;
private boolean running = false;
private int x = 0, y = 0;
public Game() {
setPreferredSize(new Dimension(WIDTH * SCALE, HEIGHT * SCALE));
screen = new Screen(WIDTH, HEIGHT);
frame = new JFrame();
initializeFrame();
image = new BufferedImage(WIDTH, HEIGHT, BufferedImage.TYPE_INT_RGB);
pixels = ((DataBufferInt)image.getRaster().getDataBuffer()).getData();
this is what I don't understand. I get the image raster, from which I get the databuffer. I typecast that databuffer into a (DatabufferInt), which allows me to retrieve an int[] through the getData() method. After this is done, pixel.length has a value of 48600, and every index contains the int value 0. Operating with this int[] makes the program render like it is supposed to. However, if I don't typecast and retrieve the int[] in the above manner, and instead say pixels = new int[48600], i end up with a black screen.
I guess what I want to know is: what is the difference between these two int[], or rather, what makes the first one work? How does it work?
key = new Keyboard();
addKeyListener(key);
setFocusable(true);
}
public void run() {
long lastTime = System.nanoTime();
double nsPerTick = 1E9/60;
double delta = 0;
long now;
int ticks = 0;
int frames = 0;
long timer = System.currentTimeMillis();
while(running) {
now = System.nanoTime();
delta += (now - lastTime) / nsPerTick;
lastTime = now;
while(delta >= 1) {
tick();
ticks++;
delta--;
}
render();
frames++;
if(System.currentTimeMillis() - timer >= 1000) {
timer += 1000;
frame.setTitle(TITLE + " | ups: " + ticks + " fps: " + frames);
ticks = 0;
frames = 0;
}
}
}
private void render() {
BufferStrategy bs = getBufferStrategy(); // retrieves the bufferstrategy from the current component (the instance of Game that calls this method)
if(bs == null) {
createBufferStrategy(3);
return;
}
screen.clear();
screen.render(x, y);
getPixels();
Graphics g = bs.getDrawGraphics(); // retrieves a graphics object from the next in line buffer in the bufferstrategy, this graphics object draws to that buffer
g.drawImage(image, 0, 0, getWidth(), getHeight(), null); // draws the bufferedimage to the available buffer
g.dispose();
bs.show(); // orders the next in line buffer (which the graphics object g is tied to) to show its contents on the canvas
}
private void tick() {
key.update();
if(key.up)
y--;
if(key.down)
y++;
if(key.left)
x--;
if(key.right)
x++;
}
public void initializeFrame() {
frame.setTitle(TITLE);
frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
frame.setResizable(false);
frame.add(this);
frame.pack();
frame.setLocationRelativeTo(null);
frame.setVisible(true);
}
public synchronized void start() {
running = true;
thread = new Thread(this);
thread.start();
}
public synchronized void stop() {
running = false;
try {
thread.join();
} catch(InterruptedException e) {
e.printStackTrace();
}
}
public static void main(String[] args) {
Game game = new Game();
game.start();
}
public void getPixels() {
for(int i = 0; i < pixels.length; i++)
pixels[i] = screen.pixels[i];
}
}
It seems like the bufferedimage gets values from the pixels array. But I don't understand how these two communicate, or how they are connected. I haven't explicitly told the bufferedimage to get its pixels from the pixels array, so how does it know?
I will also attach the Screen class, which is responsible for updating the pixels array.
package com.thomas.game.graphics;
import java.util.Random;
public class Screen {
private int width, height;
public int[] pixels;
private final int MAP_SIZE = 64;
private final int MAP_SIZE_MASK = MAP_SIZE - 1;
private int[] tiles;
private int tileIndex;
private int xx, yy;
private Random r;
public Screen(int w, int h) {
width = w;
height = h;
pixels = new int[width * height];
tiles = new int[MAP_SIZE * MAP_SIZE];
r = new Random(0xffffff);
for(int i = 0; i < tiles.length; i++) {
tiles[i] = r.nextInt();
}
tiles[0] = 0;
}
public void clear() {
for(int i = 0; i < pixels.length; i++)
pixels[i] = 0;
}
public void render(int xOffset, int yOffset) {
for(int y = 0; y < height; y++) {
yy = y + yOffset;
for(int x = 0; x < width; x++) {
xx = x + xOffset;
tileIndex = (yy >> 4 & MAP_SIZE_MASK) * MAP_SIZE + (xx >> 4 & MAP_SIZE_MASK);
pixels[y * width + x] = tiles[tileIndex];
}
}
}
}
I really hope someone can explain this to me, it would be greatly appreciated. The program is working like it is supposed to, but I don't feel comfortable continuing on the tutorial until I grasp this.
Basic types like short, int, long etc are not Objects.
However, int[] is an array. Arrays are objects in java. Java manipulates objects by reference, not value.
In this line you are not creating a new object. You are storing a reference to the object int[] in your variable pixels. Anything you change in pixels, gets changed inside of the int[] object in image:
pixels = ((DataBufferInt)image.getRaster().getDataBuffer()).getData();
I've created an example, try running this code:
public class Data {
private int[] values = {25,14};
public int[] getValues() {
return values;
}
public static void main(String[] args) {
Data d = new Data();
System.out.println(d.getValues()[0]);
int[] values = d.getValues();
values[0] = 15;
System.out.println(d.getValues()[0]);
}
}
Output:
25
15
Note that you have this code...
pixels = ((DataBufferInt)image.getRaster().getDataBuffer()).getData();
while it should be like this...
pixels = ((DataBufferInt)img.getRaster().getDataBuffer()).getData();
Change image to img.
Hope it works!
I'm writing a multithreaded fractal drawing program with JavaFX 2.2 and now I need some guidance.
What I'm trying to achieve is to create a Task or Service (haven't decided yet) which then fires up some other tasks that actually do the calculation and return sections of the whole image when ready. When all the pieces are returned to the initiating task it puts together the pieces and returns it to the main thread so it can be visualized.
Obviously, all this must happen without ever blocking the UI.
The problem is I can't figure out how these tasks could communicate with each other. For example, I need to update the progress property of the initiating task based on the average progress of the tasks inside it (or something like this), so their progress properties should be bound to the progress property of the initiating task somehow. The image pieces should be put in a list or some container and redrawn on a separate image when all of them are available.
I have already written a simpler (though still experimental) version of this program that creates only one task that calculates the whole fractal. The progress is bound to the progressBar of the GUI. The return value is handled by an EventHandler on success of the task.
I'm not asking for a complete solution but some ideas with maybe a little bit of example code would really help me.
This is the class that should be modified:
package fractal;
import fractalUtil.DefaultPalette;
import fractalUtil.PaletteInterface;
import javafx.concurrent.Task;
import javafx.scene.image.WritableImage;
import javafx.scene.paint.Color;
import org.apache.commons.math3.complex.Complex;
/**
*
* #author athelionas
*/
public abstract class AbstractFractal extends Task implements FractalInterface {
private PaletteInterface palette;
protected final int width, height, order, iterations;
protected final double scale, xReal, xIm, xCenter, yCenter, zoom;
protected final boolean julia;
protected AbstractFractal(final int width, final int height, final double xReal, final double xIm, final double xCenter, final double yCenter, final int order, final boolean julia, final int iterations, final double zoom) {
this.width = width;
this.height = height;
this.xReal = xReal;
this.xIm = xIm;
this.xCenter = xCenter;
this.yCenter = yCenter;
this.order = order;
this.julia = julia;
this.iterations = iterations;
this.zoom = zoom;
this.scale = (double) width / (double) height;
palette = new DefaultPalette();
}
#Override
public final void setPalette(final PaletteInterface palette) {
this.palette = palette;
}
#Override
public abstract Complex formula(final Complex z, final Complex c, final int order, final Complex center);
#Override
public final Color calculatePoint(final Complex z, final Complex c, final int order, final Complex center, final int iterations) {
Complex zTemp = z;
int iter = iterations;
while (zTemp.abs() <= 2.0 && iter > 0) {
zTemp = formula(zTemp, c, order, center);
iter--;
}
if (iter == 0) {
return Color.rgb(0, 0, 0);
} else {
return palette.pickColor((double) (iterations - iter) / (double) iterations);
}
}
#Override
public final WritableImage call() {
Complex z;
Complex c;
Complex center = new Complex(xCenter, yCenter);
final WritableImage image = new WritableImage(width, height);
if (julia) {
c = new Complex(xReal, xIm);
for (int y = 0; y < height; y++) {
for (int x = 0; x < width; x++) {
z = new Complex(((double) x) / (double) (width - 1) * 2.0 * scale * (1.0 / zoom) - scale * (1.0 / zoom), ((double) y) / (double) (height - 1) * 2.0 * (1.0 / zoom) - 1.0 * (1.0 / zoom));
image.getPixelWriter().setColor(x, y, calculatePoint(z, c, order, center, iterations));
}
}
} else {
z = new Complex(xReal, xIm);
for (int y = 0; y < height; y++) {
for (int x = 0; x < width; x++) {
c = new Complex(((double) x) / (double) (width - 1) * 2.0 * scale * (1.0 / zoom) - scale * (1.0 / zoom), ((double) y) / (double) (height - 1) * 2.0 * (1.0 / zoom) - 1.0 * (1.0 / zoom));
image.getPixelWriter().setColor(x, y, calculatePoint(z, c, order, center, iterations));
}
updateProgress(y, height);
}
}
return image;
}
}
Use binding and Task. This way you don't need to care about threading at all. All you need is to create a binding which will normalize each progress according to threads number and summ them up. E.g.
progressBar.progressProperty().bind(
task1.progressProperty().multiply(0.5).add(
task2.progressProperty().multiply(0.5)));
It's a bit trickier for unknown number of threads. See next example:
public class MultiProgressTask extends Application {
private static final int THREADS_NUM = 10;
// this is our Task which produces a Node and track progress
private static class MyTask extends Task<Node> {
private final int delay = new Random().nextInt(1000) + 100;
{ System.out.println("I update progress every " + delay); }
#Override
protected Node call() throws Exception {
updateProgress(0, 5);
for (int i = 0; i < 5; i++) {
System.out.println(i);
Thread.sleep(delay); // imitating activity
updateProgress(i+1, 5);
}
System.out.println("done");
return new Rectangle(20, 20, Color.RED);
}
};
#Override
public void start(Stage primaryStage) {
ProgressBar pb = new ProgressBar(0);
pb.setMinWidth(300);
final VBox root = new VBox();
root.getChildren().add(pb);
Scene scene = new Scene(root, 300, 250);
primaryStage.setScene(scene);
primaryStage.show();
DoubleBinding progress = null;
for (int i = 0; i < THREADS_NUM; i++) {
final MyTask mt = new MyTask();
// here goes binding creation
DoubleBinding scaledProgress = mt.progressProperty().divide(THREADS_NUM);
if (progress == null) {
progress = scaledProgress;
} else {
progress = progress.add(scaledProgress);
}
// here you process the result of MyTask
mt.setOnSucceeded(new EventHandler<WorkerStateEvent>() {
#Override
public void handle(WorkerStateEvent t) {
root.getChildren().add((Node)t.getSource().getValue());
}
});
new Thread(mt).start();
}
pb.progressProperty().bind(progress);
}
public static void main(String[] args) { launch(args); }
}
This is a pretty interesting problem :)
If we remove the issue of thread safety for a moment, you could pass in a double property (or whatever the progress property is bound to) and update that with the progress which would then update the progress indicator. Two problems with that:
Multiple tasks could increment the property at the same time.
The changes must be fired on the javafx thread.
I would wrap the property in it's own class with a simple API:
class ProgressModel {
private final SimpleDoubleProperty progress;
public void increment(finally double increment) {
Platform.runLater(new Runnable() {
progress.set(progress.doubleValue() + increment);
}
}
public void bindPropertyToProgress(DoubleProperty property) {
property.bind(progress);
}
}
In the above code, all updates will run on the javafx thread sequentially so it is thread safe plus no locks. I have done similar background tasks like this and performance has been good (realtime to the user's eyes) although if you're updating thousands of times a second this might not be the case! You will just need to measure. I've not shown the boiler plate code to make it a bit more readable.