Develop Reference

Fast Forward implementation in Realtime Audio byte array in java

I am managing audio capturing and playing using java sound API (targetDataLine and sourceDataLine). Now suppose in a conference environment, one participant's audio queue size got greater than jitter size (due to processing or network) and I want to fast forward the audio bytes I have of that participant to make it shorter than jitter size.
How can I fast forward the audio byte array of that participant?
I can't do it during playing as normally Player thread just deque 1 frame from every participant's queue and mix it for playing. The only way I can get that is if I deque more than 1 frame of that participant and mix(?) it for fast-forwarding before mixing it with other participants 1 dequeued frame for playing?
Thanks in advance for any kind of help or advice.

There are two ways to speed up the playback that I know of. In one case, the faster pace creates a rise in pitch. The coding for this is relatively easy. In the other case, pitch is kept constant, but it involves a technique of working with sound granules (granular synthesis), and is harder to explain.
For the situation where maintaining the same pitch is not a concern, the basic plan is as follows: instead of advancing by single frames, advance by a frame + a small increment. For example, let's say that advancing 1.1 frames over a course of 44000 frames is sufficient to catch you up. (That would also mean that the pitch increase would be about 1/10 of an octave.)
To advance a "fractional" frame, you first have to convert the bytes of the two bracketing frames to PCM. Then, use linear interpolation to get the intermediate value. Then convert that intermediate value back to bytes for the output line.
For example, if you are advancing from frame[0] to frame["1.1"] you will need to know the PCM for frame[1] and frame[2]. The intermediate value can be calculated using a weighted average:
value = PCM[1] * 9/10 + PCM[2] * 1/10
I think it might be good to make the amount by which you advance change gradually. Take a few dozen frames to ramp up the increment and allow time to ramp down again when returning to normal dequeuing. If you suddenly change the rate at which you are reading the audio data, it is possible to introduce a discontinuity that will be heard as a click.
I have used this basic plan for dynamic control of playback speed, but I haven't had the experience of employing it for the situation that you are describing. Regulating the variable speed could be tricky if you also are trying to enforce keeping the transitions smooth.
The basic idea for using granules involves obtaining contiguous PCM (I'm not clear what the optimum number of frames would be for voice, 1 to 50 millis is cited as commonly being used with this technique in synthesis), and giving it a volume envelope that allows you to mix sequential granules end-to-end (they must overlap).
I think the envelopes for the granules make use of a Hann function or Hamming window--but I'm not clear on the details, such as the overlapping placement of the granules so that they mix/transition smoothly. I've only dabbled, and I'm going to assume folks at Signal Processing will be the best bet for advice on how to code this.

I found a fantastic git repo (sonic library, mainly for audio player) which actually does exactly what I wanted with so much controls. I can input a whole .wav file or even chunks of audio byte arrays and after processing, we can get speed up play experience and so more. For real time processing I actually called this on every chunk of audio byte array.
I found another way/algo to detect whether a audio chunk/byte array is voice or not and after depending on it's result, I can simply ignore playing non voice packets which gives us around 1.5x speedup with less processing.
public class DTHVAD {
public static final int INITIAL_EMIN = 100;
public static final double INITIAL_DELTAJ = 1.0001;
private static boolean isFirstFrame;
private static double Emax;
private static double Emin;
private static int inactiveFrameCounter;
private static double Lamda; //
private static double DeltaJ;
static {
initDTH();
}
private static void initDTH() {
Emax = 0;
Emin = 0;
isFirstFrame = true;
Lamda = 0.950; // range is 0.950---0.999
DeltaJ = 1.0001;
}
public static boolean isAllSilence(short[] samples, int length) {
boolean r = true;
for (int l = 0; l < length; l += 80) {
if (!isSilence(samples, l, l+80)) {
r = false;
break;
}
}
return r;
}
public static boolean isSilence(short[] samples, int offset, int length) {
boolean isSilenceR = false;
long energy = energyRMSE(samples, offset, length);
// printf("en=%ld\n",energy);
if (isFirstFrame) {
Emax = energy;
Emin = INITIAL_EMIN;
isFirstFrame = false;
}
if (energy > Emax) {
Emax = energy;
}
if (energy < Emin) {
if ((int) energy == 0) {
Emin = INITIAL_EMIN;
} else {
Emin = energy;
}
DeltaJ = INITIAL_DELTAJ; // Resetting DeltaJ with initial value
} else {
DeltaJ = DeltaJ * 1.0001;
}
long thresshold = (long) ((1 - Lamda) * Emax + Lamda * Emin);
// printf("e=%ld,Emin=%f, Emax=%f, thres=%ld\n",energy,Emin,Emax,thresshold);
Lamda = (Emax - Emin) / Emax;
if (energy > thresshold) {
isSilenceR = false; // voice marking
} else {
isSilenceR = true; // noise marking
}
Emin = Emin * DeltaJ;
return isSilenceR;
}
private static long energyRMSE(short[] samples, int offset, int length) {
double cEnergy = 0;
float reversOfN = (float) 1 / length;
long step = 0;
for (int i = offset; i < length; i++) {
step = samples[i] * samples[i]; // x*x/N=
// printf("step=%ld cEng=%ld\n",step,cEnergy);
cEnergy += (long) ((float) step * reversOfN);// for length =80
// reverseOfN=0.0125
}
cEnergy = Math.pow(cEnergy, 0.5);
return (long) cEnergy;
}
}
Here I can convert my byte array to short array and detect whether it is voice or non voice by
frame.silence = DTHVAD.isSilence(encodeShortBuffer, 0, shortLen);

Android AudioTrack getPlaybackHeadPosition always returns 0

I am writing some code that intends to take a Wave file, and write it out to and AudioTrack in mode stream. This is a minimum viable test to get AudioTrack stream mode working.
But once I write some buffer of audio to the AudioTrack, and subsequently call play(), the method getPlaybackHeadPosition() continually returns 0.
EDIT: If I ignore my available frames check, and just continually write buffers to the AudioTrack, the write method returns 0 (after the the first buffer write), indicating that it simply did not write any more audio. So it seems that the AudioTrack just doesn't want to start playing.
My code is properly priming the audiotrack. The play method is not throwing any exceptions, so I am not sure what is going wrong.
When stepping through the code, everything on my end is exactly how I anticipate it, so I am thinking somehow I have the AudioTrack configured wrong.
I am running on an emulator, but I don't think that should be an issue.
The WavFile class I am using is a vetted class that I have up and running reliably in lots of Java projects, it is tested to work well.
Observe the following log write, which is a snippet from the larger chunk of code. This log write is never hitting...
if (headPosition > 0)
Log.e("headPosition is greater than zero!!");
..
public static void writeToAudioTrackStream(final WavFile wave)
{
Log.e("writeToAudioTrackStream");
Thread thread = new Thread()
{
public void run()
{
try {
final float[] data = wave.getData();
int format = -1;
if (wave.getChannel() == 1)
format = AudioFormat.CHANNEL_OUT_MONO;
else if (wave.getChannel() == 2)
format = AudioFormat.CHANNEL_OUT_STEREO;
else
throw new RuntimeException("writeToAudioTrackStatic() - unsupported number of channels value = "+wave.getChannel());
final int bufferSizeInFrames = 2048;
final int bytesPerSmp = wave.getBytesPerSmp();
final int bufferSizeInBytes = bufferSizeInFrames * bytesPerSmp * wave.getChannel();
AudioTrack audioTrack = new AudioTrack(AudioManager.STREAM_MUSIC, wave.getSmpRate(),
format,
AudioFormat.ENCODING_PCM_FLOAT,
bufferSizeInBytes,
AudioTrack.MODE_STREAM);
int index = 0;
float[] buffer = new float[bufferSizeInFrames * wave.getChannel()];
boolean started = false;
int framesWritten = 0;
while (index < data.length) {
// calculate the available space in the buffer
int headPosition = audioTrack.getPlaybackHeadPosition();
if (headPosition > 0)
Log.e("headPosition is greater than zero!!");
int framesInBuffer = framesWritten - headPosition;
int availableFrames = bufferSizeInFrames - framesInBuffer;
// once the buffer has no space, the prime is done, so start playing
if (availableFrames == 0) {
if (!started) {
audioTrack.play();
started = true;
}
continue;
}
int endOffset = availableFrames * wave.getChannel();
for (int i = 0; i < endOffset; i++)
buffer[i] = data[index + i];
int samplesWritten = audioTrack.write(buffer , 0 , endOffset , AudioTrack.WRITE_BLOCKING);
// could return error values
if (samplesWritten < 0)
throw new RuntimeException("AudioTrack write error.");
framesWritten += samplesWritten / wave.getChannel();
index = endOffset;
}
}
catch (Exception e) {
Log.e(e.toString());
}
}
};
thread.start();
}

Per the documentation,
For portability, an application should prime the data path to the maximum allowed by writing data until the write() method returns a short transfer count. This allows play() to start immediately, and reduces the chance of underrun.
With a strict reading, this might be seen to contradict the earlier statement:
...you can optionally prime the data path prior to calling play(), by writing up to bufferSizeInBytes...
(emphasis mine), but the intent is clear enough: You're supposed to get a short write first.
This is just to get play started. Once that takes place, you can, in fact, use
getPlaybackHeadPosition() to determine when more space is available. I've used that technique successfully in my own code, on many different devices/API levels.
As an aside: You should be prepared for getPlaybackHeadPosition() to change only in large increments (if I remember correctly, it's getMinBufferSize()/2). This is the max resolution available from the system; onMarkerReached() cannot be used to do any better.

Compare graph values or structure

I have an android application which is getting gesture coordinates (3 axis - x,y,z). I need to compare them with coordinates which I have in my DB and determine whether they are the same or not.
I also need to add some tolerance, since accelerometer (device which captures gestures) is very sensitive. It would be easy, but I also want to consider e.g. "big circle" drawn in the air, same as "small circle" drawn in the air. meaning that there would be different values, but structure of the graph would be the same, right?
I have heard about translating graph values into bits and then compare. Is that the right approach? Is there any library for such comparison?
So far I just hard coded it, covering all my requirements except the last one (big circle vs small circle).
My code now:
private int checkWhetherGestureMatches(byte[] values, String[] refValues) throws IOException {
int valuesSize = 32;
int ignorePositions = 4;
byte[] valuesX = new byte[valuesSize];
byte[] valuesY = new byte[valuesSize];
byte[] valuesZ = new byte[valuesSize];
for (int i = 0; i < valuesSize; i++) {
int position = i * 3 + ignorePositions;
valuesX[i] = values[position];
valuesY[i] = values[position + 1];
valuesZ[i] = values[position + 2];
}
Double[] valuesXprevious = new Double[valuesSize];
Double[] valuesYprevious = new Double[valuesSize];
Double[] valuesZprevious = new Double[valuesSize];
for (int i = 0; i < valuesSize; i++) {
int position = i * 3 + ignorePositions;
valuesXprevious[i] = Double.parseDouble(refValues[position]);
valuesYprevious[i] = Double.parseDouble(refValues[position + 1]);
valuesZprevious[i] = Double.parseDouble(refValues[position + 2]);
}
int incorrectPoints = 0;
for (int j = 0; j < valuesSize; j++) {
if (valuesX[j] < valuesXprevious[j] + 20 && valuesX[j] > valuesXprevious[j] - 20
&& valuesY[j] < valuesYprevious[j] + 20 && valuesY[j] > valuesYprevious[j] - 20
&& valuesZ[j] < valuesZprevious[j] + 20 && valuesZ[j] > valuesZprevious[j] - 20) {
} else {
incorrectPoints++;
}
}
return incorrectPoints;
}
EDIT:
I found JGraphT, it might work. If you know anything about that already, let me know.
EDIT2:
See these images, they are the same gesture but one is done in a slower motion than another.
Faster one:
Slower one:
I haven't captured images of the same gesture where one would be smaller than another, might add that later.

If your list of gestures is complex, I would suggest training a neural network which can classify the gestures based on the graph value bits you mentioned. The task is very similar to classification of handwritten numerical digits, for which lots of resources are there on the net.
The other approach would be to mathematically guess the shape of the gesture, but I doubt it will be useful considering the tolerance of the accelerometer and the fact that users won't draw accurate shapes.

(a) convert your 3D coordinates into 2D plain figure. Use matrix transformations.
(b) normalize your gesture scale - again with matrix transformations
(c) normalize the number of points or use interpolation on the next step.
(d) calculate the difference between your stored (s) gesture and current (c) gesture as
Sum((Xs[i] - Xc[i])^2 + (Ys[i] - Yc[i])^2) where i = 0 .. num of points
If the difference is below your predefined precision - gestures are equal.

I have used a Java implementation of Dynamic Time Wrapping algorithm. The library is called fastDTW.
Unfortunately from what I undersood they don't support it anymore, though I found a use for it.
https://code.google.com/p/fastdtw/
I can't recall now, but I think I used this one and compiled it myself:
https://github.com/cscotta/fastdtw/tree/master/src/main/java/com/fastdtw/dtw

How do apps measure CPU usage (as a %)?

So I'm trying to write an app that measures CPU usage (ie, the time CPU is working vs the time it isn't). I've done some research, but unfortunately there are a bunch of different opinions on how it should be done.
These different solutions include, but aren't limited to:
Get Memory Usage in Android
and
http://juliano.info/en/Blog:Memory_Leak/Understanding_the_Linux_load_average
I've tried writing some code myself, that I though might do the trick, because the links above don't take into consideration when the core is off (or do they?)
long[][] cpuUseVal = {{2147483647, 0} , {2147483647, 0} , {2147483647, 0} ,
{2147483647, 0} , {2147483647, 0}};
public float[] readCPUUsage(int coreNum) {
int j=1;
String[] entries; //Array to hold entries in the /proc/stat file
int cpu_work;
float percents[] = new float[5];
Calendar c = Calendar.getInstance();
// Write the dataPackage
long currentTime = c.getTime().getTime();
for (int i = 0; i <= coreNum; i++){
try {
//Point the app to the file where CPU values are located
RandomAccessFile reader = new RandomAccessFile("/proc/stat", "r");
String load = reader.readLine();
while (j <= i){
load = reader.readLine();
j++;
}
//Reset j for use later in the loop
j=1;
entries = load.split("[ ]+");
//Pull the CPU working time from the file
cpu_work = Integer.parseInt(entries[1]) + Integer.parseInt(entries[2]) + Integer.parseInt(entries[3])
+ Integer.parseInt(entries[6]) + Integer.parseInt(entries[6]) + Integer.parseInt(entries[7]);
reader.close();
percents[i] = (float)(cpu_work - cpuUseVal[i][1]) / (currentTime - cpuUseVal[i][0]);
cpuUseVal[i][0] = currentTime;
cpuUseVal[i][1] = cpu_work;
//In case of an error, print a stack trace
} catch (IOException ex) {
ex.printStackTrace();
}
}
//Return the array holding the usage values for the CPU, and all cores
return percents;
}
So here is the idea of the code I wrote...I have a global array with some dummy values that should return negative percentages the first time the function is run. The values are being stored in a database, so I would know to disregard anything negative. Anyway, the function runs, getting values of time the cpu is doing certain things, and comparing it to the last time the function is run (with the help of the global array). These values are divided by the amount of time that has passed between the function runs (with the help of the calendar)
I've downloaded some of the existing cpu usage monitors and compared them to values I get from my app, and mine are never even close to what they get. Can someone explain what I'm doing wrong?
Thanks to some help I have changed my function to look like the following, hope this helps others who have this question
// Function to read values from /proc/stat and do computations to compute CPU %
public float[] readCPUUsage(int coreNum) {
int j = 1;
String[] entries;
int cpu_total;
int cpu_work;
float percents[] = new float[5];
for (int i = 0; i <= coreNum; i++) {
try {
// Point the app to the file where CPU values are located
RandomAccessFile reader = new RandomAccessFile("/proc/stat","r");
String load = reader.readLine();
// Loop to read down to the line that corresponds to the core
// whose values we are trying to read
while (j <= i) {
load = reader.readLine();
j++;
}
// Reset j for use later in the loop
j = 1;
// Break the line into separate array elements. The end of each
// element is determined by any number of spaces
entries = load.split("[ ]+");
// Pull the CPU total time on and "working time" from the file
cpu_total = Integer.parseInt(entries[1])
+ Integer.parseInt(entries[2])
+ Integer.parseInt(entries[3])
+ Integer.parseInt(entries[4])
+ Integer.parseInt(entries[5])
+ Integer.parseInt(entries[6])
+ Integer.parseInt(entries[7]);
cpu_work = Integer.parseInt(entries[1])
+ Integer.parseInt(entries[2])
+ Integer.parseInt(entries[3])
+ Integer.parseInt(entries[6])
+ Integer.parseInt(entries[7]);
reader.close();
//If it was off the whole time, say 0
if ((cpu_total - cpuUseVal[i][0]) == 0)
percents[i] = 0;
//If it was on for any amount of time, compute the %
else
percents[i] = (float) (cpu_work - cpuUseVal[i][1])
/ (cpu_total - cpuUseVal[i][0]);
//Save the values measured for future comparison
cpuUseVal[i][0] = cpu_total;
cpuUseVal[i][1] = cpu_work;
// In case of an error, print a stack trace
} catch (IOException ex) {
ex.printStackTrace();
}
}
// Return the array holding the usage values for the CPU, and all cores
return percents;
}

Apps don't measure CPU usage, the kernel does by interrupting the process 100 times per second (or some other frequency depending on how the kernel is tuned) and incrementing a counter which corresponds to what it was doing when interrupted.
If in the process => increment the user counter.
If in the kernel => increment the system counter
If waiting for disk or network or a device => increment the waiting for IO
Otherwise increment the idle counter.
The uptime is determined by the decaying average length of the run queue i.e. how many threads are waiting to run. The first number is the average length over the last minute. You can get the load average via JMX.

Audio - Byte to Int and back in java

I'm playing around with looping a sound snippet at succesively faster speeds, for the fun of it, and have stumbled upon this question, which solves it nicely, I guess. It does get kind of buggy when you go to high speeds, because it drops out anything in between and just takes one byte every so often. So I wanted to change it to take the average of all bytes in a stretch within the array. Problem is, bytes don't lend themselves to be divided by ints, and I'm a bit stupid when it comes to changing from bytes to ints. My solution was to do this (again complementing from the mentioned question.
import javax.swing.JOptionPane;
import javax.swing.JFileChooser;
import javax.sound.sampled.*;
import java.net.URL;
import java.io.ByteArrayOutputStream;
import java.io.ByteArrayInputStream;
import java.util.Date;
import java.io.File;
class AcceleratePlayback {
public static void main(String[] args) throws Exception {
int playBackSpeed = 3;
File soundFile;
if (args.length>0) {
try {
playBackSpeed = Integer.parseInt(args[0]);
} catch (Exception e) {
e.printStackTrace();
System.exit(1);
}
}
System.out.println("Playback Rate: " + playBackSpeed);
JFileChooser chooser = new JFileChooser();
chooser.showOpenDialog(null);
soundFile = chooser.getSelectedFile();
System.out.println("FILE: " + soundFile);
AudioInputStream ais = AudioSystem.getAudioInputStream(soundFile);
AudioFormat af = ais.getFormat();
int frameSize = af.getFrameSize();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] b = new byte[2^16];
int read = 1;
while( read>-1 ) {
read = ais.read(b);
if (read>0) {
baos.write(b, 0, read);
}
}
System.out.println("End entire: \t" + new Date());
//This is the important bit
byte[] b1 = baos.toByteArray();
byte[] b2 = new byte[b1.length/playBackSpeed];
for (int ii=0; ii<b2.length/frameSize; ii++) {
for (int jj=0; jj<frameSize; jj++) {
int b3=0;
for (int kk = 0; kk < playBackSpeed; kk++){
b3 = b3+(int)b1[(ii*frameSize*playBackSpeed)+jj+kk];
}
b3 = b3/playBackSpeed;
b2[(ii*frameSize)+jj] = (byte)b3;
}
}
//ends here
System.out.println("End sub-sample: \t" + new Date());
ByteArrayInputStream bais = new ByteArrayInputStream(b2);
AudioInputStream aisAccelerated = new AudioInputStream(bais, af, b2.length);
Clip clip = AudioSystem.getClip();
clip.open(aisAccelerated);
clip.loop(2*playBackSpeed);
clip.start();
JOptionPane.showMessageDialog(null, "Exit?");
}
}
I do realize this is probably the wrong way to do it, but I'm not sure what else I could do, any thoughts??
Best, Alex.

Since my earlier "solution" was cited, I'll lay out what I use for variable speed playback in a more detail. I confess, I don't totally understand the approach used in this question, and so am not going to make an attempt to improve upon the code. I'm risking "not answering the question" in doing this, but perhaps the increased detail about using linear interpolation will show that this can be a sufficient way to make the higher-speed loops you are going for.
I'm NOT claiming the approach that I came up with is the best. I'm not a sound engineer. But it seems to work. (Am always grateful for any suggested improvements.)
This is for a sound library I've made for my own games. It is based on the idea of a Java Clip, but with some extra capabilities. In my library, there's a place to store data, and another couple structures for playback, one for concurrent single plays, and another for looping. Both allow vari-speed, even to the extent of playing the sound backwards.
To load and hold the "clip" data, I just use a single int[], called 'clipData', but I use it for both L & R, so odd & even ints are for either ear.
Loading 'clipData' initially:
while((bytesRead = ais.read(buffer, 0, 1024)) != -1)
{
bufferIdx = 0;
for (int i = 0, n = bytesRead / 2; i < n; i ++)
{
clipData[(int)clipIdx++] =
( buffer[(int)bufferIdx++] & 0xff )
| ( buffer[(int)bufferIdx++] << 8 ) ;
}
}
For playback, the object that holds this data array has two get() methods. The first is for normal speed. An int is used to index into the clipData array (maybe should be a 'long', for larger audio files!):
public double[] get(int idx) throws ArrayIndexOutOfBoundsException
{
idx *= 2; // assumed: stereo data
double[] audioVals = new double[2];
audioVals[0] = clipData[idx++];
audioVals[1] = clipData[idx];
return audioVals;
}
Maybe returning a float array is acceptable, in place of the double[]?
Here is the enhanced get() method for variable speed. It uses linear interpolation to account for the fractional part of the double used as an index into the clipData:
public double[] get(double idx) throws ArrayIndexOutOfBoundsException
{
int intPart = (int)idx * 2;
double fractionalPart = idx * 2 - intPart;
int valR1 = clipData[intPart++];
int valL1 = clipData[intPart++];
int valR2 = clipData[intPart++];
int valL2 = clipData[intPart];
double[] audioVals = new double[2];
audioVals[0] = (valR1 * (1 - fractionalPart)
+ valR2 * fractionalPart);
audioVals[1] = (valL1 * (1 - fractionalPart)
+ valL2 * fractionalPart);
return audioVals;
}
The while(playing) loop (for loading data into the playback SourceDataLine) has a variable associated with the clipData which I call "cursor" that iterates through the sound data array. For normal playback, 'cursor' is incremented by 1, and tested to make sure it goes back to zero when it reaches the end of the clipData.
You could write something like: audioData = clipData.get(cursor++) to read successive frames of the data.
For varispeed, the above would be more like this:
audioData = clipData.get(cursor += speedIncrement);
'speedIncrement' is a double. If it is set to 2.0, the playback is twice as fast. If it is set to 0.5, it is half as fast. If you put in the right checks you can even make speedIncrement equal a negative value for reverse playback.
This works as long as the speed doesn't go above the Nyquist value (at least theoretically). And again, you have to test to make sure 'cursor' hasn't gone off the edge of the clipData, but restarts at the appropriate spot at the other end of the sound data array.
Hope this helps!
Another note: you may want to rewrite the above get() methods to send a buffer's worth of reads instead of single frames. I'm currently experimenting with doing things on a per-frame basis. I think it makes the code a little easier to understand, and helps with per-frame processing and responsiveness, but it surely slows things down.