I know that there are a lot of resources online explaining how to deinterleave PCM data. In the course of my current project I have looked at most of them...but I have no background in audio processing and I have had a very hard time finding a detailed explanation of how exactly this common form of audio is stored.
I do understand that my audio will have two channels and thus the samples will be stored in the format [left][right][left][right]...
What I don't understand is what exactly this means. I have also read that each sample is stored in the format [left MSB][left LSB][right MSB][right LSB]. Does this mean the each 16 bit integer actually encodes two 8 bit frames, or is each 16 bit integer its own frame destined for either the left or right channel?
Thank you everyone. Any help is appreciated.
Edit: If you choose to give examples please refer to the following.
Method Context
Specifically what I have to do is convert an interleaved short[] to two float[]'s each representing the left or right channel. I will be implementing this in Java.
public static float[][] deinterleaveAudioData(short[] interleavedData) {
//initialize the channel arrays
float[] left = new float[interleavedData.length / 2];
float[] right = new float[interleavedData.length / 2];
//iterate through the buffer
for (int i = 0; i < interleavedData.length; i++) {
//THIS IS WHERE I DON'T KNOW WHAT TO DO
}
//return the separated left and right channels
return new float[][]{left, right};
}
My Current Implementation
I have tried playing the audio that results from this. It's very close, close enough that you could understand the words of a song, but is still clearly not the correct method.
public static float[][] deinterleaveAudioData(short[] interleavedData) {
//initialize the channel arrays
float[] left = new float[interleavedData.length / 2];
float[] right = new float[interleavedData.length / 2];
//iterate through the buffer
for (int i = 0; i < left.length; i++) {
left[i] = (float) interleavedData[2 * i];
right[i] = (float) interleavedData[2 * i + 1];
}
//return the separated left and right channels
return new float[][]{left, right};
}
Format
If anyone would like more information about the format of the audio the following is everything I have.
Format is PCM 2 channel interleaved big endian linear int16
Sample rate is 44100
Number of shorts per short[] buffer is 2048
Number of frames per short[] buffer is 1024
Frames per packet is 1
I do understand that my audio will have two channels and thus the samples will be stored in the format [left][right][left][right]... What I don't understand is what exactly this means.
Interleaved PCM data is stored one sample per channel, in channel order before going on to the next sample. A PCM frame is made up of a group of samples for each channel. If you have stereo audio with left and right channels, then one sample from each together make a frame.
Frame 0: [left sample][right sample]
Frame 1: [left sample][right sample]
Frame 2: [left sample][right sample]
Frame 3: [left sample][right sample]
etc...
Each sample is a measurement and digital quantization of pressure at an instantaneous point in time. That is, if you have 8 bits per sample, you have 256 possible levels of precision that the pressure can be sampled at. Knowing that sound waves are... waves... with peaks and valleys, we are going to want to be able to measure distance from the center. So, we can define center at 127 or so and subtract and add from there (0 to 255, unsigned) or we can treat those 8 bits as signed (same values, just different interpretation of them) and go from -128 to 127.
Using 8 bits per sample with single channel (mono) audio, we use one byte per sample meaning one second of audio sampled at 44.1kHz uses exactly 44,100 bytes of storage.
Now, let's assume 8 bits per sample, but in stereo at 44.1.kHz. Every other byte is going to be for the left, and every other is going to be for the R.
LRLRLRLRLRLRLRLRLRLRLR...
Scale it up to 16 bits, and you have two bytes per sample (samples set up with brackets [ and ], spaces indicate frame boundaries)
[LL][RR] [LL][RR] [LL][RR] [LL][RR] [LL][RR] [LL][RR]...
I have also read that each sample is stored in the format [left MSB][left LSB][right MSB][right LSB].
Not necessarily. The audio can be stored in any endianness. Little endian is the most common, but that isn't a magic rule. I do think though that all channels go in order always, and front left would be channel 0 in most cases.
Does this mean the each 16 bit integer actually encodes two 8 bit frames, or is each 16 bit integer its own frame destined for either the left or right channel?
Each value (16-bit integer in this case) is destined for a single channel. Never would you have two multi-byte values smashed into each other.
I hope that's helpful. I can't run your code but given your description, I suspect you have an endian problem and that your samples aren't actual big endian.
Let's start by getting some terminology out of the way
A channel is a monaural stream of samples. The term does not necessarily imply that the samples are contiguous in the data stream.
A frame is a set of co-incident samples. For stereo audio (e.g. L & R channels) a frame contains two samples.
A packet is 1 or more frames, and is typically the minimun number of frames that can be processed by a system at once. For PCM Audio, a packet often contains 1 frame, but for compressed audio it will be larger.
Interleaving is a term typically used for stereo audio, in which the data stream consists of consecutive frames of audio. The stream therefore looks like L1R1L2R2L3R3......LnRn
Both big and little endian audio formats exist, and depend on the use-case. However, it's generally ever an issue when exchanging data between systems - you'll always use native byte-order when processing or interfacing with operating system audio components.
You don't say whether you're using a little or big endian system, but I suspect it's probably the former. In which case you need to byte-reverse the samples.
Although not set in stone, when using floating point samples are usually in the range -1.0<x<+1.0, so you want to divide the samples by 1<<15. When 16-bit linear types are used, they are typically signed.
Taking care of byte-swapping and format conversions:
int s = (int) interleavedData[2 * i];
short revS = (short) (((s & 0xff) << 8) | ((s >> 8) & 0xff))
left[i] = ((float) revS) / 32767.0f;
Actually your are dealing with an almost typical WAVE file at Audio CD quality, that is to say :
2 channels
sampling rate of 44100 kHz
each amplitude sample quantized on a 16-bits signed integer
I said almost because big-endianness is usually used in AIFF files (Mac world), not in WAVE files (PC world). And I don't know without searching how to deal with endianness in Java, so I will leave this part to you.
About how the samples are stored is quite simple:
each sample takes 16-bits (integer from -32768 to +32767)
if channels are interleaved: (L,1),(R,1),(L,2),(R,2),...,(L,n),(R,n)
if channels are not: (L,1),(L,2),...,(L,n),(R,1),(R,2),...,(R,n)
Then to feed an audio callback, it is usually required to provide 32-bits floating point, ranging from -1 to +1. And maybe this is where something may be missing in your aglorithm. Dividing your integers by 32768 (2^(16-1)) should make it sound as expected.
I ran into a similar issue with de-interleaving the short[] frames that came in through Spotify Android SDK's onAudioDataDelivered().
The documentation for onAudioDelivered was poorly written a year ago. See Github issue. They've updated the docs with a better description and more accurate parameter names:
onAudioDataDelivered(short[] samples, int sampleCount, int sampleRate, int channels)
What can be confusing is that samples.length can be 4096. However, it contains only sampleCount valid samples. If you're receiving stereo audio, and sampleCount = 2048 there are only 1024 frames (each frame has two samples) of audio in samples array!
So you'll need to update your implementation to make sure you're working with sampleCount and not samples.length.
Related
I have changed my question message...
I have two streams with audio in Java. What I want is to combine these two audios into one OutputStream.
I've being searching and it seems that if you have both streams with the same audio format, and using PCM, you just need to do the following operation with the two byte arrays:
mixAudio[i] = (byte) ((audio1[i] + audio2[i]) >> 1);
However I am writing this into a file and I get a file without any audio.
Does anyone know how to combine two audios when I have the audios in two streams (not two audio files)?
Thank you in advance.
decent quality audio consumes two bytes of data per sample per channel to give the audio curve a bit depth of 16 bits which gives your audio curve 2^16 distinct values when digitizing the analog audio curve ... knowing this you cannot do your adding while the data lives as simply bytes ... so to add together two channels you first need to get your audio out of its bytes and into a two byte integer ... then you need to pluck out of that two byte integer each of those two bytes one by one and stow into your output array
in pseudo code ( this puts into an integer two consecutive bytes of your audio array which represents one sample in your audio curve )
assign into a 16 bit integer value of your most significant byte
left shift this integer by 8 bits something like ( myint = myint << 8 )
bit level add to this integer your 2nd byte which is your least significant byte
Top Tip : after you have written code to populate one integer from two bytes then do the reverse namely convert a multi byte integer into two bytes in some array ... bonus points if you plot these integers so you can visualize your raw audio curve
To perform above you must know your endianness ( are you doing little endian or big endian ) which will determine the order of your bytes ... specifically since we now know each audio sample consumes two bytes (or more say for 24 bit audio ) the bytes myarray[i] and myarray[i + 1] are one audio sample however only after knowing your endianness will you realize which array element to use first when populating the above myint ... if none of this makes sense please invest time and effort to research notion of raw audio in PCM format
I highly encourage you to do all of above in your code at least once to appreciate what is happening inside some audio library which may do this for you
going back to your question instead of simply doing
mixAudio[i] = (byte) ((audio1[i] + audio2[i]) >> 1);
you should be doing something like this (untested especially regarding endianess)
twoByteAnswer = (byte) ((audio1[i] << 8) + audio1[i + 1]) + (audio2[i] << 8 + audio2[i + 1])) >> 1);
now you need to spread out your twoByteAnswer into two bytes of array mixAudio ... something like this (also untested)
mixAudio[i] = twoByteAnswer >> 8 // throw away its least sig byte only using its most sig byte
mixAudio[i + 1] = twoByteAnswer && 0x0000FFFF // do a bit AND operator mask
I am trying to join byte-arrays of wav-sound and it works except for backgroundnoise. Anyone knows any algoritm to add two byte-arrays of sound.
This is what I have tried so far
for(int i=0;i<bArr1.length;i++)
{
bArrJoined[i]=bArr1[i] + bArr2[i];
}
also tried to divide by 2 not to be to high numbers
for(int i=0;i<bArr1.length;i++)
{
bArrJoined[i]=(bArr1[i] + bArr2[i]) / 2;
}
Anyone knows how to make this work without the noise?
A number of things could cause artifacts here. Different audio sampling rates or data bit sizes could do it.
Assuming those are non-issues, you should be aware you can't add a byte with another byte without overflow (256 will become 0, etc.). So convert to int before adding. Clipping will occur if you exceed the max volume, so your divide by 2 operation is smart and should stop that issue. The divide operation should occur with the int versions. Only cast back to byte at the end.
However, if you aren't working with 8-bit audio, then a byte is not your atomic unit. For example, 16-bit audio uses 2 bytes and you would need to convert every two consecutive bytes to an int (with respect to proper endianness) before you perform any mathematical operations on the values. 32-bit audio data occupies 4 consecutive bytes for each single numeric value. Just having an array of bytes does not in itself tell you where the data boundaries are.
I have a thread that takes screen shots using java.awt.Robot and then encodes them into a video using Xuggler in a loop.
The loop encodes the image, then makes the thread sleep for some time depending upon the frame rate.
All good so far. The problems arise when I try to encode audio.
Specifically, maintaining the sample rate and size
I am using TargetDataLine to read data into a byte[]. This data is already BigEndian formatted.
The magic is in providing proper amount of data at proper time.
My AudioFormat looks like this:
Sample Rate: 44000Hz
Sample Size In Bits: 16
Signed: true
BigEndian: true
Assuming 10fps and 44000Hz sample rate, I will need to provide
what should be the size of the byte[]?
how much data? (measured in short because that is what Xuggler wants)
and at what time do I call the encodeAudio() method? I mean after 10 passes of the loop or 5 passes, etc.
Misc:
Community member Alex I gave me this formula:
shortArray.length == ((timeStamp - lastTimeStamp) / 1e+9) * sampleRate * channels;
a rough calculation got me the answer of 4782 shorts for one second.
I know when you pass the audio to be encoded it must be for one full second
So I must capture 480 shorts per pass, and then encode it finally after the 10th pass.
Please tell me if this deduction is correct?
If you have 16 bits (two bytes) at a sample rate of 44,000 Hz that is 88,000 bytes per second. If you have stereo it is double that. In 1/10 second you need a 1/10th of that. i.e. 8,800 bytes per deci-second (1/10th of a second)
How do i retrieve short[] from ByteBuffer that has a byte[] as backing array
byte[] bytes = { };
ByteBuffer bb = ByteBuffer.wrap(bytes);
short[] shorts = new short[bb.remaining()/2];
bb.order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(shorts);
If the order is BigEndian you don't need to change it.
This is a quick question, how is audio stored when it's in a byte array, like when a image is stored in a byte array there is three(Red, green, blue) bytes per pixel. So how is a audio stored in a byte array?
Thanks,
Liam.
There are various possible encodings that are supported in Java. See:
http://docs.oracle.com/javase/1.5.0/docs/api/javax/sound/sampled/AudioFormat.html
http://docs.oracle.com/javase/1.5.0/docs/api/javax/sound/sampled/AudioFormat.Encoding.html
The most simple form is PCM coding, in which each sample is a linear number that represents the sound waveform (which could be 1 byte for 8-bit encoding).
You also have to consider the number of channels (1 for mono, 2 for stereo). So 16-bit PCM-encoded stereo sound will require 4 bytes per sample, for example.
It's a combination of signals (analog / digital) having a unique frequency for each and every tone. And as it's said in the previous answer, yes, Pulse Code Modulation (PCM) is supported in java.
I have pcm audio stored in a byte array. It is 16 bits per sample. I want to make it 8 bit per sample audio.
Can anyone suggest a good algorithm to do that?
I haven't mentioned the bitrate because I think it isn't important for the algorithm - right?
I can't see right now why it's not enough to just take the upper byte, i.e. discard the lower 8 bits of each sample.
That of course assumes that the samples are linear; if they're not then maybe you need to do something to linearize them before dropping bits.
short sixteenBit = 0xfeed;
byte eightBit = sixteenBit >> 8;
// eightBit is now 0xfe.
As suggested by AShelly in a comment, it might be a good idea to round, i.e. add 1 if the byte we're discarding is higher than half its maximum:
eightBit += eightBit < 0xff && ((sixteenBit & 0xff) > 0x80);
The test against 0xff implements clamping, so we don't risk adding 1 to 0xff and wrapping that to 0x00 which would be bad.
16-bit samples are usually signed, and 8-bit samples are usually unsigned, so the simplest answer is that you need to convert the 16-bit samples from signed (16-bit samples are almost always stored as a range from -32768 to +32767) to unsigned and then take the top 8 bits of the result. In C, this could be expressed as output = (unsigned char)((unsigned short)(input + 32768) >> 8). This is a good start, and might be good enough for your needs, but it won't sound very nice. It sounds rough because of "quantization noise".
Quantization noise is the difference between the original input and your algorithm's output. No matter what you do, you're going to have noise, and the noise will be "half a bit" on average. There's nothing you can do about that, but there are ways to make the noise less noticeable.
The main problem with the quantization noise is that it tends to form patterns. If the difference between input and output were completely random, things would actually sound fine, but instead the output will repeatedly be too high for a certain part of the waveform and too low for the next part. Your ear picks up on this pattern.
To have a result that sounds good, you need to add dithering. Dithering is a technique that tries to smooth-out the quantization noise. The simplest dithering just removes the patterns from the noise so that the noise patterns don't distract from the actual signal patterns. Better dithering can go a step further and take steps to reduce the noise by adding together the error values from multiple samples and then adding in a correction when the total error gets large enough to be worth correcting.
You can find explanations and code samples for various dithering algorithms online. One good area to investigate might be the SoX tool, http://en.wikipedia.org/wiki/SoX. Check the source for its dithering effect, and experiment with converting various sounds from 16-bit to 8-bit with and without dithering enabled. You will be surprised by the difference in quality that dithering can make when converting to 8-bit sound.
byteData = (byte) (((shortData +32768)>>8)& 0xFF)
this worked for me.
Normalize the 16 bit samples, then rescale by the maximum value of your 8 bit sample.
This yields a more accurate conversion as the lower 8 bits of each sample aren't being discarded. However, my solution is more computationally expensive than the selected answer.