Develop Reference

Trying to use BufferedInputStream and Base64 to Encode a large file in Java

I am new to the Java I/O so please help.
I am trying to process a large file(e.g. a pdf file of 50mb) using the apache commons library.
At first I try:
byte[] bytes = FileUtils.readFileToByteArray(file);
String encodeBase64String = Base64.encodeBase64String(bytes);
byte[] decoded = Base64.decodeBase64(encodeBase64String);
But knowing that the
FileUtils.readFileToByteArray in org.apache.commons.io will load the whole file into memory, I try to use BufferedInputStream to read the file piece by piece:
BufferedInputStream bis = new BufferedInputStream(inputStream);
StringBuilder pdfStringBuilder = new StringBuilder();
int byteArraySize = 10;
byte[] tempByteArray = new byte[byteArraySize];
while (bis.available() > 0) {
if (bis.available() < byteArraySize) { // reaching the end of file
tempByteArray = new byte[bis.available()];
}
int len = Math.min(bis.available(), byteArraySize);
read = bis.read(tempByteArray, 0, len);
if (read != -1) {
pdfStringBuilder.append(Base64.encodeBase64String(tempByteArray));
} else {
System.err.println("End of file reached.");
}
}
byte[] bytes = Base64.decodeBase64(pdfStringBuilder.toString());
However, the 2 decoded bytes array don't look quite the same... ... In fact, the only give 10 bytes, which is my temp array size... ...
Can anyone please help:
what am I doing it wrong to read the file piece by piece?
why is the decoded byte array only returns 10 bytes in the 2nd solution?
Thanks in advance:)

After some digging, it turns out that the byte array's size has to be multiple of 3 in order to avoid padding. After using a temp array size with multiple of 3, the program is able to go through.
I simply change
int byteArraySize = 10;
to be
int byteArraySize = 1024 * 3;

Audio samples mixing or changing volume causes saturation and white noise

I have a multichannel input (i'm using Soundflower 64ch on mac), and I'm trying to mixdown 4 channels of the 64 channels to an stereo output.
What i am doing is, reading chunks of 1024 frames, with 64 channels every frame, then converting the bytebuffer to Short array (values between -32,768 <-> 32,767, because samples are 16 bits).
This way I add for example channel1[sample] + channel2[sample] and I get the mix of both channels.
But here is a problem, the sum can overflow the Short (16 bit) range, introducing saturation in the sound. So what I'm doing is (channel1[sample] + channel2[sample]) / 2 but when I divide by 2, I hear a lot of white sound.
Also if I try to reduce the volumen of a channel by doing channel1[sample] * 0.5 there is a lot of saturation.
Why does it happen?
Here is my full code, note that I'm converting bytes to short to handle better, and then I'm converting back to bytes for write the mix to the stereo output:
public static void main(String[] args) throws LineUnavailableException {
int inputChannels = 64;
AudioFormat inputFormat = new AudioFormat(48000, 16, inputChannels, true, false);
AudioFormat outputFormat = new AudioFormat(48000, 16, 2, true, false);
TargetDataLine mic = AudioSystem.getTargetDataLine(inputFormat);
SourceDataLine speaker = AudioSystem.getSourceDataLine(outputFormat);
mic.open(inputFormat);
speaker.open(outputFormat);
mic.start();
speaker.start();
AudioInputStream audioInputStream = new AudioInputStream(mic);
int bytesPerFrame = audioInputStream.getFormat().getFrameSize();
// Set an arbitrary buffer size of 1024 frames.
int CHUNK = 1024 ;
int numBytes = CHUNK * bytesPerFrame;
byte[] audioBytes = new byte[numBytes];
try {
byte[][] frames = new byte[CHUNK][bytesPerFrame];
int i = 0, j = 0
;
while (true) {
// read to audioBytes.
audioInputStream.read(audioBytes);
// split audioBytes in _CHUNK_ frames (1024 frames)
for(j=0; j<CHUNK; j++) {
frames[j] = Arrays.copyOfRange(audioBytes, j * bytesPerFrame, j * bytesPerFrame + bytesPerFrame);
}
// convert bytearray to shortarray
short[][] shortFrames = new short[CHUNK][inputChannels];
for(i=0; i < frames.length; i++) {
ByteBuffer.wrap(frames[i]).order(ByteOrder.BIG_ENDIAN).asShortBuffer().get(shortFrames[i]);
}
short[] leftOutput = new short[CHUNK*2];
short[] rightOutput = new short[CHUNK*2];
for (i=0; i<CHUNK; i++) {
short channel1 = shortFrames[i][0];
short channel2 = shortFrames[i][1];
short channel3 = shortFrames[i][2];
short channel4 = shortFrames[i][3];
leftOutput[i] = (short)(channel4);
rightOutput[i] = (short)(channel4);;
}
//convert shortarray in byte buffer
ByteBuffer byteBuf = ByteBuffer.allocate(CHUNK * 2 * 2); // 2 bytes * 2 output channels
for (i=0; i<CHUNK; i++) {
byteBuf.putShort(leftOutput[i]);
byteBuf.putShort(rightOutput[i]);
}
speaker.write(byteBuf.array(),0,byteBuf.array().length);
}
} catch (Exception ex) {
// Handle the error...
System.out.println("exception");
System.out.println(ex.toString());
}
}

IDK if the issue is how the bytes are being converted to shorts and back, but since you asked about this in the comment, I will post it. Assume buffer has contiguous little-endian bytes at 16-bit encoding. Just reverse the byte indexes for big-endian.
pcmShort = ( buffer[i] & 0xff ) | ( buffer[i+1] << 8 );
The conversion of pcm to byte that I use follows (for little-endian, reverse the indexes for big-endian):
outBuffer[i] = (byte)pcmShort[0];
outBuffer[i+1] = (byte)((int)pcmShort[0] >> 8);
Maybe you can use the two methods (your attempt with ByteBuffer and getShort, and the above) side-by-side on the same data and check if the resulting arrays hold the same values?
Another thing I'd try to do is to just get a single track working. If that sounds okay, then check on the mixing. It's kind of unlikely that the signals are so hot that they are overrunning. So something else is probably going on.
I should try this out myself, I'm not sure when I'll get to it. It could potentially be an improvement over what I've been doing.

Force Java Android Socket to Send Data Immediately

As a hobby project, I'm writing an android voip client. When writing voice data to the socket (Vars.mediaSocket), many times, the data isn't immediately sent out over the wifi but just stalls and then all at once it will send 20 seconds worth of voice. Then it will stall again and wait for 30 seconds and then send 30 seconds of voice. The wait is not consistent but after a while it will continuously send voice data immediately. I've tried everything from using DataOutputStream to setting the socket output buffer size, setting the sendbuffer size huge, small, and lastly, buffering the voice data from its 32 byte chunks to anything from 128bytes to 32kb.
Utils.logcat(Const.LOGD, encTag, "MediaCodec encoder thread has started");
isEncoding = true;
byte[] amrbuffer = new byte[32];
short[] wavbuffer = new short[160];
int outputCounter = 0;
//setup the wave audio recorder. since it is released and restarted, it needs to be setup here and not onCreate
wavRecorder = null; //remove pointer to the old recorder for safety
wavRecorder = new AudioRecord(MediaRecorder.AudioSource.MIC, SAMPLESWAV, AudioFormat.CHANNEL_IN_MONO, FORMAT, 160);
wavRecorder.startRecording();
AmrEncoder.init(0);
while(!micMute)
{
int totalRead = 0, dataRead;
while(totalRead < 160)
{//although unlikely to be necessary, buffer the mic input
dataRead = wavRecorder.read(wavbuffer, totalRead, 160 - totalRead);
totalRead = totalRead + dataRead;
}
int encodeLength = AmrEncoder.encode(AmrEncoder.Mode.MR122.ordinal(), wavbuffer, amrbuffer);
try
{
Vars.mediaSocket.getOutputStream().write(amrbuffer);
Vars.mediaSocket.getOutputStream().flush();
}
catch (IOException i)
{
Utils.logcat(Const.LOGE, encTag, "Cannot send amr out the media socket");
Utils.dumpException(tag, i);
}
Is there something I'm missing? To simulate a second cell phone, I have another client which just simply reads the voice data, throws it away, and reads again in a loop. I can confirm in the simulated second cell phone when the real cell phone stops sending voice, the simulated one's socket.read hangs until the real one starts sending voice again.
I'm really hoping not to have to write a jni for the socket as I don't know anything about that and was hoping I could write the app as a standard java app.
CASE CLOSED: turned out to be a server side bug but the simplifying back to basics suggestions is still a good idea.

You are adding most of the latency yourself by reading large amounts of data before writing any of it. You should just use the standard Java copy loop:
byte[] buffer = new byte[8192];
int count;
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
You need to adapt this to incorporate your codec step. Note that you don't need a buffer the size of the entire input. You can tune its size to suit yourself but 8192 is a good starting point. You can increase it to say 32k but don't decrease it. If your codec needs the data in fixed-size chunks, use a buffer of that size and DataInputStream.readFully(). But the larger the buffer the more the latency.
EDIT Specific issues with your code:
byte[] amrbuffer = new byte[AMRBUFFERSIZE];
byte[] outputbuffer = new byte [outputBufferSize];
Remove (see below).
short[] wavbuffer = new short[WAVBUFFERSIZE];
int outputCounter = 0;
Remove outputCounter.
//setup the wave audio recorder. since it is released and restarted, it needs to be setup here and not onCreate
wavRecorder = null; //remove pointer to the old recorder for safety
Pointless. Remove.
wavRecorder = new AudioRecord(MediaRecorder.AudioSource.MIC, SAMPLESWAV, AudioFormat.CHANNEL_IN_MONO, FORMAT, WAVBUFFERSIZE);
wavRecorder.startRecording();
AmrEncoder.init(0);
OK.
try
{
Vars.mediaSocket.setSendBufferSize(outputBufferSize);
}
catch (SocketException e)
{
e.printStackTrace();
}
Pointless. Remove. The socket send buffer should be as large as possible. Unless you know that its default size is < outputBufferSize there is no benefit to this. In any case we are getting rid of outputBuffer altogether.
while(!micMute)
{
int totalRead = 0, dataRead;
while(totalRead < WAVBUFFERSIZE)
{//although unlikely to be necessary, buffer the mic input
dataRead = wavRecorder.read(wavbuffer, totalRead, WAVBUFFERSIZE - totalRead);
totalRead = totalRead + dataRead;
}
int encodeLength = AmrEncoder.encode(AmrEncoder.Mode.MR122.ordinal(), wavbuffer, amrbuffer);
OK.
if(outputCounter == outputBufferSize)
{
Utils.logcat(Const.LOGD, encTag, "Sending output buffer");
try
{
Vars.mediaSocket.getOutputStream().write(outputbuffer);
Vars.mediaSocket.getOutputStream().flush();
}
catch (IOException i)
{
Utils.logcat(Const.LOGE, encTag, "Cannot send amr out the media socket");
Utils.dumpException(tag, i);
}
outputCounter = 0;
}
System.arraycopy(amrbuffer, 0, outputbuffer, outputCounter, encodeLength);
outputCounter = outputCounter + encodeLength;
Utils.logcat(Const.LOGD, encTag, "Output buffer fill: " + outputCounter);
Remove all the above and substitute
Vars.mediaSocket.getOutputStream().write(amrbuffer, 0, encodeLength);
This also means you can get rid of 'outputBuffer' as promised.
NB Don't flush inside loops. As a matter of fact flushing a socket output stream does nothing, but the general principle still holds.

java: read large binary file

I need to read out a given large file that contains 500000001 binaries. Afterwards I have to translate them into ASCII.
My Problem occurs while trying to store the binaries in a large array. I get the warning at the definition of the array ioBuf:
"The literal 16000000032 of type int is out of range."
I have no clue how to save these numbers to work with them! Has somebody an idea?
Here is my code:
public byte[] read(){
try{
BufferedInputStream in = new BufferedInputStream(new FileInputStream("data.dat"));
ByteArrayOutputStream bs = new ByteArrayOutputStream();
BufferedOutputStream out = new BufferedOutputStream(bs);
byte[] ioBuf = new byte[16000000032];
int bytesRead;
while ((bytesRead = in.read(ioBuf)) != -1){
out.write(ioBuf, 0, bytesRead);
}
out.close();
in.close();
return bs.toByteArray();
}

The maximum Index of an Array is Integer.MAX_VALUE and 16000000032 is greater than Integer.MAX_VALUE
Integer.MAX_VALUE = 2^31-1 = 2147483647
2147483647 < 16000000032
You could overcome this by checking if the Array is full and create another and continue reading.
But i'm not quite sure if your approach is the best way to perform this. byte[Integer_MAX_VALUE] is huge ;)
Maybe you can split the input file in smaller chunks process them.
EDIT: This is how you could read a single int of your file. You can resize the buffer's size to the amount of data you want to read. But you tried to read the whole file at once.
//Allocate buffer with 4byte = 32bit = Integer.SIZE
byte[] ioBuf = new byte[4];
int bytesRead;
while ((bytesRead = in.read(ioBuf)) != -1){
//if bytesRead == 4 you read 1 int
//do your stuff
}

If you need to declare a large constant, append an 'L' to it which indicates to the compiler that is a long constant. However, as mentioned in another answer you can't declare arrays that large.
I suspect the purpose of the exercise is to learn how to use the java.nio.Buffer family of classes.

I made some progress by starting from scratch! But I still have a problem.
My idea is to read up the first 32 bytes, convert them to a int number. Then the next 32 bytes etc. Unfortunately I just get the first and don't know how to proceed.
I discovered following method for converting these numbers to int:
public static int byteArrayToInt(byte[] b){
final ByteBuffer bb = ByteBuffer.wrap(b);
bb.order(ByteOrder.LITTLE_ENDIAN);
return bb.getInt();
}
so now I have:
BufferedInputStream in=null;
byte[] buf = new byte[32];
try {
in = new BufferedInputStream(new FileInputStream("ndata.dat"));
in.read(buf);
System.out.println(byteArrayToInt(buf));
in.close();
} catch (IOException e) {
System.out.println("error while reading ndata.dat file");
}

Example code for java.util.zip.Deflater that uses batch modes for both input and output

The example code in the javadoc for java.util.zip.Deflater is too optimistic, and assumes that you have a byte array containing all input, and a byte array that is large enough for the output. >:(
Is there an example somewhere that calls both needsInput() to add input in batches, and finished() to get output in batches? I can't seem to find one, and the docs are a little hazy about what is the right order of operations.
Example:
I have a ByteBuffer that is a 100MB memory-mapped file
I'm writing the output to a stream
I have a these byte arrays of length 1024 for batches of input and output:
byte[] inbatch = new byte[1024];
byte[] outbatch = new byte[1024];
It seems like something like this might work, but I'm not sure, and I suspect there may be some subtle edge cases...
Deflater deflater = ...;
byte[] inbatch = new byte[1024];
byte[] outbatch = new byte[1024];
boolean inputDone = false;
while (true)
{
if (!inputDone && deflater.needsInput())
{
int n = get_more_input(..., inbatch);
if (n == 0)
inputDone = true;
else
deflater.setInput(inbatch, 0, n);
}
if (deflater.finished()
{
if (inputDone)
break;
}
else
{
int n = deflater.deflate(outbatch);
handle_output(outbatch, n);
}
}