Develop Reference

java: read large binary file

I need to read out a given large file that contains 500000001 binaries. Afterwards I have to translate them into ASCII.
My Problem occurs while trying to store the binaries in a large array. I get the warning at the definition of the array ioBuf:
"The literal 16000000032 of type int is out of range."
I have no clue how to save these numbers to work with them! Has somebody an idea?
Here is my code:
public byte[] read(){
try{
BufferedInputStream in = new BufferedInputStream(new FileInputStream("data.dat"));
ByteArrayOutputStream bs = new ByteArrayOutputStream();
BufferedOutputStream out = new BufferedOutputStream(bs);
byte[] ioBuf = new byte[16000000032];
int bytesRead;
while ((bytesRead = in.read(ioBuf)) != -1){
out.write(ioBuf, 0, bytesRead);
}
out.close();
in.close();
return bs.toByteArray();
}

The maximum Index of an Array is Integer.MAX_VALUE and 16000000032 is greater than Integer.MAX_VALUE
Integer.MAX_VALUE = 2^31-1 = 2147483647
2147483647 < 16000000032
You could overcome this by checking if the Array is full and create another and continue reading.
But i'm not quite sure if your approach is the best way to perform this. byte[Integer_MAX_VALUE] is huge ;)
Maybe you can split the input file in smaller chunks process them.
EDIT: This is how you could read a single int of your file. You can resize the buffer's size to the amount of data you want to read. But you tried to read the whole file at once.
//Allocate buffer with 4byte = 32bit = Integer.SIZE
byte[] ioBuf = new byte[4];
int bytesRead;
while ((bytesRead = in.read(ioBuf)) != -1){
//if bytesRead == 4 you read 1 int
//do your stuff
}

If you need to declare a large constant, append an 'L' to it which indicates to the compiler that is a long constant. However, as mentioned in another answer you can't declare arrays that large.
I suspect the purpose of the exercise is to learn how to use the java.nio.Buffer family of classes.

I made some progress by starting from scratch! But I still have a problem.
My idea is to read up the first 32 bytes, convert them to a int number. Then the next 32 bytes etc. Unfortunately I just get the first and don't know how to proceed.
I discovered following method for converting these numbers to int:
public static int byteArrayToInt(byte[] b){
final ByteBuffer bb = ByteBuffer.wrap(b);
bb.order(ByteOrder.LITTLE_ENDIAN);
return bb.getInt();
}
so now I have:
BufferedInputStream in=null;
byte[] buf = new byte[32];
try {
in = new BufferedInputStream(new FileInputStream("ndata.dat"));
in.read(buf);
System.out.println(byteArrayToInt(buf));
in.close();
} catch (IOException e) {
System.out.println("error while reading ndata.dat file");
}

Java Socket Inputstream Reading Wrong Data

Can anyone tell what wrong with this code, what I done is that
Client send image data which is uchar array from c++.
Java server receives this data using server socket and store as byte array.
Here is the code...
private ServerSocket serverSocket;
InputStream in;
int imageSize=300;//921600;//expected image size 640X480X3
public imageReciver(int port) throws IOException {
serverSocket = new ServerSocket(port);
}
Socket server = null;
server = serverSocket.accept();
in = server.getInputStream();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte buffer[] = new byte[100];
int remainingBytes = imageSize; //
while (remainingBytes > 0) {
int bytesRead = in.read(buffer);
if (bytesRead < 0) {
throw new IOException("Unexpected end of data");
}
baos.write(buffer, 0, bytesRead);
remainingBytes -= bytesRead;
}
in.close();
byte imageByte[] = new byte[imageSize];
imageByte = baos.toByteArray();
baos.close();
While reading from inputstream in I am getting negative value on buffer.

I think the problem is:
Java's byte is signed and you are sending unsigned bytes.
From Java docs:
byte: The byte data type is an 8-bit signed two's complement integer.
It has a minimum value of -128 and a maximum value of 127 (inclusive).
The byte data type can be useful for saving memory in large arrays,
where the memory savings actually matters. They can also be used in
place of int where their limits help to clarify your code; the fact
that a variable's range is limited can serve as a form of
documentation.
You can try something like:
short bVal = /*signed byte*/;
if (bVal < 0)
bVal = bVal + 256; //Convert to positive
Edit:
As pointed out by #Aubin, a short will be a better choice instead of an int.

The representation of a data as signed or unsigned is a human convention. This convention is used by the processor when arithmetic is used.
It's not the case here: it's simple read and write and the fact the bit 7 is used as negative flag or not as no effect.
Reading a socket and writing a file may be done with byte buffer or better with ByteBuffer. I suggest a buffer size of 8K.
Rewriting the code using java.nio.channels may be better.

Byte [] used as a BufferInputStream ok but

Ok I know a buffer is actually an array of byte, however I have never seen the following declaration (taken from here)
URLConnection con = new URL("http://maps...").openConnection();
InputStream is = con.getInputStream();
byte bytes[] = new byte[con.getContentLength()];
is.read(bytes);
Is it the right way to avoid using a BufferInputStream object? Here we have an unbuffered stream reading from a byte []? should not be the other way around?
thanks in advance.

No, it is not the right way. Method read() reads up to N bytes where N is the length of your array. It can read less bytes (even 0) if no more byte are available. Number of bytes that have been read is returned by method read(). When end of stream is reached the method returns -1.
Therefore the right way is to read bytes in loop:
byte[] buf = new buf[MAX];
int n = 0;
while ((n = stream.read(buf)) >= 0) {
// deal with n first bytes from buf
}

or use Apache commons-io
InputStream is;
byte[] bytes = IOUtils.toByteArray(is);

Reading a binary input stream into a single byte array in Java

The documentation says that one should not use available() method to determine the size of an InputStream. How can I read the whole content of an InputStream into a byte array?
InputStream in; //assuming already present
byte[] data = new byte[in.available()];
in.read(data);//now data is filled with the whole content of the InputStream
I could read multiple times into a buffer of a fixed size, but then, I will have to combine the data I read into a single byte array, which is a problem for me.

The simplest approach IMO is to use Guava and its ByteStreams class:
byte[] bytes = ByteStreams.toByteArray(in);
Or for a file:
byte[] bytes = Files.toByteArray(file);
Alternatively (if you didn't want to use Guava), you could create a ByteArrayOutputStream, and repeatedly read into a byte array and write into the ByteArrayOutputStream (letting that handle resizing), then call ByteArrayOutputStream.toByteArray().
Note that this approach works whether you can tell the length of your input or not - assuming you have enough memory, of course.

Please keep in mind that the answers here assume that the length of the file is less than or equal to Integer.MAX_VALUE(2147483647).
If you are reading in from a file, you can do something like this:
File file = new File("myFile");
byte[] fileData = new byte[(int) file.length()];
DataInputStream dis = new DataInputStream(new FileInputStream(file));
dis.readFully(fileData);
dis.close();
UPDATE (May 31, 2014):
Java 7 adds some new features in the java.nio.file package that can be used to make this example a few lines shorter. See the readAllBytes() method in the java.nio.file.Files class. Here is a short example:
import java.nio.file.FileSystems;
import java.nio.file.Files;
import java.nio.file.Path;
// ...
Path p = FileSystems.getDefault().getPath("", "myFile");
byte [] fileData = Files.readAllBytes(p);
Android has support for this starting in Api level 26 (8.0.0, Oreo).

You can use Apache commons-io for this task:
Refer to this method:
public static byte[] readFileToByteArray(File file) throws IOException
Update:
Java 7 way:
byte[] bytes = Files.readAllBytes(Paths.get(filename));
and if it is a text file and you want to convert it to String (change encoding as needed):
StandardCharsets.UTF_8.decode(ByteBuffer.wrap(bytes)).toString()

You can read it by chunks (byte buffer[] = new byte[2048]) and write the chunks to a ByteArrayOutputStream. From the ByteArrayOutputStream you can retrieve the contents as a byte[], without needing to determine its size beforehand.

I believe buffer length needs to be specified, as memory is finite and you may run out of it
Example:
InputStream in = new FileInputStream(strFileName);
long length = fileFileName.length();
if (length > Integer.MAX_VALUE) {
throw new IOException("File is too large!");
}
byte[] bytes = new byte[(int) length];
int offset = 0;
int numRead = 0;
while (offset < bytes.length && (numRead = in.read(bytes, offset, bytes.length - offset)) >= 0) {
offset += numRead;
}
if (offset < bytes.length) {
throw new IOException("Could not completely read file " + fileFileName.getName());
}
in.close();

Max value for array index is Integer.MAX_INT - it's around 2Gb (2^31 / 2 147 483 647).
Your input stream can be bigger than 2Gb, so you have to process data in chunks, sorry.
InputStream is;
final byte[] buffer = new byte[512 * 1024 * 1024]; // 512Mb
while(true) {
final int read = is.read(buffer);
if ( read < 0 ) {
break;
}
// do processing
}

Assigning to a byte array in Java

I have a byte array I want to assign as follows:
First byte specifies the length of the string: (byte)string.length()
2nd - Last bytes contain string data from string.getBytes()
Other than using a for loop, is there a quick way to initialize a byte array using bytes from two different variables?

You can use System.arrayCopy() to copy your bytes:
String x = "xx";
byte[] out = new byte[x.getBytes().length()+1];
out[0] = (byte) (0xFF & x.getBytes().length());
System.arraycopy(x.getBytes(), 0, out, 1, x.length());
Though using something like a ByteArrayOutputStream or a ByteBuffer like other people suggested is probably a cleaner approach and will be better for your in the long run :-)

How about ByteBuffer ?
Example :
ByteBuffer bb = ByteBuffer.allocate(string.getBytes().length +1 );
bb.put((byte) string.length());
bb.put(string.getBytes());

While ByteBuffer is generally the best way to build up byte arrays, given the OP's goals I think the following will be more robust:
public static void main(String[] argv)
throws Exception
{
String s = "any string up to 64k long";
ByteArrayOutputStream bos = new ByteArrayOutputStream();
DataOutputStream out = new DataOutputStream(bos);
out.writeUTF(s);
out.close();
byte[] bytes = bos.toByteArray();
ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
DataInputStream in = new DataInputStream(bis);
String s2 = in.readUTF();
}

How about ByteArrayOutputStream?