Develop Reference

Trying to use BufferedInputStream and Base64 to Encode a large file in Java

I am new to the Java I/O so please help.
I am trying to process a large file(e.g. a pdf file of 50mb) using the apache commons library.
At first I try:
byte[] bytes = FileUtils.readFileToByteArray(file);
String encodeBase64String = Base64.encodeBase64String(bytes);
byte[] decoded = Base64.decodeBase64(encodeBase64String);
But knowing that the
FileUtils.readFileToByteArray in org.apache.commons.io will load the whole file into memory, I try to use BufferedInputStream to read the file piece by piece:
BufferedInputStream bis = new BufferedInputStream(inputStream);
StringBuilder pdfStringBuilder = new StringBuilder();
int byteArraySize = 10;
byte[] tempByteArray = new byte[byteArraySize];
while (bis.available() > 0) {
if (bis.available() < byteArraySize) { // reaching the end of file
tempByteArray = new byte[bis.available()];
}
int len = Math.min(bis.available(), byteArraySize);
read = bis.read(tempByteArray, 0, len);
if (read != -1) {
pdfStringBuilder.append(Base64.encodeBase64String(tempByteArray));
} else {
System.err.println("End of file reached.");
}
}
byte[] bytes = Base64.decodeBase64(pdfStringBuilder.toString());
However, the 2 decoded bytes array don't look quite the same... ... In fact, the only give 10 bytes, which is my temp array size... ...
Can anyone please help:
what am I doing it wrong to read the file piece by piece?
why is the decoded byte array only returns 10 bytes in the 2nd solution?
Thanks in advance:)

After some digging, it turns out that the byte array's size has to be multiple of 3 in order to avoid padding. After using a temp array size with multiple of 3, the program is able to go through.
I simply change
int byteArraySize = 10;
to be
int byteArraySize = 1024 * 3;

Add ByteArray to integer

In the following java code-snippet you'll see this line packetLengthMax += bytes.toByteArray()[43];
My question is: How does this work?
byte[] dataBuffer = new byte[265];
int packetLength = 0;
int packetLengthMax = 44;
ByteArrayOutputStream bytes = new ByteArrayOutputStream();
DataOutputStream outMessage = new DataOutputStream(bytes);
/* Client = Socket*/
DataInputStream clientIn = new DataInputStream(Client.getInputStream());
while (packetLength < packetLengthMax) {
packetLength += clientIn.read(dataBuffer);
outMessage.write(dataBuffer);
if (packetLength >= 43) {
packetLengthMax += bytes.toByteArray()[43];
}
}
My explanation:
First a socket (Client) is passed to the code. Then it does the setup of all variables. In the while loop, it reads all data that comes from the socket. Then it also writes this data to the DataOutputStream.
But in the if statement - it adds a byte array to an integer.
How does it work? I don't get that point. Thank you for helping!

It's not adding the whole byte array, it's just adding the byte at position 43. (i.e. the 44th byte in the array).

Efficient way to convert a String (representing HEX) to a byte array? Android

I'm working on a project that involves receiving a byte array over wireless, the Android app reads this as a String over a TCP connection:
input = new BufferedReader(new InputStreamReader(this.clientSocket.getInputStream()));
...
...
//Loop
String read = input.readLine();
//Do something meaningful with String read...
The String will always be of a fixed format i.e. the first 3 characters will be an ID and the next following 20 characters will be the message data. The amount of characters will not change (3+20 characters = 23, with a starting and ending character '[' and ']' so that's 25 characters in total.
An example of a String received by the application would be [01A01020304050A0B0C0D]
ID - 0x01A
Byte0 0x01
Byte1 0x02
Byte2 0x03
Byte3 0x04
Byte4 0x05
Byte5 0x0A
Byte6 0x0B
Byte7 0x0C
Byte 8 0x0D
I would guess that I would have to use the substring operation, but I'm having trouble converting the substring to a byte value (note: the app is expecting byte[] and not Byte[]) and I feel I'm not doing it efficiently. I came across this piece of code that I've been using:
public static byte[] hexStringToByteArray(String s) {
int len = s.length();
byte[] data = new byte[len / 2];
for (int i = 0; i < len; i += 2) {
data[i / 2] = (byte) ((Character.digit(s.charAt(i), 16) << 4)
+ Character.digit(s.charAt(i+1), 16));
}
return data;
This is returning a byte array of size 1 and will have to be run 9 times (9 bytes) per message. I'm a bit concerned that this may be a bit too strenuous on processing, especially when the application is receiving messages very frequently (roughly about 10-15 messages per second)
I appreciate any thoughts and many thanks in advance!

just use this :
byte[] decodedString = Base64.decode(your_string, Base64.DEFAULT);

byte[] b = string.getBytes();
byte[] b = string.getBytes(Charset.forName("UTF-8"));
byte[] b = string.getBytes("UTF-8");
There is no way to be more effecient than using this methods.

Best and simple way:
String myString = "This is my string";
byte[] myByteArray = myString.getBytes("UTF-8");
Now, you able to access id, message whatever; easily from myByteArray.

Just write your data like
byte[] data = yourData.getBytes();
os.write(data, 0, data.length) // data is of 23 bytes
os.flush();
what about reading through InputStream, as you mentioned in your question that String is of 23 characters just do like
public byte[] readData(InputStream is) {
byte[] data = new byte[23];
int read = is.read(data);
System.out.println("Read: " + read);
return data;
}
When you have data then you can split data like this
byte[] tempId = new byte[3];
System.arrayCopy(data, 0, id, 0, id.length);
byte[] tempMessage = new byte[20];
System.arrayCopy(data, 3, message, 0, message.length);
String id = new String(tempId);
String message = new String(tempMessage);
Now you id and message separated and converted into String.

byte[] array = String.getBytes("UTF-8");

Reading a binary input stream into a single byte array in Java

The documentation says that one should not use available() method to determine the size of an InputStream. How can I read the whole content of an InputStream into a byte array?
InputStream in; //assuming already present
byte[] data = new byte[in.available()];
in.read(data);//now data is filled with the whole content of the InputStream
I could read multiple times into a buffer of a fixed size, but then, I will have to combine the data I read into a single byte array, which is a problem for me.

The simplest approach IMO is to use Guava and its ByteStreams class:
byte[] bytes = ByteStreams.toByteArray(in);
Or for a file:
byte[] bytes = Files.toByteArray(file);
Alternatively (if you didn't want to use Guava), you could create a ByteArrayOutputStream, and repeatedly read into a byte array and write into the ByteArrayOutputStream (letting that handle resizing), then call ByteArrayOutputStream.toByteArray().
Note that this approach works whether you can tell the length of your input or not - assuming you have enough memory, of course.

Please keep in mind that the answers here assume that the length of the file is less than or equal to Integer.MAX_VALUE(2147483647).
If you are reading in from a file, you can do something like this:
File file = new File("myFile");
byte[] fileData = new byte[(int) file.length()];
DataInputStream dis = new DataInputStream(new FileInputStream(file));
dis.readFully(fileData);
dis.close();
UPDATE (May 31, 2014):
Java 7 adds some new features in the java.nio.file package that can be used to make this example a few lines shorter. See the readAllBytes() method in the java.nio.file.Files class. Here is a short example:
import java.nio.file.FileSystems;
import java.nio.file.Files;
import java.nio.file.Path;
// ...
Path p = FileSystems.getDefault().getPath("", "myFile");
byte [] fileData = Files.readAllBytes(p);
Android has support for this starting in Api level 26 (8.0.0, Oreo).

You can use Apache commons-io for this task:
Refer to this method:
public static byte[] readFileToByteArray(File file) throws IOException
Update:
Java 7 way:
byte[] bytes = Files.readAllBytes(Paths.get(filename));
and if it is a text file and you want to convert it to String (change encoding as needed):
StandardCharsets.UTF_8.decode(ByteBuffer.wrap(bytes)).toString()

You can read it by chunks (byte buffer[] = new byte[2048]) and write the chunks to a ByteArrayOutputStream. From the ByteArrayOutputStream you can retrieve the contents as a byte[], without needing to determine its size beforehand.

I believe buffer length needs to be specified, as memory is finite and you may run out of it
Example:
InputStream in = new FileInputStream(strFileName);
long length = fileFileName.length();
if (length > Integer.MAX_VALUE) {
throw new IOException("File is too large!");
}
byte[] bytes = new byte[(int) length];
int offset = 0;
int numRead = 0;
while (offset < bytes.length && (numRead = in.read(bytes, offset, bytes.length - offset)) >= 0) {
offset += numRead;
}
if (offset < bytes.length) {
throw new IOException("Could not completely read file " + fileFileName.getName());
}
in.close();

Max value for array index is Integer.MAX_INT - it's around 2Gb (2^31 / 2 147 483 647).
Your input stream can be bigger than 2Gb, so you have to process data in chunks, sorry.
InputStream is;
final byte[] buffer = new byte[512 * 1024 * 1024]; // 512Mb
while(true) {
final int read = is.read(buffer);
if ( read < 0 ) {
break;
}
// do processing
}

Assigning to a byte array in Java

I have a byte array I want to assign as follows:
First byte specifies the length of the string: (byte)string.length()
2nd - Last bytes contain string data from string.getBytes()
Other than using a for loop, is there a quick way to initialize a byte array using bytes from two different variables?

You can use System.arrayCopy() to copy your bytes:
String x = "xx";
byte[] out = new byte[x.getBytes().length()+1];
out[0] = (byte) (0xFF & x.getBytes().length());
System.arraycopy(x.getBytes(), 0, out, 1, x.length());
Though using something like a ByteArrayOutputStream or a ByteBuffer like other people suggested is probably a cleaner approach and will be better for your in the long run :-)

How about ByteBuffer ?
Example :
ByteBuffer bb = ByteBuffer.allocate(string.getBytes().length +1 );
bb.put((byte) string.length());
bb.put(string.getBytes());

While ByteBuffer is generally the best way to build up byte arrays, given the OP's goals I think the following will be more robust:
public static void main(String[] argv)
throws Exception
{
String s = "any string up to 64k long";
ByteArrayOutputStream bos = new ByteArrayOutputStream();
DataOutputStream out = new DataOutputStream(bos);
out.writeUTF(s);
out.close();
byte[] bytes = bos.toByteArray();
ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
DataInputStream in = new DataInputStream(bis);
String s2 = in.readUTF();
}

How about ByteArrayOutputStream?