Develop Reference

How can I optimize streaming a .osgb file into a ByteString efficiently?

I am sending a .osgb file in a Google Protobuf message which requires a Byte String. It is encoded "ISO-5589-1". In python, I can simply open(file_name, "r").read(). In Java, I have very noob-ishly created:
String model;
ByteString modelBytes = null;
try {
FileInputStream fis = new FileInputStream( filename );
DataInputStream dis = new DataInputStream(fis);
byte[] bytes = new byte[dis.available()];
if ( dis.available() != 0 ) {
dis.readFully(bytes);
}
model = new String(bytes, "ISO-8859-1");
modelBytes = ByteString.copyFrom(model, "ISO-8859-1");
}
I'm not looking to code-golf this excerpt, but I feel as though I have possibly redundant or extra code that is genuinely not needed. It feels as though I should be able to just convert the data stream immediately into a ByteString and not worry about the encoding, but I'm not familiar enough with it.
I am very inexperienced with Java so I appreciate any assistance. Thanks.

Java FileInput/OutputStream and ByteBuffer

How can I use ByteBuffer to write an integer 1432 into a file written by FileOutputStream. Since 1432 takes more than one byte, we cannot use write() method.
Furthermore, how can we get back the integer when we use FileInputStream read() method later?
I tried to use:
int i = 1432;
byte[] bytesi = ByteBuffer.allocate(4).putInt(i).array();
fileOS.write(bytesi);
But when read the file:
int e = fileIS.read();
System.out.println(e);
int e1 = fileIS.read();
System.out.println(e1);
int e2 = fileIS.read();
System.out.println(e2);
int e3 = fileIS.read();
System.out.println(e3);
I get outputs like:
255
132
201
255

Since you're using ByteBuffer to generate bytes from integer, you can use it for inverse convertion too
byte[] bytes = new byte[4];
fis.read(bytes);
int x = ByteBuffer.wrap(bytes).getInt();

XY problem. You don't need ByteBuffer, you need to write an int to a file in binary. DataOutputStream has all the methods you need, and it already works with FileOutputStream. Similarly DataInputStream already works with FileInputStream.

You can wrap your FileInputStream and FileOutputStream into DataInputStream / DataOutputStream to have helper methods for various data types like readInt / writeInt
http://docs.oracle.com/javase/7/docs/api/java/io/DataOutputStream.html
Example of usage: http://www.tutorialspoint.com/java/io/dataoutputstream_writeint.htm

Same basic answer as above, using the ByteBuffer function but not messing around with your own byte[].
ByteBuffer bytesIn = ByteBuffer.allocate(4);
fileIS.read(bytesIn.array());
int e = bytesIn.getInt();
You could/should do it this way for writing the bytes as well, e.g.
int i = 1432;
ByteBuffer bytesOut = ByteBuffer.allocate(4).putInt(test);
outFile.write(bytesOut.array());

ByteArrayOutputStream for shorts instead of bytes

I have used the ByteArrayOutputStream which is very useful but for my needs the limits are too great (ie I am dealing with numbers in the range +-32,768)
Here is the code where I would use it:
ByteArrayOutputStream leftStream = new ByteArrayOutputStream();
ByteArrayOutputStream rightStream = new ByteArrayOutputStream();
while (din.read(temp, 0, 4) != -1) {
if (decodedFormat.getChannels() == 2) {
leftStream.write(temp[1] * 256 + temp[0]);
rightStream.write(temp[3] * 256 + temp[2]);
}
}
byte[] left = leftStream.toByteArray();
byte[] right = rightStream.toByteArray();
However the ByteArrayInputStream does not allow for values over 127 or under 128. Is there an equivalent ShortArrayOutputStream that allows shorts? Or do I have to make one myself somehow?

Yes, Use the DataOutputStream:
ByteArrayOuputStream bos = new ByteArrayOutputStream();
DataOutputStream dos = new DataOutputStream(bos);
dos.writeShort(val);
This works even on embedded java devices with low java version 1.3
To read in use:
ByteArrayInputStream and DataInputStream dis, and dis.readShort():
...
byte[] bytes = bos.toByteArray();
ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
DataInputStream dis = new DataInputStream(bis);
short val = dis.readShort();

You can wrap the ByteArrayOutputStream into DataOutputStream:
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
DataOutputStream dataOutputStream = new DataOutputStream(byteArrayOutputStream);
dataOutputStream.writeShort(someShortValue);

What do you want to do?
Write out short data to a byte array?
Then wrap your byte array output stream with a DataOutputStream which has methods to writeShort(), writeInt() etc. Warning. I think the endian of DataOutputStream is BIG endian so if you want to use little endian you either have to write it yourself or use the other option:
ByteArrayOutputStream byteOut = new ByteArrayOutputStream();
DataOutputStream dataout = new DataOutputStream(byteOut)
dataout.writeShort(shortValue);
Write a short[]
the easiest is to create a ByteBuffer, then use the asShortBuffer() method to view it as a ShortBuffer. the ShortBuffer has a put(short) and put(short[]);
If you want to write out the short data in Little endian, ByteBuffer has a method asOrder(ByteOrder) which can change the endian of the data it is reading or writing.
//NOTE length should be 2* num shorts since we allocate in bytes
ByteBuffer buf = ByteBuffer.allocate(length);
ShortBuffer shortBuf = buf.asShortBuffer();
shortBuf.put(shortValue);
shortBuf.put(shortArray);
getting data out from buffers is annoying. There are optional array() methods but not all buffer impelmentations support them so you have to do the following:
//once all data written to buffer
shortBuf.flip();
short[] dataOut = new short[shortBuf.remaining()];
shortBuf.get(dataOut);
Combination of both to use a ShortBuffer on an unknown input size
If you don't know how many bytes you will write, and you don't have a reasonable, max length, then you may need to do a combination of both options. First, use option #1 to dynamically grow a byte buffer by writing shorts to it. Then convert the byte[] to a short[] with the ShortBuffer.
ByteArrayOutputStream byteOut = new ByteArrayOutputStream();
DataOutputStream dataout = new DataOutputStream(byteOut)
dataout.writeShort(shortValue);
...
ShortBuffer buf =ByteBuffer.wrap(byteOut.toByteArray())
.asShortBuffer();
int length = buf.remaining();
short[] asShorts = new short[length];
buf.get(asShorts);
It's not pretty and uses 2x the memory since you make a copy of the array.

Create a ObjectOutputStream. This has a writeShort (http://docs.oracle.com/javase/7/docs/api/java/io/ObjectOutputStream.html#writeShort(int)) method.
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(bos);
oos.writeShort(123);
...

Sending big file using FileInputStream/ObjectOutputStream

I need help on my homework, any help will be much appreciated. I can send small files without a problem. But when i try to send let’s say a 1GB file byte array sends OutOfMemoryError so i need a better solution to send file from server to client. How can i improve this code and send big files, please help me.
Server Code:
FileInputStream fis = new FileInputStream(file);
byte[] fileByte = new byte[fis.available()]; //This causes the problem.
bytesRead = fis.read(fileByte);
oos = new ObjectOutputStream(sock.getOutputStream());
oos.writeObject(fileByte);
Client Code:
ois = new ObjectInputStream(sock.getInputStream());
byte[] file = (byte[]) ois.readObject();
fos = new FileOutputStream(file);
fos.write(file);

Don't read the whole file into memory, use a small buffer and write while you are reading the file:
BufferedOutputStream bos = new BufferedOutputStream(sock.getOutputStream())
File file = new File("asd");
FileInputStream fis = new FileInputStream(file);
BufferedInputStream bis = new BufferedInputStream(fis);
byte[] buffer = new byte[1024*1024*10];
int n = -1;
while((n = bis.read(buffer))!=-1) {
bos.write(buffer,0,n):
}
Use Buffered* to optimize the writing and reading from Streams

Just split the array to smaller chunks so that you don't need to allocate any big array.
For example you could split the array into 16Kb chunks, eg new byte[16384] and send them one by one. On the receiving side you would have to wait until a chunk can be fully read and then store them somewhere and start with next chunk.
But if you are not able to allocate a whole array of the size you need on server side you won't be able to store all the data that you are going to receive anyway.
You could also compress the data before sending it to save bandwidth (and time), take a look at ZipOutputStream and ZipInputStream.

Here's how I solved it:
Client Code:
bis=new BufferedInputStream(sock.getInputStream());
fos = new FileOutputStream(file);
int n;
byte[] buffer = new byte[8192];
while ((n = bis.read(buffer)) > 0){
fos.write(buffer, 0, n);}
Server Code:
bos= new BufferedOutputStream(sock.getOutputStream());
FileInputStream fis = new FileInputStream(file);
BufferedInputStream bis = new BufferedInputStream(fis);
int n=-1;
byte[] buffer = new byte[8192];
while((n = bis.read(buffer))>-1)
bos.write(buffer,0,n);

Depending on whether or not you have to write the code yourself, there are existing libraries which solve this problem, e.g. rmiio. If you are not using RMI, just plain java serialization, you can use the DirectRemoteInputStream, which is kind of like a Serializable InputStream. (this library also has support for things like auto-magically compressing the data).
Actually, if you are only sending file data, you would be better off ditching the Object streams and use DataInput/DataOutput streams. first write an integer indicating the file length, then copy the bytes directly to the stream. on the receiving side, read the integer file length, then read exactly that many bytes.
when you copy the data between streams, use a small, fixed size byte[] to move chunks of data between the input and output streams in a loop. there are numerous examples of how to do this correctly available online (e.g. #ErikFWinter's answer).

Assigning to a byte array in Java

I have a byte array I want to assign as follows:
First byte specifies the length of the string: (byte)string.length()
2nd - Last bytes contain string data from string.getBytes()
Other than using a for loop, is there a quick way to initialize a byte array using bytes from two different variables?

You can use System.arrayCopy() to copy your bytes:
String x = "xx";
byte[] out = new byte[x.getBytes().length()+1];
out[0] = (byte) (0xFF & x.getBytes().length());
System.arraycopy(x.getBytes(), 0, out, 1, x.length());
Though using something like a ByteArrayOutputStream or a ByteBuffer like other people suggested is probably a cleaner approach and will be better for your in the long run :-)

How about ByteBuffer ?
Example :
ByteBuffer bb = ByteBuffer.allocate(string.getBytes().length +1 );
bb.put((byte) string.length());
bb.put(string.getBytes());

While ByteBuffer is generally the best way to build up byte arrays, given the OP's goals I think the following will be more robust:
public static void main(String[] argv)
throws Exception
{
String s = "any string up to 64k long";
ByteArrayOutputStream bos = new ByteArrayOutputStream();
DataOutputStream out = new DataOutputStream(bos);
out.writeUTF(s);
out.close();
byte[] bytes = bos.toByteArray();
ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
DataInputStream in = new DataInputStream(bis);
String s2 = in.readUTF();
}

How about ByteArrayOutputStream?