In the following java code-snippet you'll see this line packetLengthMax += bytes.toByteArray()[43];
My question is: How does this work?
byte[] dataBuffer = new byte[265];
int packetLength = 0;
int packetLengthMax = 44;
ByteArrayOutputStream bytes = new ByteArrayOutputStream();
DataOutputStream outMessage = new DataOutputStream(bytes);
/* Client = Socket*/
DataInputStream clientIn = new DataInputStream(Client.getInputStream());
while (packetLength < packetLengthMax) {
packetLength += clientIn.read(dataBuffer);
outMessage.write(dataBuffer);
if (packetLength >= 43) {
packetLengthMax += bytes.toByteArray()[43];
}
}
My explanation:
First a socket (Client) is passed to the code. Then it does the setup of all variables. In the while loop, it reads all data that comes from the socket. Then it also writes this data to the DataOutputStream.
But in the if statement - it adds a byte array to an integer.
How does it work? I don't get that point. Thank you for helping!
It's not adding the whole byte array, it's just adding the byte at position 43. (i.e. the 44th byte in the array).
Related
I am new to the Java I/O so please help.
I am trying to process a large file(e.g. a pdf file of 50mb) using the apache commons library.
At first I try:
byte[] bytes = FileUtils.readFileToByteArray(file);
String encodeBase64String = Base64.encodeBase64String(bytes);
byte[] decoded = Base64.decodeBase64(encodeBase64String);
But knowing that the
FileUtils.readFileToByteArray in org.apache.commons.io will load the whole file into memory, I try to use BufferedInputStream to read the file piece by piece:
BufferedInputStream bis = new BufferedInputStream(inputStream);
StringBuilder pdfStringBuilder = new StringBuilder();
int byteArraySize = 10;
byte[] tempByteArray = new byte[byteArraySize];
while (bis.available() > 0) {
if (bis.available() < byteArraySize) { // reaching the end of file
tempByteArray = new byte[bis.available()];
}
int len = Math.min(bis.available(), byteArraySize);
read = bis.read(tempByteArray, 0, len);
if (read != -1) {
pdfStringBuilder.append(Base64.encodeBase64String(tempByteArray));
} else {
System.err.println("End of file reached.");
}
}
byte[] bytes = Base64.decodeBase64(pdfStringBuilder.toString());
However, the 2 decoded bytes array don't look quite the same... ... In fact, the only give 10 bytes, which is my temp array size... ...
Can anyone please help:
what am I doing it wrong to read the file piece by piece?
why is the decoded byte array only returns 10 bytes in the 2nd solution?
Thanks in advance:)
After some digging, it turns out that the byte array's size has to be multiple of 3 in order to avoid padding. After using a temp array size with multiple of 3, the program is able to go through.
I simply change
int byteArraySize = 10;
to be
int byteArraySize = 1024 * 3;
I want to transfer the type double[] over a network, and then somehow manage to transfer it back into a double[] in the receiving side. I am not entirely sure how to do this. I tried to convert the String received to a char[] and then parse all the chars to a double[]. However this did not work, the double had different data. I need to do this to make a network protocol for opencv, to transfer Mat's easily.
So this is how the data gets sent:
private void send_info(int row,int col, double[] data) {
//Convert data to String, separated by : to indicate change
//char[] sendit = data.toString().toCharArray();
out.println("INF:ROW:"+row+":COL"+":"+col+":"+data);
}
And this is how it is received:
private void setInfo(String input) {
input = input.trim();
input=input.replace("INF:","");
String inputs[] = input.split(":");
System.out.println(inputs[1]);
int row = Integer.parseInt(inputs[1]);
int col = Integer.parseInt(inputs[3]);
//double[] data = magic(inputs[4]);
// What I need ^
frame.put(row,col,data);
}
Don't convert them at all. Waste of time and space. Just do it directly. To send double[] doubles:
DataOutputStream dos = new DataOutputStream(new BufferedOutputStream(socket.getOutputStream()));
dos.writeInt(doubles.length); // send the array length
for (d : doubles)
{
dos.writeDouble(d);
}
dos.flush();
To read:
DataInputStream din = new DataInputStream(new BufferedInputStream(socket.getInputStream()));
double[] doubles = new double[dis.readInt()];
for (int i = i; i < doubles.length; i++)
{
doubles[i] = dis.readDouble();
}
Or you can use ObjectOutputStream.writeObject() and ObjectInputStream.readObject() to write and read the entire array at once. Or you can use NIO and DoubleBuffer: left as an exercise for the reader.
As an addendum to EJP’s answer, here an NIO solution:
sending
try(SocketChannel ch=SocketChannel.open(
new InetSocketAddress(InetAddress.getLocalHost(), 12345))) {
ByteBuffer buf=ByteBuffer.allocateDirect(doubles.length*Double.BYTES+Integer.BYTES);
buf.putInt(doubles.length).asDoubleBuffer().put(doubles);
buf.clear();
while(buf.hasRemaining()) ch.write(buf);
}
receiving
final int DEFAULT_BUFFER_SIZE = 4096;
try(ServerSocketChannel ss=ServerSocketChannel.open()
.bind(new InetSocketAddress(InetAddress.getLocalHost(), 12345));
SocketChannel ch=ss.accept()) {
ByteBuffer bb=ByteBuffer.allocateDirect(DEFAULT_BUFFER_SIZE);
bb.limit(Integer.BYTES);
while(bb.hasRemaining()) if(ch.read(bb)<0) throw new EOFException();
bb.flip();
int size=bb.getInt(), byteSize=size*Double.BYTES;
if(bb.capacity()<byteSize) bb=ByteBuffer.allocateDirect(byteSize);
else bb.clear().limit(byteSize);
while(bb.hasRemaining()) if(ch.read(bb)<0) throw new EOFException();
double[] doubles=new double[size];
bb.flip();
bb.asDoubleBuffer().get(doubles);
return doubles;
}
It’s obvious that the buffer management gets more complicated on the receiving side due to the double array length which is not known beforehand.
If we want to reduce the number of transfers, i.e. avoid a distinct I/O operation just for the first four bytes, the method gets even more complicated:
final int DEFAULT_BUFFER_SIZE = 4096;
try(ServerSocketChannel ss=ServerSocketChannel.open()
.bind(new InetSocketAddress(InetAddress.getLocalHost(), 12345));
SocketChannel ch=ss.accept()) {
ByteBuffer bb=ByteBuffer.allocateDirect(DEFAULT_BUFFER_SIZE);
while(bb.position()<4) if(ch.read(bb)<0) throw new EOFException();
bb.flip();
int size=bb.getInt(), byteSize=size*Double.BYTES;
if(bb.remaining()<byteSize) {
if(bb.capacity()<byteSize) bb=ByteBuffer.allocateDirect(byteSize).put(bb);
else bb.compact().limit(byteSize);
while(bb.hasRemaining()) if(ch.read(bb)<0) throw new EOFException();
bb.flip();
}
else bb.limit(bb.position()+byteSize);
double[] doubles=new double[size];
bb.asDoubleBuffer().get(doubles);
return doubles;
}
But note that the format is identical to the one created with the DataOutputStream in EJP’s solution, so you could combine, e.g. the NIO sending code with the old I/O receiving code…
I had one question.
Is there library or etc to compose int & strings to byte array ?
Like :
byte temparray[] = new byte[10];
int a = 10;
int b = 10;
temparray << new String("12") << a << b;
Thanks.
edit
byte[] buffer = new byte[649];
byte[] charname = this.getName().getBytes();
System.arraycopy(charname, 0 , buffer, 0, charname.length);
for(int i=0;i<16;i++) //mystery crs 16 zeros
{
buffer[i+17] = (byte)0x30;
}
buffer[34] = this.faction;
if(this.characterClass == 2)
{
buffer[40] = 2;
} else
{
buffer[40] = 1;
}
System.arraycopy(BitTools.shortToByteArray(face), 0, buffer, 42, 2);
buffer[44] = 1;
buffer[48] = (byte)this.characterClass; //class byte
buffer[52] = 2; explanation yet
buffer[54] = (byte)this.getLevel();
This is an example of my packet generator and i wanted to simplify it, but in packet i use only shorts, ints and strings.
java.io.ByteArrayOutputStream is a stream implementation that collects content on an internal byte array, and you can wrap it in a java.io.OutputStreamWriter to write character content to it.
ByteArrayOutputStream byteOut = new ByteArrayOutputStream();
Writer out = new OutputStreamWriter(byteOut, "UTF-8"); // Uses UTF-8 encoding
out.write("12");
out.write(10);
out.write(10);
out.close();
byte[] bytes = byteOut.toByteArray();
After this, bytes.length is just long enough for the bytes written to byteOut.
Yes. See java.io.ByteArrayOutputStream. Note that you can wrap this stream to support writing of other types like String: PrintWriter pw = new PrintWriter(yourByteArrayOutputStream); pw.print("Hello");
And afterwards use yourByteArrayOutputStream.toByteArray(); to get the byte array.
http://docs.oracle.com/javase/7/docs/api/
Integer.byteValue();
Double.byteValue();
String.getBytes();
// etc.
Take a look at String#getBytes and ByteBuffer. Charsets and byte order might be important depending on your use case.
I am developing an Android application, it is going to fetch a big chunk of JSON data in stream. Calling the web service is OK, but I have a little problem. In my old version I was using Gson for reading the stream then I've tried to insert data to database, it was OK without any problem except performance. So I tried to change approach of loading data, I am trying to read data to char[] first then insert them to database.
This is my new code:
HttpEntity responseEntity = response.getEntity();
final int contentLength = (int) responseEntity.getContentLength();
InputStream stream = responseEntity.getContent();
InputStreamReader reader = new InputStreamReader(stream);
int readCount = 10 * 1024;
int hasread = 0;
char[] buffer = new char[contentLength];
int mustWrite = 0;
int hasread2 = 0;
while (hasread < contentLength) {
// problem is here
hasread += reader.read(buffer, hasread, contentLength - hasread);
}
Reader reader2 = new CharArrayReader(buffer);
The problem is that the reader starts reading correctly but at near of the end of stream, the hasread variable value decreases (by 1) instead of increasing. Very strange to me, and then the while loop never finishes. What's wrong with this code?
You should use a fixed size for the buffer, not the size of whole data (the contentLength). And an important note: the length of a char[] array is different to byte[] array's. The char data type is a single 16-bit Unicode character. While the byte data type is an 8-bit signed two's complement integer.
Also your while loop is wrong, you can fix it as:
import java.io.BufferedInputStream;
private static final int BUF_SIZE = 10 * 1024;
// ...
HttpEntity responseEntity = response.getEntity();
final int contentLength = (int) responseEntity.getContentLength();
InputStream stream = responseEntity.getContent();
BufferedInputStream reader = new BufferedInputStream(stream);
int hasread = 0;
byte[] buffer = new byte[BUF_SIZE];
while ((hasread = reader.read(buffer, 0, BUF_SIZE)) > 0) {
// For example, convert the buffer to a String
String data = new String(buffer, 0, hasread, "UTF-8");
}
Make sure to use your own charset ("UTF-8", "UTF-16"…).
With reference to my previous question.
I have made the program with the following approach:
The program first reads 2k of data from the file and stores it into a byte array.
Then the data to be added to each packet is also stored in an array and both are added to an array list.
The array list is then written to an output stream for the file.
The Code is here:
File bin=chooser.getSelectedFile();
int filesize=(int)bin.length();
int pcount=filesize/2048;
byte[] file=new byte[filesize];
byte[] meta=new byte[12];
int arraysize=pcount*12+filesize;
byte[] rootfile=new byte[46];
ArrayList al = new ArrayList();
String root;
prbar.setVisible(true);
int mark=0;
String metas;
try{
FileInputStream fis=new FileInputStream(bin);
FileOutputStream fos=new FileOutputStream(bin.getName().replace(".bin", ".xyz"));
ObjectOutputStream os=new ObjectOutputStream(fos);
root="46kb"+"5678"+"0000"+pcount+"MYBOX"+"13"+"S208";
rootfile=root.getBytes();
for(int n=0;n<=pcount;n++)
{
fis.read(file, 0, 2048);
mark=mark+2048;
int v=(mark/filesize)*100;
prbar.setValue(v);
metas="02KB"+"1234"+n;
meta=metas.getBytes();
al.add(rootfile);
al.add(meta);
al.add(file);
}
os.writeObject(al.toArray());
}
catch(Exception ex){
erlabel.setText(ex.getMessage());
}
The program runs without any errors but the file is not created correctly.
Either the approach is wrong or the code.
Please Help
You appear to be writing your own binary format but you are using ObjectOutputStream which has it own header. writeObject write an Object not data in a manner that lets a Java process deserialize that object e.g. with it class hierarchy and field names.
For binary, I suggest you use a plain DataOutputStream with a BufferedOutputStream which will be more efficient and do what you want.
I also suggest you write the data as you generate it rather than using an ArrayList. This will use less memory, make the code simpler and be faster.
I would write the code more like this
File bin = chooser.getSelectedFile();
int filesize = (int) bin.length();
int pcount = (filesize + 2048 - 1) / 2048;
byte[] file = new byte[2048];
FileInputStream fis = new FileInputStream(bin);
String name2 = bin.getName().replace(".bin", ".xyz");
OutputStream os = new BufferedOutputStream(new FileOutputStream(name2));
byte[] rootfile = ("46kb" + "5678" + "0000" + pcount + "MYBOX" + "13" + "S208").getBytes("UTF-8");
for (int n = 0; n < pcount; n++) {
os.write(rootfile);
byte[] metas = ("02KB" + "1234" + n).getBytes("UTF-8");
os.write(metas);
int len = fis.read(file);
os.write(file, 0, len);
int percent = 100 * n / pcount;
prbar.setValue(percent);
}
ow.close();
With the smallest thing first:
int v=(mark/filesize)*100;
Is using integer division yielding always 0 I think.
int v = mark * 100 / filesize;
The byte[] object (file for instance) is created once and many times added to the list.
You get n copies of the last overwrite.