I am developing an Android application, it is going to fetch a big chunk of JSON data in stream. Calling the web service is OK, but I have a little problem. In my old version I was using Gson for reading the stream then I've tried to insert data to database, it was OK without any problem except performance. So I tried to change approach of loading data, I am trying to read data to char[] first then insert them to database.
This is my new code:
HttpEntity responseEntity = response.getEntity();
final int contentLength = (int) responseEntity.getContentLength();
InputStream stream = responseEntity.getContent();
InputStreamReader reader = new InputStreamReader(stream);
int readCount = 10 * 1024;
int hasread = 0;
char[] buffer = new char[contentLength];
int mustWrite = 0;
int hasread2 = 0;
while (hasread < contentLength) {
// problem is here
hasread += reader.read(buffer, hasread, contentLength - hasread);
}
Reader reader2 = new CharArrayReader(buffer);
The problem is that the reader starts reading correctly but at near of the end of stream, the hasread variable value decreases (by 1) instead of increasing. Very strange to me, and then the while loop never finishes. What's wrong with this code?
You should use a fixed size for the buffer, not the size of whole data (the contentLength). And an important note: the length of a char[] array is different to byte[] array's. The char data type is a single 16-bit Unicode character. While the byte data type is an 8-bit signed two's complement integer.
Also your while loop is wrong, you can fix it as:
import java.io.BufferedInputStream;
private static final int BUF_SIZE = 10 * 1024;
// ...
HttpEntity responseEntity = response.getEntity();
final int contentLength = (int) responseEntity.getContentLength();
InputStream stream = responseEntity.getContent();
BufferedInputStream reader = new BufferedInputStream(stream);
int hasread = 0;
byte[] buffer = new byte[BUF_SIZE];
while ((hasread = reader.read(buffer, 0, BUF_SIZE)) > 0) {
// For example, convert the buffer to a String
String data = new String(buffer, 0, hasread, "UTF-8");
}
Make sure to use your own charset ("UTF-8", "UTF-16"…).
Related
I am new to the Java I/O so please help.
I am trying to process a large file(e.g. a pdf file of 50mb) using the apache commons library.
At first I try:
byte[] bytes = FileUtils.readFileToByteArray(file);
String encodeBase64String = Base64.encodeBase64String(bytes);
byte[] decoded = Base64.decodeBase64(encodeBase64String);
But knowing that the
FileUtils.readFileToByteArray in org.apache.commons.io will load the whole file into memory, I try to use BufferedInputStream to read the file piece by piece:
BufferedInputStream bis = new BufferedInputStream(inputStream);
StringBuilder pdfStringBuilder = new StringBuilder();
int byteArraySize = 10;
byte[] tempByteArray = new byte[byteArraySize];
while (bis.available() > 0) {
if (bis.available() < byteArraySize) { // reaching the end of file
tempByteArray = new byte[bis.available()];
}
int len = Math.min(bis.available(), byteArraySize);
read = bis.read(tempByteArray, 0, len);
if (read != -1) {
pdfStringBuilder.append(Base64.encodeBase64String(tempByteArray));
} else {
System.err.println("End of file reached.");
}
}
byte[] bytes = Base64.decodeBase64(pdfStringBuilder.toString());
However, the 2 decoded bytes array don't look quite the same... ... In fact, the only give 10 bytes, which is my temp array size... ...
Can anyone please help:
what am I doing it wrong to read the file piece by piece?
why is the decoded byte array only returns 10 bytes in the 2nd solution?
Thanks in advance:)
After some digging, it turns out that the byte array's size has to be multiple of 3 in order to avoid padding. After using a temp array size with multiple of 3, the program is able to go through.
I simply change
int byteArraySize = 10;
to be
int byteArraySize = 1024 * 3;
In the following java code-snippet you'll see this line packetLengthMax += bytes.toByteArray()[43];
My question is: How does this work?
byte[] dataBuffer = new byte[265];
int packetLength = 0;
int packetLengthMax = 44;
ByteArrayOutputStream bytes = new ByteArrayOutputStream();
DataOutputStream outMessage = new DataOutputStream(bytes);
/* Client = Socket*/
DataInputStream clientIn = new DataInputStream(Client.getInputStream());
while (packetLength < packetLengthMax) {
packetLength += clientIn.read(dataBuffer);
outMessage.write(dataBuffer);
if (packetLength >= 43) {
packetLengthMax += bytes.toByteArray()[43];
}
}
My explanation:
First a socket (Client) is passed to the code. Then it does the setup of all variables. In the while loop, it reads all data that comes from the socket. Then it also writes this data to the DataOutputStream.
But in the if statement - it adds a byte array to an integer.
How does it work? I don't get that point. Thank you for helping!
It's not adding the whole byte array, it's just adding the byte at position 43. (i.e. the 44th byte in the array).
I'm trying to read first line from socket stream with BufferedReader from BufferedInputStream, it reads the first line(1), this is size of some contents(2) in this content i have the size of another content(3)
Reads correctly... ( with BufferedReader, _bin.readLine() )
Reads correctly too... ( with _in.read(byte[] b) )
Won't read, seems there's more content than my size read in (2)
I think problem is that I'm trying to read using BufferedReader and then BufferedInputStream... can anyone help me ?
public HashMap<String, byte[]> readHead() throws IOException {
JSONObject json;
try {
HashMap<String, byte[]> map = new HashMap<>();
System.out.println("reading header");
int headersize = Integer.parseInt(_bin.readLine());
byte[] parsable = new byte[headersize];
_in.read(parsable);
json = new JSONObject(new String(parsable));
map.put("id", lTob(json.getLong(SagConstants.KEY_ID)));
map.put("length", iTob(json.getInt(SagConstants.KEY_SIZE)));
map.put("type", new byte[]{(byte)json.getInt(SagConstants.KEY_TYPE)});
return map;
} catch(SocketException | JSONException e) {
_exception = e.getMessage();
_error_code = SagConstants.ERROR_OCCOURED_EXCEPTION;
return null;
}
}
sorry for bad english and for bad explanation, i tried to explain my problem, hope you understand
file format is so:
size1
{json, length is given size1, there is size2 given}
{second json, length is size2}
_in is BufferedInputStream();
_bin is BufferedReader(_in);
with _bin, i read first line (size1) and convert to integer
with _in, i read next data, where is size2 and length of this data is size1
then im trying to read the last data, its size is size2
something like this:
byte[] b = new byte[secondSize];
_in.read(b);
and nothing happens here, program is paused...
can't work with BufferedInputStream and BufferedReader together
That's correct. If you use any buffered stream or reader on a socket [or indeed any data source], you can't use any other stream or reader with it whatsoever. Data will get 'lost', that is to say read-ahead, in the buffer of the buffered stream or reader, and will not be available to the other stream/reader.
You need to rethink your design.
You create one BufferedReader _bin and BufferedInputStream _in and read a file both of them, but their cursor position is different so second read start from beginning because you use 2 object to read it. You should read size1 with _in too.
int headersize = Integer.parseInt(readLine(_in));
byte[] parsable = new byte[headersize];
_in.read(parsable);
Use below readLine to read all data with BufferedInputStream.
private final static byte NL = 10;// new line
private final static byte EOF = -1;// end of file
private final static byte EOL = 0;// end of line
private static String readLine(BufferedInputStream reader,
String accumulator) throws IOException {
byte[] container = new byte[1];
reader.read(container);
byte byteRead = container[0];
if (byteRead == NL || byteRead == EOL || byteRead == EOF) {
return accumulator;
}
String input = "";
input = new String(container, 0, 1);
accumulator = accumulator + input;
return readLine(reader, accumulator);
}
I had one question.
Is there library or etc to compose int & strings to byte array ?
Like :
byte temparray[] = new byte[10];
int a = 10;
int b = 10;
temparray << new String("12") << a << b;
Thanks.
edit
byte[] buffer = new byte[649];
byte[] charname = this.getName().getBytes();
System.arraycopy(charname, 0 , buffer, 0, charname.length);
for(int i=0;i<16;i++) //mystery crs 16 zeros
{
buffer[i+17] = (byte)0x30;
}
buffer[34] = this.faction;
if(this.characterClass == 2)
{
buffer[40] = 2;
} else
{
buffer[40] = 1;
}
System.arraycopy(BitTools.shortToByteArray(face), 0, buffer, 42, 2);
buffer[44] = 1;
buffer[48] = (byte)this.characterClass; //class byte
buffer[52] = 2; explanation yet
buffer[54] = (byte)this.getLevel();
This is an example of my packet generator and i wanted to simplify it, but in packet i use only shorts, ints and strings.
java.io.ByteArrayOutputStream is a stream implementation that collects content on an internal byte array, and you can wrap it in a java.io.OutputStreamWriter to write character content to it.
ByteArrayOutputStream byteOut = new ByteArrayOutputStream();
Writer out = new OutputStreamWriter(byteOut, "UTF-8"); // Uses UTF-8 encoding
out.write("12");
out.write(10);
out.write(10);
out.close();
byte[] bytes = byteOut.toByteArray();
After this, bytes.length is just long enough for the bytes written to byteOut.
Yes. See java.io.ByteArrayOutputStream. Note that you can wrap this stream to support writing of other types like String: PrintWriter pw = new PrintWriter(yourByteArrayOutputStream); pw.print("Hello");
And afterwards use yourByteArrayOutputStream.toByteArray(); to get the byte array.
http://docs.oracle.com/javase/7/docs/api/
Integer.byteValue();
Double.byteValue();
String.getBytes();
// etc.
Take a look at String#getBytes and ByteBuffer. Charsets and byte order might be important depending on your use case.
get data from socket (buffer)
BufferedReader is = new BufferedReader(new InputStreamReader(download.getInputStream()));
data decompression
Inflater decompresser = new Inflater();
decompresser.setInput(buffer.toString().getBytes());
byte[] dataBytes = new byte[8388608];
int resultLength = decompresser.inflate(dataBytes);
decompresser.end();
System.out.println("decompressed" + new String(dataBytes, 0, resultLength)+ " RESULTLENGHT " +resultLength);
Sending 1000 bytes, compressed ZLIB and turns (800-900 bytes)
But I do not know what exactly the size of sending. I need to read from socket 1 byte at a time and immediately unpack it until the total size of decompressed data becomes equal to 1000 bytes.
To read a byte at a time I do so:
StringBuilder buffer = new StringBuilder();
StringBuilder unpackbuffer = new StringBuilder();
do buffer.append((char)is.read());
while(buffer.charAt(buffer.length()-1) != '|' && (byte)buffer.charAt(buffer.length()-1) != -1);
How do I fill in this cycle unpackbuffer? and check its size?
Sorry, I hope my question is clear.
OK, if you can't prepend the length to the data you're sending and then read in at the other end (which would obviously be the simplest ideal solution if you were able to design the protocol to allow this), then you can compress the stream 'byte by byte'. The trick is to create a 1-byte buffer as the input buffer. The code then looks as follows:
Inflater infl = new Inflater();
byte[] buf = new byte[1];
byte[] outputBuf = new byte[512];
while (!infl.finished()) {
while (infl.needsInput()) {
buf[0] = ...next byte from stream...
infl.setInput(buf);
}
int noUnc = infl.inflate(outputBuf);
// the first "noUnc" bytes of outputBuf contain the next bit of data
}