Develop Reference

When downloading a file from a URL, why do we read into a byte array?

Why do we read into a byte array when downloading a file from a URL? In the below code a byte array ('data') is created which is allocated a number of "1024" and is passed as a parameter in the below piece of code
while ((x = in.read(data, 0, 1024)) >= 0)
Can you please explain what "reading" into a byte array means? Also, why were "0" and "1024" passed in as well?
This code was taken from Java getting download progress
URL url = new URL("http://downloads.sourceforge.net/project/bitcoin/Bitcoin/blockchain/bitcoin_blockchain_170000.zip");
HttpURLConnection httpConnection = (HttpURLConnection) (url.openConnection());
long completeFileSize = httpConnection.getContentLength();
java.io.BufferedInputStream in = new java.io.BufferedInputStream(httpConnection.getInputStream());
java.io.FileOutputStream fos = new java.io.FileOutputStream(
"package.zip");
java.io.BufferedOutputStream bout = new BufferedOutputStream(
fos, 1024);
byte[] data = new byte[1024];
long downloadedFileSize = 0;
int x = 0;
while ((x = in.read(data, 0, 1024)) >= 0) {
downloadedFileSize += x;

Can you please explain what "reading" into a byte array means?
When we read data into a byte array, we mean that we store the data from an input stream into an array for later use. We read the data into a byte array instead of a char array or an int array because it is binary data. It might be text or a picture or a video. At the end of the day, these are all binary data that we store in bytes.
Also, why were "0" and "1024" passed in as well?
The documentation for read() says it takes 3 arguments:
b - destination buffer.
off - offset at which to start storing bytes.
len - maximum number of bytes to read.
So the 0 is the "offset" where the read operation will start storing bytes. The 1024 is the number of bytes to read. These can be any sensible numbers as long as you don't try to read into a location past the end of the array.

Is it possible to peek HttpClient's org.apache.http.conn.EofSensorInputStream?

I am trying to peek at an input stream contents from HttpClient, up to 64k bytes.
The stream comes from an HttpGet, nothing unusual about it:
HttpGet requestGet = new HttpGet(encodedUrl);
HttpResponse httpResponse = httpClient.execute(requestGet);
int status = httpResponse.getStatusLine().getStatusCode();
if (status == HttpStatus.SC_OK) {
return httpResponse.getEntity().getContent();
}
The input stream it returns is of type org.apache.http.conn.EofSensorInputStream
Our use-case is such that we need to "peek" at the first (up to 64k) bytes of the input stream. I use an algorithm described here How do I peek at the first two bytes in an InputStream?
PushbackInputStream pis = new PushbackInputStream(inputStream, DEFAULT_PEEK_BUFFER_SIZE);
byte [] peekBytes = new byte[DEFAULT_PEEK_BUFFER_SIZE];
int read = pis.read(peekBytes);
if (read < DEFAULT_PEEK_BUFFER_SIZE) {
byte[] trimmed = new byte[read];
System.arraycopy(peekBytes, 0, trimmed, 0, read);
peekBytes = trimmed;
}
pis.unread(peekBytes);
When I use a ByteArrayInputStream, this works with no problem.
The Issue: When using the org.apache.http.conn.EofSensorInputStream I only get a small number of bytes at the beginning of the stream. usually around 400 bytes. When I expected up to 64k bytes.
I also tried using a BufferedInputStream where I read up to the first 64k bytes then call a .reset() but that doesn't work either. Same issue.
Why might this be? I do not think anything is closing the stream because if you call IOUtils.toString(inputStream) I do get all the content.

See InputStream#read(byte[] b,int off,int len) contract:
Reads up to len bytes of data from the input stream into an array of
bytes. An attempt is made to read as many as len bytes, but a
smaller number may be read. The number of bytes actually read is
returned as an integer
Instead of using this method, use IOUtils.read which reads until you get the number of bytes you requested in a loop.

Java Reading large files into byte array chunk by chunk

So I've been trying to make a small program that inputs a file into a byte array, then it will turn that byte array into hex, then binary. It will then play with the binary values (I haven't thought of what to do when I get to this stage) and then save it as a custom file.
I studied a lot of internet code and I can turn a file into a byte array and into hex, but the problem is I can't turn huge files into byte arrays (out of memory).
This is the code that is not a complete failure
public void rundis(Path pp) {
byte bb[] = null;
try {
bb = Files.readAllBytes(pp); //Files.toByteArray(pathhold);
System.out.println("byte array made");
} catch (Exception e) {
e.printStackTrace();
}
if (bb.length != 0 || bb != null) {
System.out.println("byte array filled");
//send to method to turn into hex
} else {
System.out.println("byte array NOT filled");
}
}
I know how the process should go, but I don't know how to code that properly.
The process if you are interested:
Input file using File
Read the chunk by chunk of the file into a byte array. Ex. each byte array record hold 600 bytes
Send that chunk to be turned into a Hex value --> Integer.tohexstring
Send that hex value chunk to be made into a binary value --> Integer.toBinarystring
Mess around with the Binary value
Save to custom file line by line
Problem:: I don't know how to turn a huge file into a byte array chunk by chunk to be processed.
Any and all help will be appreciated, thank you for reading :)

To chunk your input use a FileInputStream:
Path pp = FileSystems.getDefault().getPath("logs", "access.log");
final int BUFFER_SIZE = 1024*1024; //this is actually bytes
FileInputStream fis = new FileInputStream(pp.toFile());
byte[] buffer = new byte[BUFFER_SIZE];
int read = 0;
while( ( read = fis.read( buffer ) ) > 0 ){
// call your other methodes here...
}
fis.close();

To stream a file, you need to step away from Files.readAllBytes(). It's a nice utility for small files, but as you noticed not so much for large files.
In pseudocode it would look something like this:
while there are more bytes available
read some bytes
process those bytes
(write the result back to a file, if needed)
In Java, you can use a FileInputStream to read a file byte by byte or chunk by chunk. Lets say we want to write back our processed bytes. First we open the files:
FileInputStream is = new FileInputStream(new File("input.txt"));
FileOutputStream os = new FileOutputStream(new File("output.txt"));
We need the FileOutputStream to write back our results - we don't want to just drop our precious processed data, right? Next we need a buffer which holds a chunk of bytes:
byte[] buf = new byte[4096];
How many bytes is up to you, I kinda like chunks of 4096 bytes. Then we need to actually read some bytes
int read = is.read(buf);
this will read up to buf.length bytes and store them in buf. It will return the total bytes read. Then we process the bytes:
//Assuming the processing function looks like this:
//byte[] process(byte[] data, int bytes);
byte[] ret = process(buf, read);
process() in above example is your processing method. It takes in a byte-array, the number of bytes it should process and returns the result as byte-array.
Last, we write the result back to a file:
os.write(ret);
We have to execute this in a loop until there are no bytes left in the file, so lets write a loop for it:
int read = 0;
while((read = is.read(buf)) > 0) {
byte[] ret = process(buf, read);
os.write(ret);
}
and finally close the streams
is.close();
os.close();
And thats it. We processed the file in 4096-byte chunks and wrote the result back to a file. It's up to you what to do with the result, you could also send it over TCP or even drop it if it's not needed, or even read from TCP instead of a file, the basic logic is the same.
This still needs some proper error-handling to work around missing files or wrong permissions but that's up to you to implement that.
A example implementation for the process method:
//returns the hex-representation of the bytes
public static byte[] process(byte[] bytes, int length) {
final char[] hexchars = "0123456789ABCDEF".toCharArray();
char[] ret = new char[length * 2];
for ( int i = 0; i < length; ++i) {
int b = bytes[i] & 0xFF;
ret[i * 2] = hexchars[b >>> 4];
ret[i * 2 + 1] = hexchars[b & 0x0F];
}
return ret;
}

TCP socket data getting scrambled

I have a Multi threaded TCP socket listener program. I do a blocked read for data of a particular no of bytes(128 bytes and 4xmultiples),so my packet sizes are 128 bytes,256 bytes,384 bytes and 512 bytes.
I am having problem because sometimes data is getting messed in the socket. For eg:
Supposed to read:
<header><data payload(padded with zeros to compensate size)><footer>
ex-- ABCDddddddddd0000000000WXYZ
What i read sometimes:
ex-- ABCDdd00000000000000000dddddd00
and then the next packet looks like
00000WXYZABCDddddd00000000000000000
so i close the socket and we have defined the protocol to send back 2 or 3 old packets to avoid the loss.
my questions are
1. why does the data get scrambled/messed?
2. can it be avoided by any means?
here is my code for read data.
in = new DataInputStream(conn.getInputStream());
outStream = conn.getOutputStream();
while (m_bRunThread) {
// read incoming stream
in.readFully(rec_data_in_byte, 0, 128); //blocks until 128 bytes are read from the socket
{
//converting the read byte array into string
//finding out the size from a particular position,helps determine if any further reads are required or not.
//if the size is a multiple of 128 and the size is a multiple higher than 1 then more reads are required.
if ((Integer.parseInt(SIZE) % 128 == 0) && ((SIZE / 128) > 1)) {
for(int z = 1;z < lenSIZE;z++) {
in.readFully(rec_data_in_byte1, 0, 128);//changed from in.read(rec_data_in_byte1, 0, 128); as per suggestions
}
//extracting the data,validating and processing it
}
}
}
UPDATE:
Implemented Peters fix but the problem still persists. data is getting scrambled.
adding a few lines of extra code where the byte array is converted into a string.
byte[] REC_data=new byte[1024];
System.arraycopy(rec_data_in_byte1, 0, REC_data, 128*z, 128);
rec_data_string=MyClass2.getData(REC_data,0,Integer.parseInt(SIZE)-1,Integer.parseInt(SIZE));
the getdata() method is below:
String msg = "";//the return String
int count = 1;
for (int i = 0; i < datasize; i++) {
if (i >= startindex) {
if (count <= lengthofpacket) {
msg += String.valueOf((char) (bytedata[i]));
count++;
}
}
}
return msg;
can any of this be the reason for the scramble?
P.S-the scramble is happening the same way as it was happening before.

When you do
int lengthActuallyRead = in.read(rec_data_in_byte1, 0, 128);
You need to check the length read. Otherwise it might read 1 byte, or anything up to 128 in this case. Note, any bytes after what was actually read are untouched so they might be 0 or they could be garbage left from a previous message.
If you expect 128 bytes you can use readFully as you did previously
in.readFully(rec_data_in_byte, 0, 128);
Note: If the amount remaining is less than 128 you might want to do this.
int remaining = size - sizeReadSoFar;
int length = in.read(rec_data_in_byte1, 0, remaining);
This prevents you reading part of the next message while you are still reading the old one.

Array of bytes: how to know exactly numbers of byte received

I'm using array of bytes to store data packet received from another computer.
receivedData = new byte[1024];
receivedPacket = new DatagramPacket(receivedData, receivedData.length);
socket.receive(receivedPacket);
receivedData = receivedPacket.getData();
String res = new String(receivedData); // PROBLEM HERE
the problem is at last line: because I declare receivedData as a byte stream with length 1024. so, always last line will create a new string from whole array, although it doesn't know exactly how many real byte I received. So, I meet frustrated error: res is not received as I wish. (because length of real bytes that I received not fix whole array)
So, my question is: how can I fix this point, how can I know how many bytes I really received to convert to string?

Try using DatagramPacket.getLength().
receivedData = new byte[1024];
receivedPacket = new DatagramPacket(receivedData, receivedData.length);
socket.receive(receivedPacket);
receivedData = receivedPacket.getData();
String charsetName = "US-ASCII"; // set to desired charset
String res = new String(receivedData, 0, receivedPacket.getLength(), charsetName);
Edited to add charset. Thanks, parsifal.

From the javadoc for DatagramSocket.receive():
The length field of the datagram packet object contains the length of
the received message
You can then construct your String using the constructor that takes a byte array and offsets.

Call DatagramPacket.getLength() to find out how many bytes were actually received.
And when you construct the String from those bytes, be sure to specify the encoding (as it is, you're using the JDK default encoding, which may differ from the server's encoding).