Develop Reference

Java Send file via Socket

I'm writing a class for 2-way sending file via Sockets in Java
Here on GitHub is it.
Everything is good until file receiving finished.
Shortly:
in client.java is hardcoded way to C:\Maven\README.txt
firstly I send filename
then I send file length
at third step I'm sending file from FileInputStream to DataOutputStream
On client:
byte[] bytes = new byte[(int)forSend.length()];
InputStream fin = new FileInputStream(forSend);
int count;
while ((count = fin.read(bytes)) > 0) {
out.write(bytes, 0, count);
}
fin.close();
fout = new FileOutputStream(filename);
byte[] bytes = new byte[length];
System.out.println("receiving file...");
int count;
while ((count = in.read(bytes)) > 0) {
fout.write(bytes, 0, count);
}
fout.flush();
fout.close();
file on server is completely received (the same length and content)
When I'm trying to add code for writing something to socket after that, after start server and client are waiting for something (I don't know what)
Previously I meet this situation when lost one DataInputStream reading (message sent from server but there was no reciever on client for this message). But currently I'm trying to add flag which is changed after file transfer finished and check for it's state later. It's work both on server and client, but adding read/write from/to Socket return me back to situation when both server and client are wait for something.
What's wrong now?

My friend Denr01 helped me, so my mistake was control of file length, I don't have it anywhere in my question. And because of that my "finishing" confirmation was writen to file.
The way to solve problem is in sender:
int read = 0;
int block = 8192;
int count = 0;
byte[] bytes = new byte[block];
while (read != forSend.length()) {
count = fin.read(bytes, 0, block);
out.writeInt(count);
out.write(bytes, 0, count);
read += count;
System.out.println("already sent " + read + " bytes of " + forSend.length());
}
Sender read bytes and write in count number of them
It send count to reciever, so reciever will know how many bytes to recieve in current loop iteration
Then Sender send block of bytes and increments a counter of bytes read
Repeating this while counter not equal to file length
In sender:
int block = 8192;
int count = 0;
int read = 0;
byte[] bytes = new byte[block];
System.out.println("recieving file...");
while (read != length) {
block=in.readInt();
in.readFully(bytes, 0, block);
fout.write(bytes, 0, block);
read += block;
System.out.println("already recieved " + read + " bytes of " + length);
}
Make byte array with length equal to sender's block length
In every iteration firstly read next block length and then read this count of bytes
Increment reciever's counter
Repeat this while counter is not equal to file length which was recieved previously
In this case we have control of each file reading iteration and always know how many bytes to recieve, so when all the bytes recieved files are identical and next "messages" will not be writen into file.

Java Reading large files into byte array chunk by chunk

So I've been trying to make a small program that inputs a file into a byte array, then it will turn that byte array into hex, then binary. It will then play with the binary values (I haven't thought of what to do when I get to this stage) and then save it as a custom file.
I studied a lot of internet code and I can turn a file into a byte array and into hex, but the problem is I can't turn huge files into byte arrays (out of memory).
This is the code that is not a complete failure
public void rundis(Path pp) {
byte bb[] = null;
try {
bb = Files.readAllBytes(pp); //Files.toByteArray(pathhold);
System.out.println("byte array made");
} catch (Exception e) {
e.printStackTrace();
}
if (bb.length != 0 || bb != null) {
System.out.println("byte array filled");
//send to method to turn into hex
} else {
System.out.println("byte array NOT filled");
}
}
I know how the process should go, but I don't know how to code that properly.
The process if you are interested:
Input file using File
Read the chunk by chunk of the file into a byte array. Ex. each byte array record hold 600 bytes
Send that chunk to be turned into a Hex value --> Integer.tohexstring
Send that hex value chunk to be made into a binary value --> Integer.toBinarystring
Mess around with the Binary value
Save to custom file line by line
Problem:: I don't know how to turn a huge file into a byte array chunk by chunk to be processed.
Any and all help will be appreciated, thank you for reading :)

To chunk your input use a FileInputStream:
Path pp = FileSystems.getDefault().getPath("logs", "access.log");
final int BUFFER_SIZE = 1024*1024; //this is actually bytes
FileInputStream fis = new FileInputStream(pp.toFile());
byte[] buffer = new byte[BUFFER_SIZE];
int read = 0;
while( ( read = fis.read( buffer ) ) > 0 ){
// call your other methodes here...
}
fis.close();

To stream a file, you need to step away from Files.readAllBytes(). It's a nice utility for small files, but as you noticed not so much for large files.
In pseudocode it would look something like this:
while there are more bytes available
read some bytes
process those bytes
(write the result back to a file, if needed)
In Java, you can use a FileInputStream to read a file byte by byte or chunk by chunk. Lets say we want to write back our processed bytes. First we open the files:
FileInputStream is = new FileInputStream(new File("input.txt"));
FileOutputStream os = new FileOutputStream(new File("output.txt"));
We need the FileOutputStream to write back our results - we don't want to just drop our precious processed data, right? Next we need a buffer which holds a chunk of bytes:
byte[] buf = new byte[4096];
How many bytes is up to you, I kinda like chunks of 4096 bytes. Then we need to actually read some bytes
int read = is.read(buf);
this will read up to buf.length bytes and store them in buf. It will return the total bytes read. Then we process the bytes:
//Assuming the processing function looks like this:
//byte[] process(byte[] data, int bytes);
byte[] ret = process(buf, read);
process() in above example is your processing method. It takes in a byte-array, the number of bytes it should process and returns the result as byte-array.
Last, we write the result back to a file:
os.write(ret);
We have to execute this in a loop until there are no bytes left in the file, so lets write a loop for it:
int read = 0;
while((read = is.read(buf)) > 0) {
byte[] ret = process(buf, read);
os.write(ret);
}
and finally close the streams
is.close();
os.close();
And thats it. We processed the file in 4096-byte chunks and wrote the result back to a file. It's up to you what to do with the result, you could also send it over TCP or even drop it if it's not needed, or even read from TCP instead of a file, the basic logic is the same.
This still needs some proper error-handling to work around missing files or wrong permissions but that's up to you to implement that.
A example implementation for the process method:
//returns the hex-representation of the bytes
public static byte[] process(byte[] bytes, int length) {
final char[] hexchars = "0123456789ABCDEF".toCharArray();
char[] ret = new char[length * 2];
for ( int i = 0; i < length; ++i) {
int b = bytes[i] & 0xFF;
ret[i * 2] = hexchars[b >>> 4];
ret[i * 2 + 1] = hexchars[b & 0x0F];
}
return ret;
}

InputStream via ReadableByteChannel does not read to end

I have an existing problem where I am using InputStreams and I want to increase the performance of reading from this channel. Therefore i read with a ReadableByteChannel.
As a result the reading is much faster with this code:
public static String readAll(InputStream is, String charset, int size) throws IOException{
try(ByteArrayOutputStream bos = new ByteArrayOutputStream()){
java.nio.ByteBuffer buffer = java.nio.ByteBuffer.allocate(size);
try(ReadableByteChannel channel = Channels.newChannel(is)){
int bytesRead = 0;
do{
bytesRead = channel.read(buffer);
bos.write(buffer.array(), 0, bytesRead);
buffer.clear();
}
while(bytesRead >= size);
}
catch(Exception ex){
ex.printStackTrace();
}
String ans = bos.toString(charset);
return ans;
}
}
The Problem is: It does not read to the end every time! If I try to read a File it works pretty good. If I read from a network Socket (to request a WebPage manually for example) it sometimes stops somewhere in between.
What can I do to read to the end?
I don't want to use something like this:
StringBuilder result = new StringBuilder();
while(true){
int ans = is.read();
if(ans == -1) break;
result.append((char)ans);
}
return result.toString();
because this implementation is slow.
I hope you can help me with my problem. maybe i have some mistake in my code.

This causes problem:
... } while (bytesRead >= size);
Reading from socket may return when at least one byte was read (or even if no bytes in case of non-blocking). So if there are not enough bytes in OS socket buffer, the condition will break the loop although obviously not full content was read. If the size identifies expected length to be received, implement total += bytesRead and break the loop when total reaches size. Or if you reach end of file of course...

Your copy loop is completely wrong. There's no reason why bytesRead should ever be >= size, and it misbehaves at end of stream. It should be something like this:
while ((bytesRead = channel.read(buffer)) > 0)
{
bos.write(buffer.array(), 0, bytesRead);
buffer.clear();
}
with suitable adjustments for limiting the transfer to size bytes, which are non-trivial.
But layering all this over an existing InputStream cannot possibly be 'much faster' tha using the InputStream directly, unless because of the premature termination. Unless your idea of use an InputStream is what you posted, which is horrifically slow. Try that with a 'BufferedInputStream.

difference between input.read and input.read(array, offset, length)

I'm trying to understand how inputstreams work. The following block of code is one of the many ways to read data from a text file:-
File file = new File("./src/test.txt");
InputStream input = new BufferedInputStream (new FileInputStream(file));
int data = 0;
while (data != -1) (-1 means we reached the end of the file)
{
data = input.read(); //if a character was read, it'll be turned to a bite and we get the integer representation of it so a is 97 b is 98
System.out.println(data + (char)data); //this will print the numbers followed by space then the character
}
input.close();
Now to use input.read(byte, offset, length) i have this code. I got it from here
File file = new File("./src/test.txt");
InputStream input = new BufferedInputStream (new FileInputStream(file));
int totalBytesRead = 0, bytesRemaining, bytesRead;
byte[] result = new byte[ ( int ) file.length()];
while ( totalBytesRead < result.length )
{
bytesRemaining = result.length - totalBytesRead;
bytesRead = input.read ( result, totalBytesRead, bytesRemaining );
if ( bytesRead > 0 )
totalBytesRead = totalBytesRead + bytesRead;
//printing integer version of bytes read
for (int i = 0; i < bytesRead; i++)
System.out.print(result[i] + " ");
System.out.println();
//printing character version of bytes read
for (int i = 0; i < bytesRead; i++)
System.out.print((char)result[i]);
}
input.close();
I'm assuming that based on the name BYTESREAD, this read method is returning the number of bytes read. In the documentation, it says that the function will try to read as many as possible. So there might be a reason why it wouldn't.
My first question is: What are these reasons?
I could replace that entire while loop with one line of code: input.read(result, 0, result.length)
I'm sure the creator of the article thought about this. It's not about the output because I get the same output in both cases. So there has to be a reason. At least one. What is it?

The documentation of read(byte[],int,int says that it:
Reads up to len bytes of data.
An attempt is made to read as many as len bytes
A smaller number may be read.
Since we are working with files that are right there in our hard disk, it seems reasonable to expect that the attempt will read the whole file, but input.read(result, 0, result.length) is not guaranteed to read the whole file (it's not said anywhere in the documentation). Relying in undocumented behaviors is a source for bugs when the undocumented behavior change.
For instance, the file stream may be implemented differently in other JVMs, some OS may impose a limit on the number of bytes that you may read at once, the file may be located in the network, or you may later use that piece of code with another implementation of stream, which doesn't behave in that way.
Alternatively, if you are reading the whole file in an array, perhaps you could use DataInputStream.readFully
About the loop with read(), it reads a single byte each time. That reduces performance if you are reading a big chunk of data, since each call to read() will perform several tests (has the stream ended? etc) and may ask the OS for one byte. Since you already know that you want file.length() bytes, there is no reason for not using the other more efficient forms.

Imagine you are reading from a network socket, not from a file. In this case you don't have any information about the total amount of bytes in the stream. You would allocate a buffer of fixed size and read from the stream in a loop. During one iteration of the loop you can't expect there are BUFFERSIZE bytes available in the stream. So you would fill the buffer as much as possible and iterate again, until the buffer is full. This can be useful, if you have data blocks of fixed size, for example serialized object.
ArrayList<MyObject> list = new ArrayList<MyObject>();
try {
InputStream input = socket.getInputStream();
byte[] buffer = new byte[1024];
int bytesRead;
int off = 0;
int len = 1024;
while(true) {
bytesRead = input.read(buffer, off, len);
if(bytesRead == len) {
list.add(createMyObject(buffer));
// reset variables
off = 0;
len = 1024;
continue;
}
if(bytesRead == -1) break;
// buffer is not full, adjust size
off += bytesRead;
len -= bytesRead;
}
} catch(IOException io) {
// stream was closed
}
ps. Code is not tested and should only point out, how this function can be useful.

You specify the amount of bytes to read because you might not want to read the entire file at once or maybe you couldn't or might not want to create a buffer as large as the file.

Send file length with outputstream and receive length and byte[] with inputstream for streaming frames from one device to the other Android/Java

I have searched and searched and everything I have found has been helpful but I keep getting an out of memory error. The images I send are .06 MB so I know the problem isn't from decoding the byte[] into a bitmap. When I remove the while loops this works like a charm for one frame but I want multiple frames. I am getting a byte[] and sending it to a different device using sockets but I am at a loss how to do this. My problem is that I don't send and receive the correct byte[] length. This is what i am doing currently.
while (count != -1) {
//first send the byte[] length
dataOutputStream.writeInt(sendPackage.length);
//pass a byte array
publishProgress("sending file to client");
showMyToastOnUiThread(String.valueOf(sendPackage.length));
outputStream.write(sendPackage, 0, sendPackage.length);
outputStream.flush();
}
Receive byte[] on different device:
int count = inputStream.read();
while (count != -1) {
int byteArrayLength = dataInputStream.readInt();
Log.i(MainActivity.TAG, "Starting convert to byte array");
byte[] receivedBytes = convertInputStreamToByteArray(inputStream, byteArrayLength);
Bitmap bitmap = BitmapFactory.decodeByteArray(receivedBytes, 0, receivedBytes.length);
publishProgress(bitmap);
}
//convert inputstream to byte[]
public byte[] convertInputStreamToByteArray(InputStream inputStream, int readLength) {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
byte[] data = new byte[readLength];
try {
Log.i(MainActivity.TAG, "Starting convert to byte array while loop");
int readTotal = 0;
int count = 0;
while (count >= 0 && readTotal < readLength) {
count = inputStream.read(data, readTotal, readLength - readTotal);
if (readLength > 0) {
readTotal += count;
}
}
Log.i(MainActivity.TAG, "Finished convert to byte array while loop");
} catch (IOException e) {
Log.e(MainActivity.TAG, "error: " + e.getMessage());
e.printStackTrace();
}
return data;
}

This is the problem:
int count = inputStream.read();
while (count != -1) {
You're consuming a byte and then ignoring it. That means the next value you read (the size) will be incorrect. You need a different way of telling whether you're at the end of the stream. Some options:
Send a -1 when you're finished; that way you can stop as soon as readInt returns -1
If you know it, send the number of images you're going to send before you start sending them
Use mark(1), then read(), then reset() - if your stream supports marking. I don't know whether it will or not. You could always wrap it in BufferedInputStream if not.
Reimplement DataInputStream.readInt yourself in a way which detects the end of the stream as being an expected possibility instead of throwing an exception
Just catch an exception in readInt (not nice - getting to the end of the stream isn't really exceptional)