For the second time I have this extremely anoying problem with an InputStream.
This InputStream belongs to a Socket that is supposed to receive an image. The code for reading this image is as below:
InputStream input = socket.getInputStream();
InputStreamReader reader = new InputStreamReader(input);
BufferedReader bufferedReader = new BufferedReader(reader);
int total = Integer.parseInt(bufferedReader.readLine());
int bytesRead = 0;
byte[] buffer = new byte[total]; // total is the total size of the image
while (bytesRead < total) {
int next = input.read(buffer, bytesRead, total-bytesRead);
if (next > 0) {
bytesRead += next;
System.out.println("Read: " + bytesRead);
}
}
Now the strange thing is that this code skips the first 1182 bytes of the image, and then reads the remaining part. So when the total size is 15000 bytes, it reads byte 1182-15000.
I checked Wireshark and the whole image is transmitted. The code throws no exceptions. input.read() returns -1 as usual.
Pervious data has been readed from the stream using a BufferedReader. This data is only 5 characters long so it can't contain the missing 1K, but my guess is that the BufferedReader.readLine() method reads (buffers) more bytes from the InputStream than needed. Could this be correct?
I've had the same problem a few months ago but I absolutely have no clue on how I solved it.
Hope anyone can help.
Thanks in advance.
EDIT: I can solve the problem by adding a 100 ms sleep between sending the image size and the image data. It solves the problem but I would still realy like to know a more appropriate solution
As the name **Buffererd**Reader indicates it will snarf more bytes than just the first line from the underlying reader and hence also from the stream. Otherwise if would not be called "buffered".
Unfortunately I'm not aware of any non-deprecated class in Java which allows mixing of binary and textual data in the way you want.
I suggest, that you modify your protocol and transfer the length of the image also in some binary encoding. Then you can stick to InputStream.
My guess is the BufferedReader will assume that whatever reading operations you're performing afterwards will go through it, so it will happily consume input in increments of its buffer size.
One thing you could do is use a BufferedInputStream on top of input, instantiate a DataInputStream on top of that to do the readLine(), and then use the BufferedInputStream in your loop. The documentation says readLine is deprecated because it doesn't convert bytes to characters properly, but I'm hoping that with your first line containing only decimal digits, it shouldn't run into that problem.
I've written a short test, I hope it covers your use case:
import java.net.*;
import java.io.*;
class test{
static byte[] b;
static {
b = new byte[123456];
java.util.Random r = new java.util.Random();
for (int i=0;i<b.length;i++)
b[i] = (byte)(r.nextInt());
}
static void client() throws Exception{
Socket socket = new Socket("127.0.0.1", 9000);
InputStream input = socket.getInputStream();
BufferedInputStream bis = new BufferedInputStream(input);
//read in length
DataInputStream dais = new DataInputStream(bis);
int total = Integer.parseInt(dais.readLine());
System.out.println("Total: "+total);
int bytesRead = 0;
byte[] buffer = new byte[total];
while (bytesRead < total) {
int next = bis.read(buffer, bytesRead, total-bytesRead);
if (next > 0) {
bytesRead += next;
System.out.println("Read: " + bytesRead);
}
}
for (int i=0;i<buffer.length;i++)
if (buffer[i]!=b[i]){
System.err.println("Error");
System.exit(1);
}
System.out.println("OK");
bis.close();
socket.close();
}
static void server() throws Exception{
ServerSocket srv = new ServerSocket(9000);
Socket sock = srv.accept();
OutputStream os = sock.getOutputStream();
BufferedOutputStream bos =new BufferedOutputStream(os);
DataOutputStream daos = new DataOutputStream(bos);
//we're sending the b buffer
//send the length in plain text, followed by newline
byte[]num = String.valueOf(b.length).getBytes();
daos.write(num,0,num.length);
daos.write(10);
//send actual buffer contents
bos.write(b, 0, b.length);
os.close();
srv.close();
sock.close();
}
public static void main(String[]args){
new Thread(new Runnable(){
public void run(){
try{server();}catch(Exception e){e.printStackTrace();}
}
}).start();
new Thread(new Runnable(){
public void run(){
try{client();}catch(Exception e){e.printStackTrace();}
}}).start();
}
}
Have you considered just sending the image data (without the preceding image size) and using an external library such as Apache Commons IO to handle this for you? In particular, I think that you will find IOUtils.toByteArray(InputStream input) interesting, e.g.
InputStream input = socket.getInputStream();
byte[] buffer = IOUtils.toByteArray(input);
Related
I'm trying to download some images provided by a hoster. This is the method I use:
public static void downloadImage(String imageLink, File f) throws IOException
{
URL url = new URL(imageLink);
byte[] buffer = new byte[1024];
BufferedInputStream in = new BufferedInputStream(url.openStream(), buffer.length);
BufferedOutputStream out = new BufferedOutputStream(new FileOutputStream(f), buffer.length);
while (in.read(buffer) > 0)
out.write(buffer);
out.flush();
out.close();
in.close();
}
However, the file turn out too big. 5MB for a 80x60 jpg is too much in my opinion.
What could be the cause of this?
You are doing things wrong here: read() returns the number of bytes that were really read; thus you have to write exactly that number from your buffer array into your output stream.
Your code is corrupting your output; and simply writing out a buffer array ... that mostly consists of 0s!
Instead do something like:
int bytesRead;
while ( ( bytesRead = in.read(buffer)) > 0) {
byte outBuffer[] = new byte[bytesRead];
... then use arraycopy to move bytesRead bytes
out.write(outBuffer);
}
( this is meant as inspiration to get you going, more pseudo like than real code )
Despite hours of researching this problem, I have made very little progress. According to my professor, the code should be working as written...
I have a server that stays open, and a client that requests a file. Once the client receives the file, the client closes.
When I open the server, I am able to transfer a complete .jpg image file. The client then closes while the server remains open. I start up another client and try to transfer the same image, and only a portion of the bytes are transferred/written to the disk. The file transfer is only completely successful for the first file transferred by the server!
Additionally strange, a simple .txt text file never successfully transfers. I believe the cause is on the server side because it remains open as opposed to the client, which starts over each time.
Server Code:
import java.io.*;
import java.net.*;
import java.util.Arrays;
class ft_server {
public static void main(String args[]) throws Exception {
/*
* Asks user for port number and listens on that port
*/
BufferedReader portFromUser = new BufferedReader(new InputStreamReader(System.in));
System.out.println("Enter the port you'd like to use: ");
int portNumber = Integer.valueOf(portFromUser.readLine());
if (portNumber < 1 || portNumber > 65535) {
System.out.println("Please choose a port number between 1 and 65535.");
return;
}
portFromUser.close();
ServerSocket listenSocket = new ServerSocket(portNumber);
/*
* Finished with user input
*/
/*
* Continuously listens for clients:
*/
while (true) {
Socket clientSocket = listenSocket.accept();
BufferedReader inFromClient = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
DataOutputStream outToClient = new DataOutputStream(clientSocket.getOutputStream());
String clientIP = clientSocket.getRemoteSocketAddress().toString();
System.out.println("The client " + clientIP + " connected!");
String clientMessage = inFromClient.readLine();
System.out.println("The client requested file: " + clientMessage);
// Get file. If doesn't exist, let's client know.
// Otherwise informs client of file size.
File myFile = new File(clientMessage);
if (!myFile.exists()) {
outToClient.writeBytes("File does not exist!\n");
return;
} else {
outToClient.writeBytes(String.valueOf((int)myFile.length()) + "\n");
}
// Create array for storage of file bytes:
byte[] byteArray = new byte[(int)myFile.length()];
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(myFile));
// Read file into array:
bis.read(byteArray, 0, byteArray.length);
// Send the file:
outToClient.write(byteArray, 0, byteArray.length);
outToClient.close();
clientSocket.close();
}
}
}
Client Code:
import java.io.*;
import java.net.*;
class ft_client {
public static void main(String args[]) throws Exception {
int byteSize = 2022386;
int bytesRead;
/*
* Asks user for IP and port:
*/
BufferedReader inFromUser = new BufferedReader(new InputStreamReader(System.in));
System.out.println("Enter an IP address: ");
String ipAddress = inFromUser.readLine();
System.out.println("Enter a port: ");
String port = inFromUser.readLine();
Socket clientSocket;
try {
// Makes socket, port, and calls connect. Assumes it's TCP:
clientSocket = new Socket(ipAddress, Integer.valueOf(port));
} catch (Exception e) {
System.out.println(e.getMessage());
return;
}
// Creates InputStream from server to get file size and other messages:
BufferedReader inFromServer = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
// Anything written to this will be sent to the server:
DataOutputStream outToServer = new DataOutputStream(clientSocket.getOutputStream());
// Asks for a file name to download from the server:
System.out.println("What file do you want?: ");
String message = inFromUser.readLine();
outToServer.writeBytes(message + "\n");
inFromUser.close();
// Listens for confirmation from server.
// If the file exists, the file size is delivered here:
String response = inFromServer.readLine();
System.out.println("File size: " + response);
if (response.equals("File does not exist!")) {
return;
}
// Receives file from server:
byteSize = (int) Integer.valueOf(response);
byte[] byteArray = new byte[byteSize];
InputStream is = clientSocket.getInputStream(); // calling clientSocket.getInputStream() twice???
FileOutputStream fos = new FileOutputStream(message);
BufferedOutputStream bos = new BufferedOutputStream(fos);
// Continuously writes the file to the disk until complete:
int total = 0;
while ((bytesRead = is.read(byteArray)) != -1) {
bos.write(byteArray, 0, bytesRead);
total += bytesRead;
}
bos.close();
System.out.println("File downloaded (" + total + " bytes read)");
clientSocket.close();
}
}
Are buffered readers interfering with output streams? Is there a better way to transfer files?
It's worth checking, in your server code, what value comes back from the file read() call, so:
int bytesRead = bis.read(byteArray, 0, byteArray.length);
System.out.println("File bytes read: " + bytesRead + " from file size: " + myFile.length());
The read() method is under no obligation to fill the byteArray - only to return something and to tell you how many bytes it read. From the docs, it:
Reads up to len bytes of data from this input stream into an array of
bytes. If len is not zero, the method blocks until some input is
available; otherwise, no bytes are read and 0 is returned.
You need to keep reading in a loop. I'd do this (actually, same as your client!):
int n;
while ((n = bis.read(byteArray, 0, byteArray.length)) != -1) {
// Send the chunk of n bytes
outToClient.write(byteArray, 0, n);
}
bis.close();
outToClient.close();
or something similar. I've closed the file too: it'd close on GC/finalize, but that could be a while, and meanwhile you're holding the file open.
EDIT
The specific problem with your image-read in this case is in your client code. You read the file size near the top of the code:
// Creates InputStream from server to get file size and other messages:
BufferedReader inFromServer = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
and then you access the client again:
InputStream is = clientSocket.getInputStream(); // calling clientSocket.getInputStream() twice???
and as your comment suggests, this is bad! Thank you to #EJP for highlighting this!
This causes a problem of buffer over-ingestion: the BufferedReader consumes more bytes into its belly than you extract from it, so when you visit the clientSocket inputstream the second time, the read-pointer has moved on. You never look again at what the BufferedReader consumed.
As a general rule, once you plug buffering code onto something, you must be careful to read only from that buffer. In this case, it's difficult, because you can't read image (raw binary) data from a Reader, because it will busily interpret the binary values as characters and read them as UTF-8 or something.
Even without buffers, it's a minor sin to mix Readers (text oriented) and binary data (DataStreams) on the same stream. HTTP and email does this, so you are in good company, but they get away with it by being very tightly specified. Problem is, you can easily get snarled with questions of local/default character encoding at each end, whether you're reading Unix "LF" vs Windows "CR/LF" line endings etc.
In this case, try not using BufferedReaders at all, and try using DataInput/Output streams all the way. Try writeUTF(s) and readUTF() for transferring the String data. Ideally, create them like this:
DataInputStream inFromServer = new DataInputStream (new BufferedInputStream(clientSocket.getInputStream()));
so you still get the benefits of buffering.
EDIT 2
So seeing the new client code:
byteSize = (int) Integer.valueOf(response);
byte[] byteArray = new byte[byteSize];
FileOutputStream fos = new FileOutputStream(message);
int readBytes = inFromServer.read(byteArray);
// Continuously writes the file to the disk until complete:
int total = 0;
for (int i=0; i<byteArray.length; i++) {
fos.write(byteArray[i]);
total++;
}
fos.close();
Here, we're assuming that because the byteArray array is set to the right size, that the inFromServer.read(byteArray) will populate it - it won't. It's good to assume that any and all read operations will return you just as much data as the system has to hand: in this case, it's probably going to return as soon as it gets the first packet or two, with an underfilled array. This is same as C and Unix read behaviour too.
Try this - I'm repeatedly reading and writing a 4K buffer, until the byte count is reached (as determined by summing the return values of the reads):
byteSize = (int) Integer.valueOf(response);
byte[] byteArray = new byte[4096];
FileOutputStream fos = new FileOutputStream(message);
int total = 0;
// Continuously writes the file to the disk until complete:
while (total < byteSize && (readBytes = inFromServer.read(byteArray)) != -1) {
fos.write(byteArray, 0, readBytes);
total += readBytes;
}
fos.close();
A variant is this - same thing, but byte at a time. Might be a bit clearer. It's going to be slow - all those reads and writes are hitting the OS, but if you put a BufferedInputStream/BufferedOutputStream around the socket/file streams, it'll iron that out. I've added them:
DataInputStream inFromServer =
new DataInputStream(new BufferedInputStream(clientSocket.getInputStream()));
...
byteSize = (int) Integer.valueOf(response);
OutputStream fos = new BufferedOutputStream(FileOutputStream(message));
int total = 0;
int ch;
// Continuously writes the file to the disk until complete:
while (total < byteSize && (ch = inFromServer.read()) != -1) {
fos.write(ch);
total ++;
}
fos.close();
And finally! the simplest answer is this. Your code, but changed to:
int readBytes = inFromServer.readFully(byteArray);
Yes! Those nice people in 1990's Javasoft added a DataInput.readFully method, which does what you want! - basically wraps the code above. It's the simplest solution, and arguably most correct approach: "use existing libraries where possible". OTOH, it's the least educational, and the time you spend getting used to read/writes like this is not deducted from your life-expectancy!
And in fact, the readFully approach has severe limitations. Try pointing it at a 1GB file and see what happens (after you've fixed up the array size at the top): you'll a) run out memory, and b) wish that while you were ingesting a huge blob, you could at least be spooling it out to disk. If you try a 2.5G file, you'll notice that some of those ints should become longs to cope with numbers >= 2^31.
If it was me, I'd do the 4K buffer one. (BTW I'm writing this on a laptop with no Java compiler installed, so I haven't actually run the above! DO respond if there are any difficulties.)
My application is unable to transfer data properly over a socket connection and write it to a file properly. Files over about 65,535 bytes get corrupted and are no longer recognized by the programs designed to run them.
I have been able to send small .doc and .txt files successfully, but .mp3 .wmv .m4a .avi and just about anything else does not work. Neither do larger docs.
I have looked all over the internet for a solution to this problem. I have repeatedly tweaked the I/O code to fix the problem but it still doesn't work! Here is the I/O code in the super class that handles sending and receiving files. If you need anymore information/other parts of code, let me know.
protected void sendFile() throws IOException {
byte[] bytes = new byte[(int) file.length()];
buffin = new BufferedInputStream(new FileInputStream(file));
int bytesRead = buffin.read(bytes,0,bytes.length);
System.out.println(bytesRead);
out = sock.getOutputStream();
out.write(bytes,0,fileBytes);
out.flush();
out.close();
}
protected void receiveFile() throws IOException {
byte[] bytes = new byte[fileBytes];
in = sock.getInputStream();
for(int i=0;i<fileBytes;i++) {
in.read(bytes);
}
fos = new FileOutputStream("/Datawire/"+fileName);
buffout = new BufferedOutputStream(fos);
buffout.write(bytes,0,fileBytes);
buffout.flush();
buffout.close();
}
UPDATED CODE (that works):
protected void sendFile() throws IOException {
if((file.length())<63000) {
byte[] bytes = new byte[(int)file.length()];
buffin = new BufferedInputStream(new FileInputStream(file));
buffin.read(bytes,0,bytes.length);
out = sock.getOutputStream();
out.write(bytes,0,bytes.length);
out.close();
} else {
byte[] bytes = new byte[32000];
buffin = new BufferedInputStream(new FileInputStream(file));
out = sock.getOutputStream();
int bytesRead;
while((bytesRead = buffin.read(bytes))>0) {
out.write(bytes,0,bytesRead);
}
out.close();
}
}
protected void receiveFile() throws IOException {
if(fileBytes<63000) {
byte[] bytes = new byte[32000];
in = sock.getInputStream();
System.out.println(in.available());
in.read(bytes,0,fileBytes);
fos = new FileOutputStream("/Datawire/"+fileName);
buffout = new BufferedOutputStream(fos);
buffout.write(bytes,0,bytes.length);
buffout.close();
} else {
byte[] bytes = new byte[16000];
in = sock.getInputStream();
fos = new FileOutputStream("/Datawire/"+fileName);
buffout = new BufferedOutputStream(fos);
int bytesRead;
while((bytesRead = in.read(bytes))>0) {
buffout.write(bytes,0,bytesRead);
}
buffout.close();
}
}
The issue is that you are sending only chunks of it. That is, you are only sending 64k of the file ever. If the file is ever larger then 64k the other end will never see it.
You want to continously read from the BufferedInputStream until the read() returns either less then the length or -1.
Your code is completely wrong. This is how to copy a stream in Java:
int count;
byte[] buffer = new byte[8192]; // more if you like but no need for it to be the entire file size
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
You should use this both when sending the file and when receiving the file. At present your sending method hopes that the entire file fits into memory; fits into INTEGER_MAX bytes; and is read in one chunk by the read method, without even checking the result. You can't assume any of those things. Your receive method is complete rubbish: it just keeps overwriting the same array, again without checking any read() results.
EDIT: Your revised code is just as bad, or worse. You are calling read() to check for EOS and then throwing that byte away, and then calling read() again and throwing away the read count it returns. You pointlessly have a different path for files < 64000, or 63000, or whatever it is, that has zero benefit except to give you two code paths to test, or possibly four, instead of one. The network only gives you 1460 bytes at a time at best anyway so what is the point? You already have (a) a BufferedInputStream with a default buffersize of 8192, and (b) my code that uses a byte[] buffer of any size you like. My code above works for any amount of data in two lines of executable code. Yours is 20. QED.
I suggest that you use some good library to read and write file contents as well as socket read/write. For example Apache Commons IO. If you insist on writig code yourself, do it smaller chunks rather than the whole file at once.
You have to consider that InputStream.read returns the number of bytes read which may be less than the total number of bytes in the file.
You would probably be better off just letting something like CopyUtils.copy take care of this for you.
You need to loop until bytesRead < 0. You need to make sure that fileBytes is => than the transferred file.
protected void receiveFile() throws IOException {
byte [] bytes = new byte [fileBytes];
InputStream is = sock.getInputStream();
FileOutputStream fos = new FileOutputStream("/Datawire/"+fileName);
BufferedOutputStream bos = new BufferedOutputStream(fos);
int bytesRead = is.read(bytes,0,bytes.length);
int current = bytesRead;
do {
bytesRead =
is.read(bytes, current, (bytes.length-current));
if(bytesRead >= 0) current += bytesRead;
} while(bytesRead > -1);
bos.write(bytes, 0 , current);
bos.flush();
bos.close();
}
I have written a java code which sends a .exe file from the server to the client using FileInputStream and BufferedInputStream, but the file gets corrupted at the client side.
What could be the reason?
command1= ServerFrame.msg1+".exe";
File p=new File(command1);
FileInputStream f=new FileInputStream(p);
BufferedInputStream bis=new BufferedInputStream(f);
Integer d=bis.available();
int d1=d;
byte b[]=new byte[d];
bis.read(b,0,d1);
System.out.println(d1);
dos=new DataOutputStream(s.getOutputStream());
BufferedOutputStream bos=new BufferedOutputStream(s.getOutputStream());
dos.writeUTF(d.toString()); // sending length in long
bos.write(b,0,d1); // sending the bytess
bos.flush();
bis.close();
bos.close();
dos.close();
I suppose that s is your socket. There are few thing that can be wong in your code:
bis.available() returns the number of bytes that can be read without bocking, not the total size of the file, you should use a loop to read the file
you use the output stream in two different buffers and you write to both of them without flushing; also, why are you writing UTF?
Here is what you intend to do:
private void copy(InputStream in, OutputStream out) {
byte[] buf = new byte[0x1000];
int r;
while ((r = in.read(buf)) >= 0) {
out.write(b, 0, r);
}
}
InputStream in = new BufferedInputStream(new FileInputStream(path));
OutputStream out = new BufferedOutputStream(s.getOutputStream());
copy(in, out);
in.close();
out.close();
bis.available() returns the bytes available for read, it may not be the full content size, u have to read in a loop till it reaches EOF.
in case someone stuck with same problem, buffer size is the culprit in this case :
Integer d=bis.available();
byte b[]=new byte[d];
It should be lesser try 1024 or something:
byte b[]=new byte[1024];
hope this helps..
I am creating a file server application for school assignment. What I currently have is a simple Client class that sends an image through TCP and a Server class that receives it and writes it to the file.
this is my client code
import java.io.*;
import java.net.*;
class Client {
public static void main(String args[]) throws Exception {
long start = System.currentTimeMillis();
Socket clientSocket = new Socket("127.0.0.1", 6789);
DataOutputStream outToServer = new DataOutputStream(clientSocket.getOutputStream());
File file = new File("hot.jpg");
FileInputStream fin = new FileInputStream(file);
byte sendData[] = new byte[(int)file.length()];
fin.read(sendData);
outToServer.write(sendData, 0, sendData.length);
clientSocket.close();
long end = System.currentTimeMillis();
System.out.println("Took " + (end - start) + "ms");
}
}
and this is my server code.
import java.io.*;
import java.net.*;
class Server {
public static void main(String args[]) throws Exception {
ServerSocket serverSocket = new ServerSocket(6789);
Socket connectionSocket = serverSocket.accept();
DataInputStream dis = new DataInputStream(connectionSocket.getInputStream());
byte[] receivedData = new byte[61500]; // <- THIS NUMBER
for(int i = 0; i < receivedData.length; i++)
receivedData[i] = dis.readByte();
connectionSocket.close();
serverSocket.close();
FileOutputStream fos = new FileOutputStream("received.jpg");
fos.write(receivedData);
fos.close();
}
}
My question is how to get the size of the file that is being sent. If you check the Server code you'll see that I've hardcoded the number i.e. 61500 at the moment. How can I retrieve this number dynamically?
Or, am I doing this the wrong way? What an alternative solution would be?
Add one "length field" before sending the file. (Note that since you read the file to memory the maximum size of the file can be ~2GB.)
Before sending the file write the length of the file:
outToServer.writeInt(sendData.length);
And when receiving read the length first and use it as a length:
int dataLength = dis.readInt()
byte[] receivedData = new byte[dataLength];
A better way would be not to read the file into memory first but to transfer it directly from the FileInputStream - then you could transfer bigger files!
If you know the length, using readFully() is much more efficient than reading a byte at a time.
In this case, you don't need to know the length, you can write the loop to read/write as much data as you get.
InputStream is = connectionSocket.getInputStream();
byte[] bytes = new byte[8192];
int len;
while((len = is.read(bytes)) > 0)
fos.write(bytes, 0, len);
You can avoid reading the whole file into memory by copying the data as you read it.
FileInputStream fis = new FileInputStream(filename);
OutputStream os = socket.getOutputStream();
byte[] bytes = new byte[8192];
int len;
while((len = fis.read(bytes)) > 0)
os.write(bytes, 0, len);
You can use Apache IOUtils.copy() to perform the copy from one stream to another if you wish.
This approach has the advantage that the file can be any size (greater than 2 GB). Using an array is limited to 2 GB (and uses more memory)