Develop Reference

File transfer stops after transmission of a certain amount of bytes

I am currently working on a file transfer program and I ran into a strange issue.
I have two classes: A sender class and a recipient class. You can read the source code along with the error message below.
Thanks for your help in advance.
Sender:
public static void sendFile(final File file, final String ip){
Thread t = new Thread(new Runnable() {
#Override
public void run() {
try {
Socket s = new Socket(ip, 4816);
DataOutputStream dos = new DataOutputStream(s.getOutputStream());
DataInputStream dis = new DataInputStream(s.getInputStream());
FileInputStream fis = new FileInputStream(file);
String filename = file.getName();
if(!dis.readUTF().equals("end")){
dos.writeUTF(filename);
dos.flush();
long size = file.length();
byte[] b = new byte[1024];
int read;
dos.writeUTF(Long.toString(size));
dos.flush();
while((read = fis.read(b)) != -1){
dos.write(b, 0, read);
dos.flush();
}
fis.close();
dos.flush();
dos.writeUTF("end");
System.out.println("Sender: Done");
dos.flush();
dis.close();
dos.close();
s.close();
}
return;
} catch (IOException e) {
e.printStackTrace();
}
}
});
t.start();
}
Recipient:
private ServerSocket sock;
private Thread t;
public listener(){
try {
sock = new ServerSocket(4816);
listen();
} catch (IOException e) {
e.printStackTrace();
}
}
private void listen(){
t = new Thread(new Runnable() {
public void run() {
Socket s;
try {
while((s = sock.accept()) != null){
DataInputStream dis = new DataInputStream(s.getInputStream());
DataOutputStream dos = new DataOutputStream(s.getOutputStream());
String filename = dis.readUTF();
long size = Long.valueOf(dis.readUTF());
byte[] b = new byte[1024];
FileOutputStream fos = new FileOutputStream(new File(filename), true);
long read;
do{
read = dis.read(b, 0, b.length);
fos.write(b, 0, b.length);
}while(!(read < 1024));
System.out.println("Recipient: Done");
fos.close();
dos.close();
dis.close();
s.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
});
t.run();
}
Error (After 1086464 out of 1513308160 bytes were transmitted. [1062 * 1024]):
java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at java.io.DataOutputStream.write(Unknown Source)
at main.sender$1.run(sender.java:60)
at java.lang.Thread.run(Unknown Source)

I think the issue is here in the recipient..
do{
read = dis.read(b, 0, b.length);
fos.write(b, 0, b.length);
}while(!(read < 1024));
You are saying to only loop while read is not less than 1024. Any read() operation can return a value less than the maximum buffer length at any time, even if the stream is not "at the end". Especially when network sockets are involved. The number of read bytes may be greater than 0 but less than 1024 on any read because that's simply how many bytes are available to the stream at that time.
The read call is giving you all the data it has (which fits the buffer) at that time, without having to block.
Try changing it to..
int read;
while ((read = dis.read(b, 0, b.length)) != -1 ) {
fos.write(b, 0, read);
}
You had this kind of loop in the sender and it is correct (although you don't need the flush within the loop).
This kind of bug is more common than you might think. I'v seen it a lot over the years, even in "enterprise products". It doesn't get picked up and fixed because most of the time it works... until it doesn't.
Another issue above is that you were always writing b.length bytes to the file, regardless of how many bytes had actually been read into the buffer.

Java , Sending Large Files Over Socket is Consuming too much CPU cycles and is slow [duplicate]

This question already has answers here:
Java sending and receiving file (byte[]) over sockets
(6 answers)
Closed 7 years ago.
I was experimenting with JAVA and found this question online.
Java sending and receiving file (byte[]) over sockets.
Just for curiosity i played with the code in the accepted answer, and with other code i found similar to the question. I tried the accepted answer,yes it works and is very fast. But the problem is Archive files are getting corrupted. So here is other code i tried. The downfall of my experimental code is it consume CPU cycles and takes more time than accepted answer (And i have no idea why it is happening so). So here is my code. Can somebody help me to optimize and improve this code more.
Time Taken by accepted Answer = 11ms for 4 Mb file.
Time taken by my experiment= 4 seconds for same file.
Server.java
public class Server implements Runnable {
private ServerSocket serverSocket = null;
private Socket socket = null;
private ObjectInputStream inStream = null;
public Server() {
}
#Override
public void run() {
try {
serverSocket = new ServerSocket(4445);
socket = serverSocket.accept();
DataInputStream dIn = new DataInputStream(socket.getInputStream());
OutputStream os = socket.getOutputStream();
DataOutputStream outToClient = new DataOutputStream(os);
System.out.println("Connected");
File myFile = new File("lib1.zip");
long flength = myFile.length();
System.out.println("File Length"+flength);
outToClient.writeLong(flength);
FileInputStream fis;
BufferedInputStream bis;
byte[] mybytearray = new byte[8192];
fis = new FileInputStream(myFile);
bis = new BufferedInputStream(fis);
int theByte = 0;
System.out.println("Sending " + myFile.getAbsolutePath() + "(" + myFile.length() + " bytes)");
while ((theByte = bis.read()) != -1) {
outToClient.write(theByte);
// bos.flush();
}
/*int count;
BufferedOutputStream bos= new BufferedOutputStream(os);
while ((count = bis.read(mybytearray))>0) {
bos.write(mybytearray, 0, count);
}*/
bis.close();
socket.close();
} catch (SocketException se) {
System.exit(0);
} catch (IOException e) {
e.printStackTrace();
}
}
public static void main(String[] args) {
Thread t = new Thread(new Server());
t.start();
}
}
ReceiveFile.java
public class RecieveFile {
public final static int SOCKET_PORT = 4445; // you may change this
String SERVER = "127.0.0.1"; // localhost
ArrayList<String> logmsg = new ArrayList<>();
public static void main(String[] args) {
new RecieveFile();
}
public RecieveFile() {
try (Socket sock = new Socket(SERVER, SOCKET_PORT)) {
System.out.println("Connecting...");
try (OutputStream os = sock.getOutputStream(); DataOutputStream outToServer = new DataOutputStream(os)) {
try (DataInputStream dIn = new DataInputStream(sock.getInputStream())) {
long fileLen, downData;
int bufferSize = sock.getReceiveBufferSize();
long starttime = System.currentTimeMillis();
File myFIle = new File("lib1.zip");
try (FileOutputStream fos = new FileOutputStream(myFIle); BufferedOutputStream bos = new BufferedOutputStream(fos)) {
fileLen = dIn.readLong();
/*for (long j = 0; j <= fileLen; j++) {
int tempint = is.read();
bos.write(tempint);
}*/
downData = fileLen;
int n = 0;
byte[] buf = new byte[8192];
while (fileLen > 0 && ((n = dIn.read(buf, 0, buf.length)) != -1)) {
bos.write(buf, 0, n);
fileLen -= n;
// System.out.println("Remaining "+fileLen);
}
/*while ((n = dIn.read(buf)) > 0) {
bos.write(buf, 0, n);
}*/
bos.flush();
long endtime = System.currentTimeMillis();
System.out.println("File " + myFIle.getAbsolutePath()
+ " downloaded (" + downData + " bytes read) in " + (endtime - starttime) + " ms");
}
}
}
} catch (IOException ex) {
Logger.getLogger(RecieveFile.class.getName()).log(Level.SEVERE, null, ex);
}
}
}

You're copying a byte at a time. This is slow. You're also declaring a byte array but not using it. Try this:
int count;
byte[] buffer = new byte[8192]; // or more, double or quadruple it
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}

Your solution takes a lot of time probably because you are reading a character at time, instead of all the buffer.
The solution is to use a construct similar to the linked question; the problem you got about corrupted file is really improbable, a malformed TCP packed that pass CRC check is really rare occurrence, and I would blame a bug instead. try to post the code you used. But you can add some hash check on the file and some part of it, if you are concerned about this

Here is a cleaned up version of your code, it should perform faster as it avoids single byte operations:
public class Server implements Runnable {
#Override
public void run() {
try {
ServerSocket serverSocket = new ServerSocket(4445);
Socket socket = serverSocket.accept();
OutputStream os = socket.getOutputStream();
DataOutputStream dos = new DataOutputStream(os);
File myFile = new File("lib1.zip");
long flength = myFile.length();
dos.writeLong(flength);
InputStream fis = new FileInputStream(myFile);
byte[] buf = new byte[16*1024]; // 16K
long written = 0;
while ((count = fis.read(buf))>0) {
dos.write(buf, 0, count);
written+=count;
}
if (written != flength)
System.out.println("Warning: file changed");
dos.close();
socket.close();
} catch (IOException e) {
e.printStackTrace();
System.exit();
}
}
An possible improvement would be to use NIO with channel.sendTo() but this should already have an acceptable performance. Note you do not need to use buffered streams on reading or writing as you use a larger byte array buffer anyway.
One possible improvement would be to not use the DataOutputStream for the long but poke the 8 bytes of it into the first buffer (array) write.
BTW: writing 4MB in 11ms is 390MB/s, that would be faster than most desktop disks can read and write.

Java sending file is always 8 bytes less when received

I have two problems with my java server-client file communication,
I have the CLIENT sends files over to the server and the SERVER receives the files.
My 2 issues are:
1) whenever i send a file, it is 8 bytes less (i do not know why)
2) the file transfer is not complete (with 8 bytes less) unless i close the socket, which i do not want. i want my connection to be persistent, so how can i send a EOF from the client to the server.
here is my client who sends files
public void sendFiles(String file) {
try {
File myFile = new File(file);
long length = myFile.length();
byte[] buffer = new byte[8192];
System.out.println(length);
FileInputStream in = new FileInputStream(myFile);
BufferedInputStream bis = new BufferedInputStream(in);
BufferedOutputStream outF = new BufferedOutputStream(sock.getOutputStream());
out.print("%SF%" + length + "$" + myFile.getName() + "#");
out.flush();
int count;
while ((count = in.read(buffer)) > 0) {
outF.write(buffer, 0, count);
}
outF.flush();
in.close();
bis.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
SERVER who receives files.
I'm passing the name and the length of the file but only using the name of the file. however, i don't know if i need to use the length of the file or not, in case of EOF or something. Please advice
Also, The code hangs in
while ((count = this.sock.getInputStream().read(buffer)) > 0) {
due to no EOF which i do not know how to implement
public void recvFile(String fileName, int length) {
try {
byte[] buffer = new byte[8192];
FileOutputStream outF = new FileOutputStream("/Users/Documents" +fileName);
BufferedOutputStream bos = new BufferedOutputStream(outF);
int count = length;
while ((count = this.sock.getInputStream().read(buffer)) > 0) {
bos.write(buffer, 0, count);
}
bos.close();
} catch (IOException ex) {
ex.printStackTrace();
}
}
UPDATE: I have removed the flush() as advised that it is not needed. Also, i have tested this code in a different class and it worked but it doesn't work here with client-server chat. Could anyone tell me why?
Any help or hints would be appreciated.
Thank you.

I would suggest to you send the file size first and/or properties of the file... You can try HTTP which is wide use for this task...
Another suggestion would be for you to open another connection on other TCP port just to send the file (this is actually how FTP sends files)

I suspect the problem you have is in code you haven't shown.
In this example you can send multiple messages or files over the same stream.
import java.io.*;
import java.net.ServerSocket;
import java.net.Socket;
import java.nio.channels.SocketChannel;
/**
* Created by peter on 1/25/15.
*/
public class DataSocket implements Closeable {
private final Socket socket;
private final DataOutputStream out;
private final DataInputStream in;
public DataSocket(Socket socket) throws IOException {
this.socket = socket;
this.out = new DataOutputStream(new BufferedOutputStream(socket.getOutputStream()));
this.in = new DataInputStream(new BufferedInputStream(socket.getInputStream()));
}
#Override
public void close() throws IOException {
out.flush();
socket.close();
}
// message format is length as UTF-8 encoded name, 32-bit int followed by data.
public void writeMessage(String description, byte[] bytes) throws IOException {
out.writeUTF(description);
out.writeInt(bytes.length);
out.write(bytes);
out.flush();
}
public byte[] readMessage(String[] description) throws IOException {
description[0] = in.readUTF();
int length = in.readInt();
byte[] bytes = new byte[length];
in.readFully(bytes);
return bytes;
}
public void writeFile(File file) throws IOException {
long length = file.length();
if (length > Integer.MAX_VALUE) throw new IllegalArgumentException("length=" + length);
out.writeUTF(file.toString());
out.writeInt((int) length);
byte[] buffer = new byte[(int) Math.min(length, 32 * 1024)];
try (FileInputStream fis = new FileInputStream(file)) {
for (int len; (len = fis.read(buffer)) > 0; ) {
out.write(buffer, 0, len);
}
}
out.flush();
}
public void readFile(File dir) throws IOException {
String fileName = in.readUTF();
int length = in.readInt();
byte[] buffer = new byte[(int) Math.min(length, 32 * 1024)];
try (FileOutputStream fos = new FileOutputStream(new File(dir, fileName))) {
while (length > 0) {
int len = in.read(buffer);
fos.write(buffer, 0, len);
length -= len;
}
}
}
// todo convert to a unit test
public static void main(String[] args) throws IOException {
// port 0 opens on a random free port.
ServerSocket sc = new ServerSocket(0);
DataSocket ds1 = new DataSocket(new Socket("localhost", sc.getLocalPort()));
DataSocket ds2 = new DataSocket(sc.accept());
sc.close();
// now ds1 and ds2 are connected.
File f = File.createTempFile("deleteme","");
f.deleteOnExit();
try (FileOutputStream fos = new FileOutputStream(f)) {
fos.write(new byte[10001]);
}
// send a request
ds1.writeMessage("Send me the file", new byte[0]);
String[] desc = { null };
byte[] data = ds2.readMessage(desc);
if (!desc[0].equals("Send me the file")) throw new AssertionError();
// return a response
ds2.writeFile(f);
f.delete();
if (f.exists()) throw new AssertionError();
ds1.readFile(new File(""));
if (f.length() != 10001) throw new AssertionError("length="+f.length());
ds1.close();
ds2.close();
System.out.println("Copied a "+f.length()+" file over TCP");
}
}

Large file transfer over java socket [duplicate]

This question already has answers here:
Java multiple file transfer over socket
(3 answers)
Closed 5 years ago.
I have written a small client-server code for transferring small file. It uses Data output stream and readFully() method of data input stream. This code does not work for larger files for obvious reasons. I was thinking of fragmenting large files into smaller chunks of 1Kb each before sending them to client. But I can't think of any solution (like how to write multiple chunks on data output stream with correct offset and how to reassemble them at receiving end. Can anyone provide a workaround? It would be very helpful if you could modify my code:
Sender (Server):
public void sendFileDOS() throws FileNotFoundException {
runOnUiThread( new Runnable() {
#Override
public void run() {
registerLog("Sending. . . Please wait. . .");
}
});
final long startTime = System.currentTimeMillis();
final File myFile= new File(filePath); //sdcard/DCIM.JPG
byte[] mybytearray = new byte[(int) myFile.length()];
FileInputStream fis = new FileInputStream(myFile);
BufferedInputStream bis = new BufferedInputStream(fis);
DataInputStream dis = new DataInputStream(bis);
try {
dis.readFully(mybytearray, 0, mybytearray.length);
OutputStream os = socket.getOutputStream();
//Sending file name and file size to the client
DataOutputStream dos = new DataOutputStream(os);
dos.writeUTF(myFile.getName());
dos.writeLong(mybytearray.length);
int i = 0;
final ProgressBar myProgBar=(ProgressBar)findViewById(R.id.progress_bar);
while (i<100) {
dos.write(mybytearray, i*(mybytearray.length/100), mybytearray.length/100);
final int c=i;
runOnUiThread( new Runnable() {
#Override
public void run() {
myProgBar.setVisibility(View.VISIBLE);
registerLog("Completed: "+c+"%");
myProgBar.setProgress(c);
if (c==99)
myProgBar.setVisibility(View.INVISIBLE);
}
});
i++;
}
dos.flush();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
runOnUiThread( new Runnable() {
#Override
public void run() {
long estimatedTime = (System.currentTimeMillis() - startTime)/1000;
registerLog("File successfully sent");
registerLog("File size: "+myFile.length()/1000+" KBytes");
registerLog("Elapsed time: "+estimatedTime+" sec. (approx)");
registerLog("Server stopped. Please restart for another session.");
final Button startServerButton=(Button)findViewById(R.id.button1);
startServerButton.setText("Restart file server");
}
});
}
Receiver (Client):
public class myFileClient {
final static String servAdd="10.141.21.145";
static String filename=null;
static Socket socket = null;
static Boolean flag=true;
/**
* #param args
*/
public static void main(String[] args) throws IOException {
// TODO Auto-generated method stub
initializeClient();
receiveDOS();
}
public static void initializeClient () throws IOException {
InetAddress serverIP=InetAddress.getByName(servAdd);
socket=new Socket(serverIP, 4444);
}
public static void receiveDOS() {
int bytesRead;
InputStream in;
int bufferSize=0;
try {
bufferSize=socket.getReceiveBufferSize();
in=socket.getInputStream();
DataInputStream clientData = new DataInputStream(in);
String fileName = clientData.readUTF();
System.out.println(fileName);
OutputStream output = new FileOutputStream("//home//evinish//Documents//Android//Received files//"+ fileName);
long size = clientData.readLong();
byte[] buffer = new byte[bufferSize];
while (size > 0
&& (bytesRead = clientData.read(buffer, 0,
(int) Math.min(buffer.length, size))) != -1) {
output.write(buffer, 0, bytesRead);
size -= bytesRead;
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Please help! Thanks in advance! :)

You're right, this is a poor way to do it. It wastes both memory and time; it assumes the file size is 32 bits; it assumes the entire file fits into memory; it assumes the entire file is read in one read; and it doesn't send anything until the entire file has been read.
The canonical way to copy a stream in Java is this:
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
It will work with any size buffer you like and therefore with any size file you can come up with. Use the same code at both ends, although you don't have to use the same size buffer at both ends. As you're copying over a network you might think that 1k or 1.5k is the best size, but that overlooks the presence of the socket send and receive buffers in the kernel. When you take them into account it is probably better to use 8k or more.

I finally solved the problem. Here is my modified source code for server and client. Hope this would help other people too! :)
Server Side code snippet (sender):
final File myFile= new File(filePath); //sdcard/DCIM.JPG
byte[] mybytearray = new byte[8192];
FileInputStream fis = new FileInputStream(myFile);
BufferedInputStream bis = new BufferedInputStream(fis);
DataInputStream dis = new DataInputStream(bis);
OutputStream os;
try {
os = socket.getOutputStream();
DataOutputStream dos = new DataOutputStream(os);
dos.writeUTF(myFile.getName());
dos.writeLong(mybytearray.length);
int read;
while((read = dis.read(mybytearray)) != -1){
dos.write(mybytearray, 0, read);
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
Client side code snippet (Receiver):
int bytesRead;
InputStream in;
int bufferSize=0;
try {
bufferSize=socket.getReceiveBufferSize();
in=socket.getInputStream();
DataInputStream clientData = new DataInputStream(in);
String fileName = clientData.readUTF();
System.out.println(fileName);
OutputStream output = new FileOutputStream("//home//evinish//Documents//Android//Received files//"+ fileName);
byte[] buffer = new byte[bufferSize];
int read;
while((read = clientData.read(buffer)) != -1){
output.write(buffer, 0, read);
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
A bit faster way of writing to output stream:
long acc=0;
long N=myFile.length();
while(acc<N){
noofbytes=dis.read(mybytearray, 0, 16384);
dos.write(mybytearray, 0, noofbytes);
acc=acc+noofbytes; } dos.flush();
I saved around 7 seconds while transferring a video file of 72MB.

Saving image from a link

I was trying to save a image from a link in a website I have written this code but this does not work ..plz help me to do this
public void imageshow(String linkText) {
try {
URL url = new URL(linkText);
InputStream in = new BufferedInputStream(url.openStream());
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] buf = new byte[4 * 1024];
int n = 0;
while (-1 != (n = in.read(buf))) {
out.write(buf, 0, n);
}
out.close();
in.close();
byte[] response = out.toByteArray();
FileOutputStream fos = new FileOutputStream("C://chart.gif");
fos.write(response);
fos.close();
} catch (Exception e) {
}
}

Taken from here
import java.net.*;
import java.io.*;
public class DownloadFile
{
public static void main(String[] args)
{
try
{
/*
* Get a connection to the URL and start up
* a buffered reader.
*/
long startTime = System.currentTimeMillis();
System.out.println("Connecting to Mura site...\n");
URL url = new URL("http://www.getmura.com/currentversion/");
url.openConnection();
InputStream reader = url.openStream();
/*
* Setup a buffered file writer to write
* out what we read from the website.
*/
FileOutputStream writer = new FileOutputStream("C:/mura-newest.zip");
byte[] buffer = new byte[153600];
int totalBytesRead = 0;
int bytesRead = 0;
System.out.println("Reading ZIP file 150KB blocks at a time.\n");
while ((bytesRead = reader.read(buffer)) > 0)
{
writer.write(buffer, 0, bytesRead);
buffer = new byte[153600];
totalBytesRead += bytesRead;
}
long endTime = System.currentTimeMillis();
System.out.println("Done. " + (new Integer(totalBytesRead).toString()) + " bytes read (" + (new Long(endTime - startTime).toString()) + " millseconds).\n");
writer.close();
reader.close();
}
catch (MalformedURLException e)
{
e.printStackTrace();
}
catch (IOException e)
{
e.printStackTrace();
}
}
}