I know that the canonical way of handling the output/error streams of an external Process in Java, is to use two extra-threads in order to pull the data from the output and error streams lest the process might be blocked.
Now what about the following?
public static void main(String[] args) throws IOException, InterruptedException {
ProcessBuilder processBuilder = new ProcessBuilder(args);
Process process = processBuilder.start();
InputStream outputStream = null, errorStream = null;
ByteArrayOutputStream outputBuffer = new ByteArrayOutputStream();
ByteArrayOutputStream errorBuffer = new ByteArrayOutputStream();
try {
outputStream = process.getInputStream();
errorStream = process.getErrorStream();
byte[] tmp = new byte[1024];
while (true) {
int outputBytes = readAvailablOnce(outputStream, outputBuffer, tmp);
int errorBytes = readAvailablOnce(errorStream, errorBuffer, tmp);
if (outputBytes == 0 && errorBytes == 0) {
try {
process.exitValue();
break;
} catch (IllegalThreadStateException e) {
// keep on looping
}
}
}
readAvailableAll(outputStream, outputBuffer, tmp);
readAvailableAll(errorStream, errorBuffer, tmp);
} finally {
closeQuietly(outputStream);
closeQuietly(errorStream);
}
System.out.println(outputBuffer.toString("ASCII"));
System.err.println(errorBuffer.toString("ASCII"));
System.err.println("exit code: " + process.exitValue());
}
private static void closeQuietly(InputStream in) {
if (in != null) {
try {
in.close();
} catch (IOException e) {
// ignored
}
}
}
private static int readAvailablOnce(
InputStream inputStream, OutputStream outputStream, byte[] buffer)
throws IOException {
int bytesRead = 0;
if (inputStream.available() > 0) {
bytesRead = inputStream.read(buffer);
outputStream.write(buffer, 0, bytesRead);
}
return bytesRead;
}
private static void readAvailableAll(
InputStream inputStream, OutputStream outputStream, byte[] buffer)
throws IOException {
if (inputStream.available() > 0) {
int bytesRead = 0;
while ((bytesRead = inputStream.read(buffer)) >= 0) {
outputStream.write(buffer, 0, bytesRead);
}
}
}
This actually runs fine in the few examples I tried (running "dir", "ps aux", etc).
It also has the disadvantage of not allowing you to easily handle the outputs line by line (here you buffer everything before doing anything), unless you start doing some more or less complicated stuff with Buffers and CharsetDecoders.
Still, it looks useful for anything without unacceptably big output (although nothing forces us to buffer the whole output before using it).
I only tried the 1.5 and 1.6 JVM (Windows XP and Linux).
Also, this code makes the assumption that the final output bits of the process will be readily available for reading (InputStream.available() > 0).
Anyone would know what (or if something) is wrong with this code (or have a better idea)?
I would use ProcessBuilder.redirectErrorStream(true) which allows you to read one stream in the current thread. You wouldn't need any background threads.
Related
I have a log analyzing tool that needs to grab *.gz files from Linux servers and unzip them on both Linux and Windows clients. I am getting "Unexpected end of ZLIB input stream" in many instances, which I assume is a difference in detail in the files on Linux and Windows.
Below is my function. It's pretty basic. How do I improved it to prevent the EOF error?
The "in" symbol is a FileInputStream that is created when constructing the class that this function is part of.
public void unzip(File fileTo) throws IOException {
OutputStream out = new FileOutputStream(fileTo);
LOGGER.info("Setting up the file for outputstream : "+fileTo);
try {
in = new GZIPInputStream(in);
byte[] buffer = new byte[65536];
int noRead;
while ((noRead = in.read(buffer)) != -1) {
out.write(buffer, 0, noRead);
}
} finally {
try { out.close(); } catch (Exception e) {}
}
}
I changed from the above to this and now it works. It seems that it was trying to load the output stream before it was done loading the input stream.
public void unzip(File fileTo, String f) throws IOException,
EOFException, InterruptedException {
LOGGER.info("Setting up the file for outputstream : "+fileTo);
GZIPInputStream cIn = new GZIPInputStream(new FileInputStream(f));
OutputStream out = new FileOutputStream(fileTo);
fileTo.setReadable(true, false);
fileTo.setWritable(true, false);
byte[] buffer = new byte[65536];
int noRead;
for (int i = 10; i > 0 && cIn.available() == 1; i--) {
Thread.sleep(1000);
}
try {
while ((noRead = cIn.read(buffer)) != -1) {
out.write(buffer, 0, noRead);
}
} finally {
try { out.close();cIn.close();in.close(); } catch (Exception e) {}
}
}
I am learning sockets and now I want to write file transfer program. I have server part and client part. Server part contains 2 ports: 5000 (commands) and 5001 (files). Now I want to send a file via socket and when I did something is wrong because only 425B of data is sending.
Here is client send method:
private void sendFile(Socket socket) {
File file2 = new File("C:\\Users\\barte\\Desktop\\dos.png");
byte[] bytes = new byte[16 * 1024];
System.out.println(file2.exists());
try (InputStream inputStream = new FileInputStream(file2);
OutputStream outputStream = socket.getOutputStream();
OutputStream secondOutput = new FileOutputStream("C:\\Users\\barte\\Desktop\\received\\dos.png")) {
int count;
while ((count = inputStream.read(bytes)) > 0) {
outputStream.write(bytes, 0, count);
secondOutput.write(bytes, 0, count);
}
socket.close();
} catch (IOException e) {
e.printStackTrace();
}
}
As you can see (image below) I am writing this file also locally and everything is ok, all of 73KB of data is writed.
Now, on server side I am trying to receive this file:
case SEND: {
new Thread(() -> {
printWriter.println("Server is receiving files right now...");
try (ServerSocket serverSocket = new ServerSocket(5001)) {
while (true) {
new FilesTransfer(serverSocket.accept()).start();
}
} catch (IOException e) {
e.printStackTrace();
}
}).start();
break;
}
And inside FilesTransfer run method:
#Override
public void run() {
System.out.println("Hello there");
try {
InputStream inputStream = inSocket.getInputStream();
OutputStream outputStream = new FileOutputStream("C:\\Users\\barte\\Desktop\\received\\file");
byte[] bytes = new byte[16 * 1024];
int count;
while ((count = inputStream.read()) > 0) {
outputStream.write(bytes, 0, count);
}
outputStream.close();
inputStream.close();
inSocket.close();
} catch (IOException e) {
e.printStackTrace();
}
}
Where is a bug? Why only empty bytes are sending when locally everything it's fine?
The problem is:
while ((count = inputStream.read()) > 0) {
Your code uses InputStream.read(), which reads individual bytes (or -1 when end-of-stream). Right now, you are reading individual bytes, interpreting that as a length, and then writing that number of 0x00 bytes from bytes to the file. This stops when you read a 0x00 byte from the stream.
You need to change this to use InputStream.read(byte[]):
while ((count = inputStream.read(bytes)) != -1) {
That is, you need to pass bytes in, and check for the result being unequal to -1, not if it is greater than zero (0), although read(byte[]) will only return 0 if the passed in byte array has length zero, so that is not a real concern.
You could do it in this way:
#Override
public void run() {
System.out.println("Hello there");
try {
InputStream inputStream = inSocket.getInputStream();
OutputStream outputStream = new FileOutputStream("C:\\Users\\barte\\Desktop\\received\\file");
byte[] bytes = new byte[16 * 1024];
int byteRead= 1;
while (byteRead > -1) {
byteRead= inputStream.read();
outputStream.write(byteRead);
}
outputStream.close();
inputStream.close();
inSocket.close();
} catch (IOException e) {
e.printStackTrace();
}
}
Actually END OF FILE or EOF means -1 and you did > 0 so 0 was taken and it stopped the connection saving the file.
I also recommend to write a logic to transfer the filename as a command to the server so that the file is saved with the correct name and extension!
I have two problems with my java server-client file communication,
I have the CLIENT sends files over to the server and the SERVER receives the files.
My 2 issues are:
1) whenever i send a file, it is 8 bytes less (i do not know why)
2) the file transfer is not complete (with 8 bytes less) unless i close the socket, which i do not want. i want my connection to be persistent, so how can i send a EOF from the client to the server.
here is my client who sends files
public void sendFiles(String file) {
try {
File myFile = new File(file);
long length = myFile.length();
byte[] buffer = new byte[8192];
System.out.println(length);
FileInputStream in = new FileInputStream(myFile);
BufferedInputStream bis = new BufferedInputStream(in);
BufferedOutputStream outF = new BufferedOutputStream(sock.getOutputStream());
out.print("%SF%" + length + "$" + myFile.getName() + "#");
out.flush();
int count;
while ((count = in.read(buffer)) > 0) {
outF.write(buffer, 0, count);
}
outF.flush();
in.close();
bis.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
SERVER who receives files.
I'm passing the name and the length of the file but only using the name of the file. however, i don't know if i need to use the length of the file or not, in case of EOF or something. Please advice
Also, The code hangs in
while ((count = this.sock.getInputStream().read(buffer)) > 0) {
due to no EOF which i do not know how to implement
public void recvFile(String fileName, int length) {
try {
byte[] buffer = new byte[8192];
FileOutputStream outF = new FileOutputStream("/Users/Documents" +fileName);
BufferedOutputStream bos = new BufferedOutputStream(outF);
int count = length;
while ((count = this.sock.getInputStream().read(buffer)) > 0) {
bos.write(buffer, 0, count);
}
bos.close();
} catch (IOException ex) {
ex.printStackTrace();
}
}
UPDATE: I have removed the flush() as advised that it is not needed. Also, i have tested this code in a different class and it worked but it doesn't work here with client-server chat. Could anyone tell me why?
Any help or hints would be appreciated.
Thank you.
I would suggest to you send the file size first and/or properties of the file... You can try HTTP which is wide use for this task...
Another suggestion would be for you to open another connection on other TCP port just to send the file (this is actually how FTP sends files)
I suspect the problem you have is in code you haven't shown.
In this example you can send multiple messages or files over the same stream.
import java.io.*;
import java.net.ServerSocket;
import java.net.Socket;
import java.nio.channels.SocketChannel;
/**
* Created by peter on 1/25/15.
*/
public class DataSocket implements Closeable {
private final Socket socket;
private final DataOutputStream out;
private final DataInputStream in;
public DataSocket(Socket socket) throws IOException {
this.socket = socket;
this.out = new DataOutputStream(new BufferedOutputStream(socket.getOutputStream()));
this.in = new DataInputStream(new BufferedInputStream(socket.getInputStream()));
}
#Override
public void close() throws IOException {
out.flush();
socket.close();
}
// message format is length as UTF-8 encoded name, 32-bit int followed by data.
public void writeMessage(String description, byte[] bytes) throws IOException {
out.writeUTF(description);
out.writeInt(bytes.length);
out.write(bytes);
out.flush();
}
public byte[] readMessage(String[] description) throws IOException {
description[0] = in.readUTF();
int length = in.readInt();
byte[] bytes = new byte[length];
in.readFully(bytes);
return bytes;
}
public void writeFile(File file) throws IOException {
long length = file.length();
if (length > Integer.MAX_VALUE) throw new IllegalArgumentException("length=" + length);
out.writeUTF(file.toString());
out.writeInt((int) length);
byte[] buffer = new byte[(int) Math.min(length, 32 * 1024)];
try (FileInputStream fis = new FileInputStream(file)) {
for (int len; (len = fis.read(buffer)) > 0; ) {
out.write(buffer, 0, len);
}
}
out.flush();
}
public void readFile(File dir) throws IOException {
String fileName = in.readUTF();
int length = in.readInt();
byte[] buffer = new byte[(int) Math.min(length, 32 * 1024)];
try (FileOutputStream fos = new FileOutputStream(new File(dir, fileName))) {
while (length > 0) {
int len = in.read(buffer);
fos.write(buffer, 0, len);
length -= len;
}
}
}
// todo convert to a unit test
public static void main(String[] args) throws IOException {
// port 0 opens on a random free port.
ServerSocket sc = new ServerSocket(0);
DataSocket ds1 = new DataSocket(new Socket("localhost", sc.getLocalPort()));
DataSocket ds2 = new DataSocket(sc.accept());
sc.close();
// now ds1 and ds2 are connected.
File f = File.createTempFile("deleteme","");
f.deleteOnExit();
try (FileOutputStream fos = new FileOutputStream(f)) {
fos.write(new byte[10001]);
}
// send a request
ds1.writeMessage("Send me the file", new byte[0]);
String[] desc = { null };
byte[] data = ds2.readMessage(desc);
if (!desc[0].equals("Send me the file")) throw new AssertionError();
// return a response
ds2.writeFile(f);
f.delete();
if (f.exists()) throw new AssertionError();
ds1.readFile(new File(""));
if (f.length() != 10001) throw new AssertionError("length="+f.length());
ds1.close();
ds2.close();
System.out.println("Copied a "+f.length()+" file over TCP");
}
}
This question already has answers here:
Java multiple file transfer over socket
(3 answers)
Closed 5 years ago.
I have written a small client-server code for transferring small file. It uses Data output stream and readFully() method of data input stream. This code does not work for larger files for obvious reasons. I was thinking of fragmenting large files into smaller chunks of 1Kb each before sending them to client. But I can't think of any solution (like how to write multiple chunks on data output stream with correct offset and how to reassemble them at receiving end. Can anyone provide a workaround? It would be very helpful if you could modify my code:
Sender (Server):
public void sendFileDOS() throws FileNotFoundException {
runOnUiThread( new Runnable() {
#Override
public void run() {
registerLog("Sending. . . Please wait. . .");
}
});
final long startTime = System.currentTimeMillis();
final File myFile= new File(filePath); //sdcard/DCIM.JPG
byte[] mybytearray = new byte[(int) myFile.length()];
FileInputStream fis = new FileInputStream(myFile);
BufferedInputStream bis = new BufferedInputStream(fis);
DataInputStream dis = new DataInputStream(bis);
try {
dis.readFully(mybytearray, 0, mybytearray.length);
OutputStream os = socket.getOutputStream();
//Sending file name and file size to the client
DataOutputStream dos = new DataOutputStream(os);
dos.writeUTF(myFile.getName());
dos.writeLong(mybytearray.length);
int i = 0;
final ProgressBar myProgBar=(ProgressBar)findViewById(R.id.progress_bar);
while (i<100) {
dos.write(mybytearray, i*(mybytearray.length/100), mybytearray.length/100);
final int c=i;
runOnUiThread( new Runnable() {
#Override
public void run() {
myProgBar.setVisibility(View.VISIBLE);
registerLog("Completed: "+c+"%");
myProgBar.setProgress(c);
if (c==99)
myProgBar.setVisibility(View.INVISIBLE);
}
});
i++;
}
dos.flush();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
runOnUiThread( new Runnable() {
#Override
public void run() {
long estimatedTime = (System.currentTimeMillis() - startTime)/1000;
registerLog("File successfully sent");
registerLog("File size: "+myFile.length()/1000+" KBytes");
registerLog("Elapsed time: "+estimatedTime+" sec. (approx)");
registerLog("Server stopped. Please restart for another session.");
final Button startServerButton=(Button)findViewById(R.id.button1);
startServerButton.setText("Restart file server");
}
});
}
Receiver (Client):
public class myFileClient {
final static String servAdd="10.141.21.145";
static String filename=null;
static Socket socket = null;
static Boolean flag=true;
/**
* #param args
*/
public static void main(String[] args) throws IOException {
// TODO Auto-generated method stub
initializeClient();
receiveDOS();
}
public static void initializeClient () throws IOException {
InetAddress serverIP=InetAddress.getByName(servAdd);
socket=new Socket(serverIP, 4444);
}
public static void receiveDOS() {
int bytesRead;
InputStream in;
int bufferSize=0;
try {
bufferSize=socket.getReceiveBufferSize();
in=socket.getInputStream();
DataInputStream clientData = new DataInputStream(in);
String fileName = clientData.readUTF();
System.out.println(fileName);
OutputStream output = new FileOutputStream("//home//evinish//Documents//Android//Received files//"+ fileName);
long size = clientData.readLong();
byte[] buffer = new byte[bufferSize];
while (size > 0
&& (bytesRead = clientData.read(buffer, 0,
(int) Math.min(buffer.length, size))) != -1) {
output.write(buffer, 0, bytesRead);
size -= bytesRead;
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Please help! Thanks in advance! :)
You're right, this is a poor way to do it. It wastes both memory and time; it assumes the file size is 32 bits; it assumes the entire file fits into memory; it assumes the entire file is read in one read; and it doesn't send anything until the entire file has been read.
The canonical way to copy a stream in Java is this:
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
It will work with any size buffer you like and therefore with any size file you can come up with. Use the same code at both ends, although you don't have to use the same size buffer at both ends. As you're copying over a network you might think that 1k or 1.5k is the best size, but that overlooks the presence of the socket send and receive buffers in the kernel. When you take them into account it is probably better to use 8k or more.
I finally solved the problem. Here is my modified source code for server and client. Hope this would help other people too! :)
Server Side code snippet (sender):
final File myFile= new File(filePath); //sdcard/DCIM.JPG
byte[] mybytearray = new byte[8192];
FileInputStream fis = new FileInputStream(myFile);
BufferedInputStream bis = new BufferedInputStream(fis);
DataInputStream dis = new DataInputStream(bis);
OutputStream os;
try {
os = socket.getOutputStream();
DataOutputStream dos = new DataOutputStream(os);
dos.writeUTF(myFile.getName());
dos.writeLong(mybytearray.length);
int read;
while((read = dis.read(mybytearray)) != -1){
dos.write(mybytearray, 0, read);
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
Client side code snippet (Receiver):
int bytesRead;
InputStream in;
int bufferSize=0;
try {
bufferSize=socket.getReceiveBufferSize();
in=socket.getInputStream();
DataInputStream clientData = new DataInputStream(in);
String fileName = clientData.readUTF();
System.out.println(fileName);
OutputStream output = new FileOutputStream("//home//evinish//Documents//Android//Received files//"+ fileName);
byte[] buffer = new byte[bufferSize];
int read;
while((read = clientData.read(buffer)) != -1){
output.write(buffer, 0, read);
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
A bit faster way of writing to output stream:
long acc=0;
long N=myFile.length();
while(acc<N){
noofbytes=dis.read(mybytearray, 0, 16384);
dos.write(mybytearray, 0, noofbytes);
acc=acc+noofbytes; } dos.flush();
I saved around 7 seconds while transferring a video file of 72MB.
The following code doesn't work to download a file (btw clen is file's length):
int pos = 0, total_pos = 0;
byte[] buffer = new byte[BUFFER_SIZE];
while (pos != -1) {
pos = in.read(buffer, 0, BUFFER_SIZE);
total_pos += pos;
out.write(buffer);
setProgress((int) (total_pos * 100 / clen));
}
...but this works fine:
int buf;
while ((buf = in.read()) != -1)
out.write(buf);
I'm wondering why, even though the second code segment works quickly. On that note, is there any particular reason to use a byte[] buffer (since it doesn't seem to be faster, and BufferedInputStream already uses a buffer of its own....?)
Here's how it should be done.
public static void copyStream(InputStream is, OutputStream os)
{
byte[] buff = new byte[4096];
int count;
try {
while((count = is.read(buff)) > 0)
os.write(buff, 0, count);
}catch (Exception e) {
e.printStackTrace();
}finally {
try {
if(is != null)
is.close();
} catch (IOException e) {
e.printStackTrace();
}
try {
if(os != null)
os.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
I've tried to make the minimum changes necessary to your code to get it working. st0le did a good job of providing a neater version of stream copying.
public class Test {
private static final String FORMAT = "UTF-8";
private static final int BUFFER_SIZE = 10; // for demonstration purposes.
public static void main(String[] args) throws Exception {
String string = "This is a test of the public broadcast system";
int clen = string.length();
ByteArrayInputStream in = new ByteArrayInputStream(string.getBytes(FORMAT));
OutputStream out = System.out;
int pos = 0, total_pos = 0;
byte[] buffer = new byte[BUFFER_SIZE];
while (pos != -1) {
pos = in.read(buffer, 0, BUFFER_SIZE);
if (pos > 0) {
total_pos += pos;
out.write(buffer, 0, pos);
setProgress((int) (total_pos * 100 / clen));
}
}
}
private static void setProgress(int i) {
}
}
You were ignoring the value of pos when you were writing out the buffer to the output stream.
You also need to re-check the value of pos because it may have just read the end of the file. You don't increment the total_pos in that case (although you should probably report that you are 100% complete)
Be sure to handle your resources correctly with close()s in the appropriate places.
-edit-
The general reason for using an array as a buffer is so that the output stream can do as much work as it can with a larger set of data.
Writing to a console there might not be much of a delay, but it might be a network socket being written to or some other slow device. As the JavaDoc states
The write method of OutputStream calls the write method of one argument on each of the bytes to be written out. Subclasses are encouraged to override this method and provide a more efficient implementation.
The benefit of using it when using a Buffered Input/Output Stream are probably minimal.