Specifically, the problem is to write a method like this:
int maybeRead(InputStream in, long timeout)
where the return value is the same as in.read() if data is available within 'timeout' milliseconds, and -2 otherwise. Before the method returns, any spawned threads must exit.
To avoid arguments, the subject here java.io.InputStream, as documented by Sun (any Java version). Please note this is not as simple as it looks. Below are some facts which are supported directly by Sun's documentation.
The in.read() method may be non-interruptible.
Wrapping the InputStream in a Reader or InterruptibleChannel doesn't help, because all those classes can do is call methods of the InputStream. If it were possible to use those classes, it would be possible to write a solution that just executes the same logic directly on the InputStream.
It is always acceptable for in.available() to return 0.
The in.close() method may block or do nothing.
There is no general way to kill another thread.
Using inputStream.available()
It is always acceptable for System.in.available() to return 0.
I've found the opposite - it always returns the best value for the number of bytes available. Javadoc for InputStream.available():
Returns an estimate of the number of bytes that can be read (or skipped over)
from this input stream without blocking by the next invocation of a method for
this input stream.
An estimate is unavoidable due to timing/staleness. The figure can be a one-off underestimate because new data are constantly arriving. However it always "catches up" on the next call - it should account for all arrived data, bar that arriving just at the moment of the new call. Permanently returning 0 when there are data fails the condition above.
First Caveat: Concrete subclasses of InputStream are responsible for available()
InputStream is an abstract class. It has no data source. It's meaningless for it to have available data. Hence, javadoc for available() also states:
The available method for class InputStream always returns 0.
This method should be overridden by subclasses.
And indeed, the concrete input stream classes do override available(), providing meaningful values, not constant 0s.
Second Caveat: Ensure you use carriage-return when typing input in Windows.
If using System.in, your program only receives input when your command shell hands it over. If you're using file redirection/pipes (e.g. somefile > java myJavaApp or somecommand | java myJavaApp ), then input data are usually handed over immediately. However, if you manually type input, then data handover can be delayed. E.g. With windows cmd.exe shell, the data are buffered within cmd.exe shell. Data are only passed to the executing java program following carriage-return (control-m or <enter>). That's a limitation of the execution environment. Of course, InputStream.available() will return 0 for as long as the shell buffers the data - that's correct behaviour; there are no available data at that point. As soon as the data are available from the shell, the method returns a value > 0. NB: Cygwin uses cmd.exe too.
Simplest solution (no blocking, so no timeout required)
Just use this:
byte[] inputData = new byte[1024];
int result = is.read(inputData, 0, is.available());
// result will indicate number of bytes read; -1 for EOF with no data read.
OR equivalently,
BufferedReader br = new BufferedReader(new InputStreamReader(System.in, Charset.forName("ISO-8859-1")),1024);
// ...
// inside some iteration / processing logic:
if (br.ready()) {
int readCount = br.read(inputData, bufferOffset, inputData.length-bufferOffset);
}
Richer Solution (maximally fills buffer within timeout period)
Declare this:
public static int readInputStreamWithTimeout(InputStream is, byte[] b, int timeoutMillis)
throws IOException {
int bufferOffset = 0;
long maxTimeMillis = System.currentTimeMillis() + timeoutMillis;
while (System.currentTimeMillis() < maxTimeMillis && bufferOffset < b.length) {
int readLength = java.lang.Math.min(is.available(),b.length-bufferOffset);
// can alternatively use bufferedReader, guarded by isReady():
int readResult = is.read(b, bufferOffset, readLength);
if (readResult == -1) break;
bufferOffset += readResult;
}
return bufferOffset;
}
Then use this:
byte[] inputData = new byte[1024];
int readCount = readInputStreamWithTimeout(System.in, inputData, 6000); // 6 second timeout
// readCount will indicate number of bytes read; -1 for EOF with no data read.
Assuming your stream is not backed by a socket (so you can't use Socket.setSoTimeout()), I think the standard way of solving this type of problem is to use a Future.
Suppose I have the following executor and streams:
ExecutorService executor = Executors.newFixedThreadPool(2);
final PipedOutputStream outputStream = new PipedOutputStream();
final PipedInputStream inputStream = new PipedInputStream(outputStream);
I have writer that writes some data then waits for 5 seconds before writing the last piece of data and closing the stream:
Runnable writeTask = new Runnable() {
#Override
public void run() {
try {
outputStream.write(1);
outputStream.write(2);
Thread.sleep(5000);
outputStream.write(3);
outputStream.close();
} catch (Exception e) {
e.printStackTrace();
}
}
};
executor.submit(writeTask);
The normal way of reading this is as follows. The read will block indefinitely for data and so this completes in 5s:
long start = currentTimeMillis();
int readByte = 1;
// Read data without timeout
while (readByte >= 0) {
readByte = inputStream.read();
if (readByte >= 0)
System.out.println("Read: " + readByte);
}
System.out.println("Complete in " + (currentTimeMillis() - start) + "ms");
which outputs:
Read: 1
Read: 2
Read: 3
Complete in 5001ms
If there was a more fundamental problem, like the writer not responding, the reader would block for ever.
If I wrap the read in a future, I can then control the timeout as follows:
int readByte = 1;
// Read data with timeout
Callable<Integer> readTask = new Callable<Integer>() {
#Override
public Integer call() throws Exception {
return inputStream.read();
}
};
while (readByte >= 0) {
Future<Integer> future = executor.submit(readTask);
readByte = future.get(1000, TimeUnit.MILLISECONDS);
if (readByte >= 0)
System.out.println("Read: " + readByte);
}
which outputs:
Read: 1
Read: 2
Exception in thread "main" java.util.concurrent.TimeoutException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:228)
at java.util.concurrent.FutureTask.get(FutureTask.java:91)
at test.InputStreamWithTimeoutTest.main(InputStreamWithTimeoutTest.java:74)
I can catch the TimeoutException and do whatever cleanup I want.
If your InputStream is backed by a Socket, you can set a Socket timeout (in milliseconds) using setSoTimeout. If the read() call doesn't unblock within the timeout specified, it will throw a SocketTimeoutException.
Just make sure that you call setSoTimeout on the Socket before making the read() call.
I would question the problem statement rather than just accept it blindly. You only need timeouts from the console or over the network. If the latter you have Socket.setSoTimeout() and HttpURLConnection.setReadTimeout() which both do exactly what is required, as long as you set them up correctly when you construct/acquire them. Leaving it to an arbitrary point later in the application when all you have is the InputStream is poor design leading to a very awkward implementation.
I have not used the classes from the Java NIO package, but it seems they might be of some help here. Specifically, java.nio.channels.Channels and java.nio.channels.InterruptibleChannel.
Here is a way to get a NIO FileChannel from System.in and check for availability of data using a timeout, which is a special case of the problem described in the question. Run it at the console, don't type any input, and wait for the results. It was tested successfully under Java 6 on Windows and Linux.
import java.io.FileInputStream;
import java.io.FilterInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.lang.reflect.Field;
import java.nio.ByteBuffer;
import java.nio.channels.ClosedByInterruptException;
public class Main {
static final ByteBuffer buf = ByteBuffer.allocate(4096);
public static void main(String[] args) {
long timeout = 1000 * 5;
try {
InputStream in = extract(System.in);
if (! (in instanceof FileInputStream))
throw new RuntimeException(
"Could not extract a FileInputStream from STDIN.");
try {
int ret = maybeAvailable((FileInputStream)in, timeout);
System.out.println(
Integer.toString(ret) + " bytes were read.");
} finally {
in.close();
}
} catch (Exception e) {
throw new RuntimeException(e);
}
}
/* unravels all layers of FilterInputStream wrappers to get to the
* core InputStream
*/
public static InputStream extract(InputStream in)
throws NoSuchFieldException, IllegalAccessException {
Field f = FilterInputStream.class.getDeclaredField("in");
f.setAccessible(true);
while( in instanceof FilterInputStream )
in = (InputStream)f.get((FilterInputStream)in);
return in;
}
/* Returns the number of bytes which could be read from the stream,
* timing out after the specified number of milliseconds.
* Returns 0 on timeout (because no bytes could be read)
* and -1 for end of stream.
*/
public static int maybeAvailable(final FileInputStream in, long timeout)
throws IOException, InterruptedException {
final int[] dataReady = {0};
final IOException[] maybeException = {null};
final Thread reader = new Thread() {
public void run() {
try {
dataReady[0] = in.getChannel().read(buf);
} catch (ClosedByInterruptException e) {
System.err.println("Reader interrupted.");
} catch (IOException e) {
maybeException[0] = e;
}
}
};
Thread interruptor = new Thread() {
public void run() {
reader.interrupt();
}
};
reader.start();
for(;;) {
reader.join(timeout);
if (!reader.isAlive())
break;
interruptor.start();
interruptor.join(1000);
reader.join(1000);
if (!reader.isAlive())
break;
System.err.println("We're hung");
System.exit(1);
}
if ( maybeException[0] != null )
throw maybeException[0];
return dataReady[0];
}
}
Interestingly, when running the program inside NetBeans 6.5 rather than at the console, the timeout doesn't work at all, and the call to System.exit() is actually necessary to kill the zombie threads. What happens is that the interruptor thread blocks (!) on the call to reader.interrupt(). Another test program (not shown here) additionally tries to close the channel, but that doesn't work either.
As jt said, NIO is the best (and correct) solution. If you really are stuck with an InputStream though, you could either
Spawn a thread who's exclusive job is to read from the InputStream and put the result into a buffer which can be read from your original thread without blocking. This should work well if you only ever have one instance of the stream. Otherwise you may be able to kill the thread using the deprecated methods in the Thread class, though this may cause resource leaks.
Rely on isAvailable to indicate data that can be read without blocking. However in some cases (such as with Sockets) it can take a potentially blocking read for isAvailable to report something other than 0.
Inspired in this answer I came up with a bit more object-oriented solution.
This is only valid if you're intending to read characters
You can override BufferedReader and implement something like this:
public class SafeBufferedReader extends BufferedReader{
private long millisTimeout;
( . . . )
#Override
public int read(char[] cbuf, int off, int len) throws IOException {
try {
waitReady();
} catch(IllegalThreadStateException e) {
return 0;
}
return super.read(cbuf, off, len);
}
protected void waitReady() throws IllegalThreadStateException, IOException {
if(ready()) return;
long timeout = System.currentTimeMillis() + millisTimeout;
while(System.currentTimeMillis() < timeout) {
if(ready()) return;
try {
Thread.sleep(100);
} catch (InterruptedException e) {
break; // Should restore flag
}
}
if(ready()) return; // Just in case.
throw new IllegalThreadStateException("Read timed out");
}
}
Here's an almost complete example.
I'm returning 0 on some methods, you should change it to -2 to meet your needs, but I think that 0 is more suitable with BufferedReader contract. Nothing wrong happened, it just read 0 chars. readLine method is a horrible performance killer. You should create a entirely new BufferedReader if you actually want to use readLine. Right now, it is not thread safe. If someone invokes an operation while readLines is waiting for a line, it will produce unexpected results
I don't like returning -2 where I am. I'd throw an exception because some people may just be checking if int < 0 to consider EOS. Anyway, those methods claim that "can't block", you should check if that statement is actually true and just don't override'em.
import java.io.BufferedReader;
import java.io.IOException;
import java.io.Reader;
import java.nio.CharBuffer;
import java.util.concurrent.TimeUnit;
import java.util.stream.Stream;
/**
*
* readLine
*
* #author Dario
*
*/
public class SafeBufferedReader extends BufferedReader{
private long millisTimeout;
private long millisInterval = 100;
private int lookAheadLine;
public SafeBufferedReader(Reader in, int sz, long millisTimeout) {
super(in, sz);
this.millisTimeout = millisTimeout;
}
public SafeBufferedReader(Reader in, long millisTimeout) {
super(in);
this.millisTimeout = millisTimeout;
}
/**
* This is probably going to kill readLine performance. You should study BufferedReader and completly override the method.
*
* It should mark the position, then perform its normal operation in a nonblocking way, and if it reaches the timeout then reset position and throw IllegalThreadStateException
*
*/
#Override
public String readLine() throws IOException {
try {
waitReadyLine();
} catch(IllegalThreadStateException e) {
//return null; //Null usually means EOS here, so we can't.
throw e;
}
return super.readLine();
}
#Override
public int read() throws IOException {
try {
waitReady();
} catch(IllegalThreadStateException e) {
return -2; // I'd throw a runtime here, as some people may just be checking if int < 0 to consider EOS
}
return super.read();
}
#Override
public int read(char[] cbuf) throws IOException {
try {
waitReady();
} catch(IllegalThreadStateException e) {
return -2; // I'd throw a runtime here, as some people may just be checking if int < 0 to consider EOS
}
return super.read(cbuf);
}
#Override
public int read(char[] cbuf, int off, int len) throws IOException {
try {
waitReady();
} catch(IllegalThreadStateException e) {
return 0;
}
return super.read(cbuf, off, len);
}
#Override
public int read(CharBuffer target) throws IOException {
try {
waitReady();
} catch(IllegalThreadStateException e) {
return 0;
}
return super.read(target);
}
#Override
public void mark(int readAheadLimit) throws IOException {
super.mark(readAheadLimit);
}
#Override
public Stream<String> lines() {
return super.lines();
}
#Override
public void reset() throws IOException {
super.reset();
}
#Override
public long skip(long n) throws IOException {
return super.skip(n);
}
public long getMillisTimeout() {
return millisTimeout;
}
public void setMillisTimeout(long millisTimeout) {
this.millisTimeout = millisTimeout;
}
public void setTimeout(long timeout, TimeUnit unit) {
this.millisTimeout = TimeUnit.MILLISECONDS.convert(timeout, unit);
}
public long getMillisInterval() {
return millisInterval;
}
public void setMillisInterval(long millisInterval) {
this.millisInterval = millisInterval;
}
public void setInterval(long time, TimeUnit unit) {
this.millisInterval = TimeUnit.MILLISECONDS.convert(time, unit);
}
/**
* This is actually forcing us to read the buffer twice in order to determine a line is actually ready.
*
* #throws IllegalThreadStateException
* #throws IOException
*/
protected void waitReadyLine() throws IllegalThreadStateException, IOException {
long timeout = System.currentTimeMillis() + millisTimeout;
waitReady();
super.mark(lookAheadLine);
try {
while(System.currentTimeMillis() < timeout) {
while(ready()) {
int charInt = super.read();
if(charInt==-1) return; // EOS reached
char character = (char) charInt;
if(character == '\n' || character == '\r' ) return;
}
try {
Thread.sleep(millisInterval);
} catch (InterruptedException e) {
Thread.currentThread().interrupt(); // Restore flag
break;
}
}
} finally {
super.reset();
}
throw new IllegalThreadStateException("readLine timed out");
}
protected void waitReady() throws IllegalThreadStateException, IOException {
if(ready()) return;
long timeout = System.currentTimeMillis() + millisTimeout;
while(System.currentTimeMillis() < timeout) {
if(ready()) return;
try {
Thread.sleep(millisInterval);
} catch (InterruptedException e) {
Thread.currentThread().interrupt(); // Restore flag
break;
}
}
if(ready()) return; // Just in case.
throw new IllegalThreadStateException("read timed out");
}
}
Related
I'm trying to make a simple text editor which can be shared accross multiple terminals at the same time. I have a Server waiting for new users, when a user enters the shared editor it just starts waiting for input characters.
public class Server {
public static final int PORT = 8080;
public static void main(String[] args) throws IOException {
ServerSocket ss = new ServerSocket(PORT);
while (true) {
Socket socket = ss.accept();
System.out.println("A new user entered the sever");
new Thread(() -> serve(socket)).start();
}
}
private static void serve(Socket socket) {
try {
while (!socket.isClosed() && !socket.isInputShutdown()) {
System.out.println("hey " + socket.isClosed() + " " + socket.isInputShutdown());
System.out.print(new String(SocketUtil.receiveBytes(socket,1)));
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
When a user closes the editor, the socket is closed on client side. However, the socket on the server side does not get closed and the server starts looping infinitly in the "wait for input" loop.
The Client is a singleton containing the following methods, called at the openning and closing of the editor.
public static void init() {
try {
if (socket == null) socket = new Socket(HOST,Server.PORT);
} catch (IOException e) {
e.printStackTrace();
kill();
throw new Error(e.getMessage());
}
}
public static void kill() {
Check.notNull(socket);
try {
SocketUtil.terminateCommunication(socket);
System.out.println(socket.isClosed());
} catch (IOException e) {
e.printStackTrace();
}
}
Finally, here are utilitary methods (in SocketUtil) used in both classes :
public static void terminateCommunication(Socket socket) throws IOException {
socket.shutdownInput();
socket.shutdownOutput();
socket.close();
}
public static char[] receiveBytes(Socket socket, int nBytes) throws IOException {
char[] bytes = new char[nBytes];
InputStreamReader isr = new InputStreamReader(socket.getInputStream());
isr.read(bytes);
return bytes;
}
Any idea of why the socket on server side is not closed after the Client gets killed ?
It is not quite clear from the Javadoc, but isClosed() only returns true when you have explicitly called close() on the socket (see the sources to confirm that). You should check for exceptions and the return value of read() instead. If you read -1 or catch an IOException while trying to read (or write, for that matter), it essentially means that the other side has closed the connection, so you should close your socket as well (better to it in a finally block) and you're done with that particular connection. You don't check for -1 in receiveBytes(), but you really should. Perhaps throw a EOFException() if you want to merge these two possibility into one, so that the code up the stack (in serve()) doesn't have to figure out what exactly happened:
public static char[] receiveBytes(Socket socket, int nBytes) throws IOException {
char[] bytes = new char[nBytes];
InputStreamReader isr = new InputStreamReader(socket.getInputStream());
if (isr.read(bytes) == -1)
throw new EOFException();
return bytes;
}
One exception from the IOException rule (sorry for the pun) is the SocketTimeoutException. If you get this, the connection is still alive, and you may just as well retry your read(). But I believe that in order to get these, you must call Socket.setSoTimeout() somewhere, and if you haven't, then you probably shouldn't worry about SocketTimeoutException.
You should also note that read() may sometimes return partial reads (that is, less than bytes.length). If it's important that receiveBytes() reads exactly nBytes (which probably is, since you never return the number of actual characters read), then you should call it in a loop, like this:
int pos = 0;
while (pos < bytes.length) {
int l;
if ((l = isr.read(bytes, pos, bytes.length - pos)) == -1) {
throw new EOFException();
}
pos += l;
}
I know this is cumbersome, which is exactly why many developers create utility methods like your receiveBytes().
The proper way to detect that the client has closed its connection is by checking the reception of 0 bytes.
System.out.print(new String(SocketUtil.receiveBytes(socket,1)));
just check if the string is empty should do the trick.
Note that I am not that familiar with java, but I do know socket programming.
Receiving 0 bytes, checking for that, and closing the socket if you do is a good solution.
You can use exception handling too, but you'll detect that the peer closed it socket an iteration later. Receiving 0 bytes is not really an error condition it is just a signal from the peer that he has closed its end of the socket and won't send anymore data. If you ignore this, and keep using the socket, you'll receive an exception in the next iteration because there is nothing to receive anymore.
public static void waitUntil(String prompt, InputStream instr) {
while (true) {
try {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
if (instr.available() >= 5) {
byte[] buff = new byte[1024];
int ret_read = 0;
ret_read = instr.read(buff);
if (ret_read > 0) {
if ((new String(buff, 0, ret_read)).contains(prompt)
&& flag) {
break;
}
}
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
If if remove that thread.sleep(1000) or even i reduce the to less than 1000 its not working properly.
Question : How to read java socket inputstream without thread.sleep() till all all incoming bytes are arrived?
if (instr.available() >= 5) {
Don't do that.
Instead of checking how many bytes are available, just try to read some into a buffer.
That will block until at least one byte is available, and then return as many as there are (that also fit into the buffer).
If that does not return all the bytes you need, loop until you get them.
If you just want to read everything, check out this thread: Convert InputStream to byte array in Java . Personally, I use Commons IO for this.
Use DataInputStream.readFully() with a buffer size of 5 (in this case, or more generally the size of data you're expecting), and get rid of both the sleep and the available() test.
I'm trying to write code that reads from a Reader in a non-blocking way, i.e. only calling read() after ensuring ready() returns true.
To test the code, I use the PipedReader/PipedWriter combo, and ready() works great, until I close the PipedWriter, which also marks the end of the stream on the side of PipedReader. However, although its ready() method will never again return true, read() instantly returns -1 when called.
In this case, is there a non-blocking way to figure out that a call to read() would not block?
Here's an example that reproduces the issue.
package test;
import java.io.*;
public class Piper
{
public static void main(String[] args) throws Exception
{
int sleepBetweenChars = 100;
boolean throttle = true;
Reader reader = new StringReader("abc");
if (throttle)
reader = throttle(sleepBetweenChars, reader);
while (true)
{
if (reader.ready())
{
int c = reader.read();
System.out.println("Read " + c);
if (c == -1)
break;
}
else
{
System.out.println("Not ready");
Thread.sleep(sleepBetweenChars);
if (!reader.ready())
{
System.out.println("Still not ready, sleep some more");
Thread.sleep(20 * sleepBetweenChars);
if (!reader.ready())
{
int c = reader.read();
System.out.println("Still not ready, we read nonetheless: " + c);
if (c == -1)
break;
}
}
}
}
}
private static Reader throttle(final int sleepBetweenChars, final Reader in) throws IOException
{
final PipedWriter pw = new PipedWriter();
PipedReader pr = new PipedReader(pw);
(new Thread(new Runnable()
{
public void run()
{
System.out.println("Start piping");
int c;
try
{
while ((c = in.read()) != -1)
{
pw.write(c);
Thread.sleep(sleepBetweenChars);
}
pw.close();
System.out.println("Closed PipedWriter");
}
catch (Throwable t)
{
t.printStackTrace();
}
}
})).start();
return pr;
}
}
PipedReader never ready() at end of stream
I agree. That's not what it's for. ready() tells you whether there is data available to be read, not end-of-file conditions. See the Javadoc: ' A piped character stream is ready if the circular buffer is not empty'.
There isn't a way to do what you're trying to do. Streams and Readers are irredemiably blocking. Your code is basically just an elaborate way of implementing blocking yourself, with added latency and added bugs. It's pointless.
I would also question your use of piped readers and writers at all. They are singularly useless things, really only a proof-of-concept for wait() and notify(). Possibly what you're really looking for is a Queue.
I have looked at many examples and tried to understand what i`m doing wrong but with no success, maybe you can help me. It always stops at the second file, but the first one is just crated on c:\ with 0kb size.
files_server.get(i) is ArrayList with all files that i wish to download.
My code:
public FTPConnection() {
StartD std = new StartD();
std.start();
}
class StartD extends Thread{
#Override
public void run()
{
for (int i = 0; i < files_server.size(); i++) {
err = ftpDownload(files_server.get(i), "C:/"+ files_server.get(i));
if (!err)
{
System.out.println("Error in download, breaking");
break;
}
}
}
public boolean ftpDownload(String srcFilePath, String desFilePath)
{
try {
FileOutputStream desFileStream = new FileOutputStream(desFilePath);
InputStream input = mFTPClient.retrieveFileStream(srcFilePath);
byte[] data = new byte[1024];
int count;
while ((count = input.read(data)) != -1)
{
desFileStream.write(data, 0, count);
}
desFileStream.close();
} catch (Exception e) {
return false;
}
return true;
}}
If I use the finction:
public boolean ftpDownload(String srcFilePath, String desFilePath) {
boolean status = false;
try {
FileOutputStream desFileStream = new FileOutputStream(desFilePath);
status = mFTPClient.retrieveFile(srcFilePath, desFileStream);
desFileStream.close();
return status;
} catch (Exception e) {
}
return status;
}
instead, everything works just fine, but i can`t monitor file download progress.
I've only used it for file unzipping and not FTP, but in that case InputStream buffers can return zero, so I'd say it's worth trying changing your while loop to something like:
while ((count = input.read(data)) >= 0)
public int read(byte[] b) throws IOException
Reads some number of bytes from the input stream and stores them into the buffer array b.
The number of bytes actually read is returned as an integer. This
method blocks until input data is available, end of file is detected,
or an exception is thrown.
If the length of b is zero, then no bytes are read and 0 is returned;
It could also be that you're assigning count twice, which could chop the first byte off the data:
int count = input.read(data);
while ((count = input.read(data)) != -1)
So don't assign anything to count when you declare it.
Let's assume your library is the FTP client from the commons-net package. It's not easy to figure out what's wrong with your code, because we can't run it and because your description (the second file stops) is not sufficient (does it throw an exception? Does it hang forever? Does it complete without any side effect?). Anyway I have a couple of advices:
Use a CountingOutputStream (from Apache commons-io) to monitor progress
Use a ProtocolCommandListener to log what's going on
Also, note that the first 1024 bytes are always lost. Eventually, I don't know how safe it is to put a file in C:\ with the same name it has on the server. At the best, it could lead to permission troubles, at the worst it may originate a security flaw - anyway this doesn't hold if you have some degree of control over the filenames, but hey consider this advice.
This is a sample client
public class FTP {
public static void main(String[] args) throws SocketException, IOException {
FTPClient client = new FTPClient();
client.addProtocolCommandListener(new ProtocolCommandListener(){
#Override
public void protocolCommandSent(ProtocolCommandEvent evt) {
logger.debug(evt.getMessage());
}
#Override
public void protocolReplyReceived(ProtocolCommandEvent evt) {
logger.debug(evt.getMessage());
}
});
client.connect("ftp.mozilla.org");
client.login("anonymous", "");
client.enterLocalPassiveMode();
OutputStream out = new CountingOutputStream(new NullOutputStream()) {
#Override
public void beforeWrite(int count) {
super.beforeWrite(count);
logger.info("Downloaded " + getCount() + " bytes");
}
};
for (String filename: new String[] {"MD5SUMS", "SHA1SUMS"})
client.retrieveFile("pub/firefox/releases/15.0b4/" + filename, out);
out.close();
client.disconnect();
}
private static Logger logger;
static {
logger = Logger.getLogger(FTP.class.getCanonicalName());
}
}
Once configured, the logger will output all the raw socket conversation, and it may help you to better understand the problem, provided it's on the FTP side and not in application IO
I am writing a video application in Java by executing ffmpeg and capturing its output to standard output. I decided to use Apache Commons-Exec instead of Java's Runtime, because it seems better. However, I am have a difficult time capturing all of the output.
I thought using pipes would be the way to go, because it is a standard way of inter-process communication. However, my setup using PipedInputStream and PipedOutputStream is wrong. It seems to work, but only for the first 1042 bytes of the stream, which curiously happens to be the value of PipedInputStream.PIPE_SIZE.
I have no love affair with using pipes, but I want to avoid use disk I/O (if possible), because of speed and volume of data (a 1m 20s video at 512x384 resolution produces 690M of piped data).
Thoughts on the best solution to handle large amounts of data coming from a pipe? My code for my two classes are below. (yes, sleep is bad. Thoughts on that? wait() and notifyAll() ?)
WriteFrames.java
public class WriteFrames {
public static void main(String[] args) {
String commandName = "ffmpeg";
CommandLine commandLine = new CommandLine(commandName);
File filename = new File(args[0]);
String[] options = new String[] {
"-i",
filename.getAbsolutePath(),
"-an",
"-f",
"yuv4mpegpipe",
"-"};
for (String s : options) {
commandLine.addArgument(s);
}
PipedOutputStream output = new PipedOutputStream();
PumpStreamHandler streamHandler = new PumpStreamHandler(output, System.err);
DefaultExecutor executor = new DefaultExecutor();
try {
DataInputStream is = new DataInputStream(new PipedInputStream(output));
YUV4MPEGPipeParser p = new YUV4MPEGPipeParser(is);
p.start();
executor.setStreamHandler(streamHandler);
executor.execute(commandLine);
} catch (IOException e) {
e.printStackTrace();
}
}
}
YUV4MPEGPipeParser.java
public class YUV4MPEGPipeParser extends Thread {
private InputStream is;
int width, height;
public YUV4MPEGPipeParser(InputStream is) {
this.is = is;
}
public void run() {
try {
while (is.available() == 0) {
Thread.sleep(100);
}
while (is.available() != 0) {
// do stuff.... like write out YUV frames
}
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
The problem is in the run method of YUV4MPEGPipeParser class. There are two successive loops. The second loop terminates immediately if there are no data currently available on the stream (e.g. all input so far was processed by parser, and ffmpeg or stream pump were not fast enough to serve some new data for it -> available() == 0 -> loop is terminated -> pump thread finishes).
Just get rid of these two loops and sleep and just perform a simple blocking read() instead of checking if any data are available for processing. There is also probably no need for wait()/notify() or even sleep() because the parser code is started on a separate thread.
You can rewrite the code of run() method like this:
public class YUV4MPEGPipeParser extends Thread {
...
// optimal size of buffer for reading from pipe stream :-)
private static final int BUFSIZE = PipedInputStream.PIPE_SIZE;
public void run() {
try {
byte buffer[] = new byte[BUFSIZE];
int len = 0;
while ((len = is.read(buffer, 0, BUFSIZE) != -1) {
// we have valid data available
// in first 'len' bytes of 'buffer' array.
// do stuff.... like write out YUV frames
}
} catch ...
}
}