Develop Reference

Java concurrent writes from multiple threads to a single text file?

I have a multi-threaded Java 7 program (a jar file) which uses JDBC to perform work (it uses a fixed thread pool).
The program works fine and it logs things as it progresses to the command shell console window (System.out.printf()) from multiple concurrent threads.
In addition to the console output I also need to add the ability for this program to write to a single plain ASCII text log file - from multiple threads.
The volume of output is low, the file will be relatively small as its a log file, not a data file.
Can you please suggest a good and relatively simple design/approach to get this done using Java 7 features (I dont have Java 8 yet)?
Any code samples would also be appreciated.
thank you very much
EDIT:
I forgot to add: in Java 7 using Files.newOutputStream() static factory method is stated to be thread safe - according to official Java documentation. Is this the simplest option to write a single shared text log file from multiple threads?

If you want to log output, why not use a logging library, like e.g. log4j2? This will allow you to tailor your log to your specific needs, and can log without synchronizing your threads on stdout (you know that running System.out.print involves locking on System.out?)
Edit: For the latter, if the things you log are thread-safe, and you are OK with adding LMAX' disruptor.jar to your build, you can configure async loggers (just add "async") that will have a logging thread take care of the whole message formatting and writing (and keeping your log messages in order) while allowing your threads to run on without a hitch.

Given that you've said the volume of output is low, the simplest option would probably be to just write a thread-safe writer which uses synchronization to make sure that only one thread can actually write to the file at a time.
If you don't want threads to block each other, you could have a single thread dedicated to the writing, using a BlockingQueue - threads add write jobs (in whatever form they need to - probably just as strings) to the queue, and the single thread takes the values off the queue and writes them to the file.
Either way, it would be worth abstracting out the details behind a class dedicated for this purpose (ideally implementing an interface for testability and flexibility reasons). That way you can change the actual underlying implementation later on - for example, starting off with the synchronized approach and moving to the producer/consumer queue later if you need to.

Keep a common PrintStream reference where you'll write to (instead of System.out) and set it to System.out or channel it through to a FileOutputStream depending on what you want.
Your code won't change much (barely at all) and PrintStream is already synchronized too.

Python: Multithreading between Java subproccess and Python listener?

I am monitoring and Minecraft server and I am making a setup file in Python. I need to be able to run two threads, one running the minecraft_server.jar in the console window, while a second thread is constantly checking the output of the minecraft_server. Also, how would I input into the console from Python after starting the Java process?
Example:
thread1 = threading.Thread(target=listener)
thread2 = minecraft_server.jar
def listener():
if minecraft_server.jarOutput == "Server can't keep up!":
sendToTheJavaProccessAsUserInputSomeCommandsToRestartTheServer

It's pretty hard to tell here, but I think what you're asking is how to:
Launch a program in the background.
Send it input, as if it came from a user on the console.
Read its output that it tries to display to a user on the console.
At the same time, run another thread that does other stuff.
The last one is pretty easy; in fact, you've mostly written it, you just need to add a thread1.start() somewhere.
The subprocess module lets you launch a program and control its input and output. It's easiest if you want to just feed in all the input at once, wait until it's done, then process all the output, but obviously that's not your case here, so it's a bit more involved:
minecraft = subprocess.Popen(['java', 'path/to/minecraft_server.jar', '-other', 'args],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
I'm merging stdout and stderr together into one pipe; if you want to read them separately, or send stderr to /dev/null, or whatever, see the docs; it's all pretty simple. While we're making assumptions here, I'm going to assume that minecraft_server uses a simple line-based protocol, where every command, every response, and every info message is exactly one line (that is, under 1K of text ending in a \n).
Now, to send it input, you just do this:
minecraft.stdin.write('Make me a sandwich\n')
Or, in Python 3.x:
minecraft.stdin.write(b'Make me a sandwich\n')
To read its output, you do this:
response = minecraft.stdout.readline()
That works just like a regular file. But note that it works like a binary file. In Python 2.x, the only difference is that newlines don't get automatically converted, but in Python 3.x, it means you can only write bytes (and compatible objects), not strs, and you will receive bytes back. There are good reasons for that, but if you want to get pipes that act like text files instead, see the universal_newlines (and possibly bufsize) arguments under Frequently Used Arguments and Popen Constructor.
Also, it works like a blocking file. With a regular file, this rarely matters, but with a pipe, it's quite possible that there will be data later, but there isn't data yet (because the server hasn't written it yet). So, if there is no output yet (or not a complete line's worth, since I used readline()), your thread just blocks, waiting until there is.
If you don't want that, you probably want to create another thread to service stdout. And its function can actually look pretty similar to what you've got:
def listener():
for line in minecraft.stdout:
if line.strip() == "Server can't keep up!":
minecraft.stdin.write("Restart Universe\n")
Now that thread can block all day and there's no problem, because your other threads are still going.
Well, not quite no problem.
First it's going to be hard to cleanly shut down your program.
More seriously, the pipes between processes have a fixed size; if you don't service stdout fast enough, or the child doesn't service stdin fast enough, the pipe can block. And, the way I've written things, if the stdin pipe blocks, we'll be blocked forever in that stdin.write and won't get to the next read off stdout, so that can block too, and suddenly we're both waiting on each other forever.
You can solve this by having another thread to service stdout. The subprocess module itself includes an example, in the Popen._communicate function used by all the higher-level functions. (Make sure to look at Python 3.3 or later, because earlier versions had bugs.)
If you're in Python 3.4+ (or 3.3 with a backport off PyPI), you can instead use asyncio to rewrite your program around an event loop and handle the input and output the same way you'd write a reactor-based network server. That's what all the cool kids are doing in 2017, but back in late 2014 many people still thought it looked new and scary.
If all of this is sounding like a lot more work than you signed on for, you may want to consider using pexpect, which wraps up a lot of the tedious details, and makes some simplifying assumptions that are probably true in your case.

Java Non Blocking Client

I´ve a question concerning non blocking Sockets: I understand how to register for example, two socketchannels for write/read events.
But how does such an event look like? If I want to write some data on SocketChannel1 (for example when I press a button) to a server how can I do this?
All examples I´ve found only deal with the registration of the sockets, like this:
http://rox-xmlrpc.sourceforge.net/niotut/#About%20the%20author
Greetings,
Flo

I would look at the examples which come with the JDK under the sample directory.
If you use non blocking IO, you should wait until after you have a write op from the socket to perform the write. While you are waiting, you can buffer the data. However, this rarely needed as this is only required when the write buffer of the socket is full (which shouldn't happen very often) and if this is the case for a long period fo time you may deside you have a slow consumer and close the connection instead.
Personally, I wouldn't suggest you use non-blocking NIO directly unless you have a very good understanding of what is going on. Instead I suggest you use a library like Netty which will handle all the edge cases for you. Or you could use blocking NIO which is much simpler (and can be faster for a small number of connections)

Better alternative for PipedReader/PipedWriter?

I need to have a buffered char stream, into which I write in one thread and from which I read in another thread. Right now I'm using PipedReader and PipedWriter for it, but those classes cause a performance problem: PipedReader does a wait(1000) when its internal buffer is empty, which causes my application to lag visibly.
Would there be some library which does the same thing as PipedReader/PipedWriter, but with better performance? Or will I have to implement my own wheels?

The problem was that when something is written to the PipedWriter, it does not automatically notify the PipedReader that there is some data to read. When one tries to read PipedReader and the buffer is empty, the PipedReader will loop and wait using a wait(1000) call until the buffer has some data.
The solution is to call PipedWriter.flush() always after writing something to the pipe. All that the flush does is call notifyAll() on the reader. The fix to the code in question looks like this.
(To me the PipedReader/PipedWriter implementation looks very much like a case of premature optimization - why not to notifyAll on every write? Also the readers wait in an active loop, waking up every second, instead of waking only when there is something to read. The code also contains some todo comments, that the reader/writer thread detection which it does is not sophisticated enough.)
This same problem appears to be also in PipedOutputStream. In my current project calling flush() manually is not possible (can't modify Commons IO's IOUtils.copy()), so I fixed it by creating low-latency wrappers for the pipe classes. They work much better than the original classes. :-)

It should be fairly easy to wrap a char stream API around BlockingQueue.
I must say, however, it seems quite perverse that PipedReader would use polling to wait for data. Is this documented somewhere, or did you discover it for yourself somehow?

#Esko Luontola, I've been reading through your code in the sbt package to try to understand what you are doing. It seems like you want to start up a Process and pass input to it, and have the result of the action be teed to different places. Is this at all correct?
I would try modifying the main loop in ReaderToWriterCopier so that instead of doing a read() - a blocking operation that apparently when a PipedReader is involved causes polling - you explicitly wait for the Writer to flush. The documentation is clear that flush causes any Readers to be notified.
I'm not sure how to run your code so I can't get deeper into it. Hope this helps.

I implemented something a little similar, and asked a question whether anyone else had any better thought out and tested code.

Concurrent/Non-blocking console keyboard input

I'm working on a MUD in java. I read player input every tick, but I'm using Scanner which uses blocking operations. I want to have non-blocking input.
I've looked at the nio package which has a Selector class, but I'm not sure how to use it with regard to System.in. I figure I'll definitely need it once I'm running a server, but for now everything is offline.
I've tried extending the main class from Applet and overriding keyDown, but that just meant input was no longer accepted after the first one. Sure, I wasn't blocking anything anymore, but then there was no more input. keyDown never got called again, I guess.
Perhaps threads can be interrupted even when they are executing blocking operations?
Thanks for any insight into this problem.

You can't do that with the system console because by now it can't be done in a multi-platform way.
You can use swing window as console or find a JNI based approach but it might not work on some platforms.
You may use JCurses. It might work, it's based on JNI and supports Windows and Linux.

keyDown() is deprecated so I'd suggest to use processKeyEvent and a keyListener instead.
Perhaps threads can be interrupted even when they are executing blocking operations?
Yes if you have a reference to the thread object you want to interrupt, you can simply call interrupt() method on that instance. And in the run method of the thread you can handle the interrupted exception. However, this seems a little bit hack-ish. I don't see how this is more helpful than using a simple KeyListener.

JLine might fit your bill as well:
http://jline.sourceforge.net/apidocs/src-html/jline/ConsoleReader.html#line.1447

I had had to solve similar problem with blocking writing/reading from http. In that particular case I used local buffer and Threads.
Idea is simple, one Thread read from stdin and put content in buffer. Second do same with writing.
And then you use nonblocking queries into your buffer.
Sample code:
class NonBlockingReader implements Runnable{
Reader in;
List buffer;
public void run(){
String line=null;
while((line=in.readLine())!=null){
storeLine(line);
}
}
private synchronized storeLine(String line){
buffer.add(line);
}
public synchronized String getLine(){
if(list.size()>0)
return list.removeFirst();
return null;
}
}
// .. same for writer, then you jast pass stdin and stdout ...