What are use cases of Piped streams? Why just not read data into buffer and then write them out?
BlockingQueue or similiar collections may serve you better, which is thread safe, robust, and scales better.
Pipes in Java IO provides the ability for two threads running in the same JVM to communicate. As such pipes are a common source or destination of data.
This useful if you have two long running Threads and one is setup to produce data and the other consume it.
As the other answers have said, they are designed for use between threads. In practice they are best avoided. I've used them once in 13 years and I wish I hadn't.
They are usually used for simultaneously reading and writing, usually by two different threads.
(They design is quite bad. You can't switch threads at one end and then have that thread exit without disrupting the pipe.)
One advantage of using Piped streams is that they provide stream functionality in our code without compelling us to build new specialized streams.
For e.g. we can use pipes to create simple logging facility for our application.We can send messages to logging facility through ordinaty Printwritter and then it can do whatever processing or buffering is required before sending message off to final destination.
more details refer : http://docstore.mik.ua/orelly/java/exp/ch08_01.htm
Related
I have a multi-threaded Java 7 program (a jar file) which uses JDBC to perform work (it uses a fixed thread pool).
The program works fine and it logs things as it progresses to the command shell console window (System.out.printf()) from multiple concurrent threads.
In addition to the console output I also need to add the ability for this program to write to a single plain ASCII text log file - from multiple threads.
The volume of output is low, the file will be relatively small as its a log file, not a data file.
Can you please suggest a good and relatively simple design/approach to get this done using Java 7 features (I dont have Java 8 yet)?
Any code samples would also be appreciated.
thank you very much
EDIT:
I forgot to add: in Java 7 using Files.newOutputStream() static factory method is stated to be thread safe - according to official Java documentation. Is this the simplest option to write a single shared text log file from multiple threads?
If you want to log output, why not use a logging library, like e.g. log4j2? This will allow you to tailor your log to your specific needs, and can log without synchronizing your threads on stdout (you know that running System.out.print involves locking on System.out?)
Edit: For the latter, if the things you log are thread-safe, and you are OK with adding LMAX' disruptor.jar to your build, you can configure async loggers (just add "async") that will have a logging thread take care of the whole message formatting and writing (and keeping your log messages in order) while allowing your threads to run on without a hitch.
Given that you've said the volume of output is low, the simplest option would probably be to just write a thread-safe writer which uses synchronization to make sure that only one thread can actually write to the file at a time.
If you don't want threads to block each other, you could have a single thread dedicated to the writing, using a BlockingQueue - threads add write jobs (in whatever form they need to - probably just as strings) to the queue, and the single thread takes the values off the queue and writes them to the file.
Either way, it would be worth abstracting out the details behind a class dedicated for this purpose (ideally implementing an interface for testability and flexibility reasons). That way you can change the actual underlying implementation later on - for example, starting off with the synchronized approach and moving to the producer/consumer queue later if you need to.
Keep a common PrintStream reference where you'll write to (instead of System.out) and set it to System.out or channel it through to a FileOutputStream depending on what you want.
Your code won't change much (barely at all) and PrintStream is already synchronized too.
I´ve a question concerning non blocking Sockets: I understand how to register for example, two socketchannels for write/read events.
But how does such an event look like? If I want to write some data on SocketChannel1 (for example when I press a button) to a server how can I do this?
All examples I´ve found only deal with the registration of the sockets, like this:
http://rox-xmlrpc.sourceforge.net/niotut/#About%20the%20author
Greetings,
Flo
I would look at the examples which come with the JDK under the sample directory.
If you use non blocking IO, you should wait until after you have a write op from the socket to perform the write. While you are waiting, you can buffer the data. However, this rarely needed as this is only required when the write buffer of the socket is full (which shouldn't happen very often) and if this is the case for a long period fo time you may deside you have a slow consumer and close the connection instead.
Personally, I wouldn't suggest you use non-blocking NIO directly unless you have a very good understanding of what is going on. Instead I suggest you use a library like Netty which will handle all the edge cases for you. Or you could use blocking NIO which is much simpler (and can be faster for a small number of connections)
I have an application that needs to read hundreds of socket communications.
I am using a ThreadPool, with a upper limit on the number of threads, to service these sockets. This causes blocking on all threads if the sockets do not have incoming messages.
I am currently using a soTimeout of 100ms to avoid a permanent blocking. I do not like this approach as it might timeout just as it starts receiving input.
Is there anyway other to approach this?
I tried checking with ObjectInputStream.isAvailable(), but that always returns 0, whether there is data in the stream or not.
I can't find any other way to check whether there is data on the stream. This would be ideal, as then I could check if there is data, if not then move on to next stream.
This is exactly the kind of problem NIO frameworks are meant to solve. Unfortunately, using raw NIO is a bit more difficult than using blocking IO. If you can, my recommendation would be to try out a framework like Netty which would ease the job for you.
You can give NIO a chance.
Use Selector and SocketChannels to wait for data instead of creating thread for each socket.
Selector
SocketChannel
Is there a non-blocking file read API in java? If not would it be wise to build one in C++ and call it from a java app via JNI?
My original answer is now wrong, since the addition of AsynchronousFileChannel in Java 7.
You still cannot select on a file, but there are now two asynchronous file read methods: one that takes a callback and another that returns a Future.
It may be cleaner to use the callback method (and dispatch an event from the callback) than to have a dedicated thread polling a pipe.
No, FileChannel does not extend SelectableChannel.
Probably because not all OSes support it.
Windows does, and in theory you could write a windows-specific C++ library and call it via JNI, but it is a lot of work to integrate this with java.nio.
I would rather have a worker thread copy the file contents to a pipe and do non-blocking reads on the other end of the pipe.
AsynchronousFileChannel is the right answer. Yet, it does not provide an easy API. It is quite verbose to use it comparing with the similar usage of java.nio.file.Files that provides simple static methods, such as: readAllLines or lines. Unfortunately Files methods are synchronous.
The AsyncFiles alternative from RxIo provides the corresponding non-blocking methods, with 3 different APIs: callback based, CompletableFuture and also with reactive streams. Here it is an example with reactive streams:
AsyncFiles
.lines(path)
.subscribe(doOnNext(line -> /*... use line from background thread ...*/));
I'm in the process of converting our java code to use NIO, but I'm not sure of the best way to design it.
My initial approach was to create a pool of selector threads. The threads are started/killed as needed, and channels are registered to a selector thread when they are connected/accepted in a round-robin fashion. From there, each thread blocks on select(), and when woken up will run the appropriate callback associated with each channel that has a selected key.
In addition to this "multiple selector thread" design, I've also seen people say to use a single selector thread, and a pool of dispatch threads. When an IO operation is ready to be performed, the selector notifies a dispatcher thread, which then processes the request. This model has the benefit of not blocking the IO thread, but now we're forcing all of the IO into a single thread and dealing with synchronization/an event queue in the dispatcher.
Additionally I wouldn't be able to use a single direct byte buffer for reading each channel, passing it directly into the callback. Instead I'd have to copy the data out each time a read occurs into an array and reset. (I think..)
What's the best way to implement this?
Take a look at the Reactor Pattern
http://gee.cs.oswego.edu/dl/cpjslides/nio.pdf
How you want your selectors to work really depends on your usecase. (Number of connections, message size, etc)
What is the problem that you are trying to solve by converting from IO to NIO?
You really should look into Mina,
http://mina.apache.org/
It solves all the problems you mentioned.
Also have a look at netty which is really fast and feature rich and also is used in big systems and by big companies like Redhat (jboss), Twitter, Facebook... .