Develop Reference

java rmi - how to handle RemoteException

I have checked some of the Java RMI examples. But none of them demonstrate how to recover from a potential server failure, or a RemoteException, which might be thrown from the remote interface we define. And I couldn't find any relevant information either.
For instance,
public static void main(String[] args)
{
try
{
Server server = new Server();
Hello stub = (Hello) UnicastRemoteObject.exportObject(server, 0);
Naming.rebind("rmi://localhost/Hello", stub);
System.out.println("Server is up.");
}
catch (RemoteException e)
{
// TODO Auto-generated catch block
e.printStackTrace();
}
catch (MalformedURLException e)
{
// TODO Auto-generated catch block
e.printStackTrace();
}
}
After the exception is caught, the server just exits. Is here the intent for the server to die when such exceptions are thrown? Can we recover from such failures, like restarting the server? But how?
Thank you,
-----------Updates----------
To clarify, initially, I thought the RemoteException specified in the remote interface might be caught here. But after some experiments, I find they will actually show up on the client side and should be handled by the client.

Your RMI server only throws that exception at initialisation / start-up in UnicastRemoteObject.exportObject or Naming.rebind. The chances are that any error here isn't recoverable simply by running a loop to perform the same action again. There is some other problem to investigate.
If you server gets past the exportObject/rebind you see the "Server is up" message, the execution passes through the catch block and main(String[] args) ends normally. Even the thread handling [main] probably ends here.
But, the JVM process does not exit as there are non-daemon threads running. Any exceptions in the server are not passed back to the catch(RemoteException e) block shown in your main because main() has ended already.
In live systems I've observed several measures used by developers to counter issues found with RMI servers:
server startup scripts use limited loop to retry startup if it dies (this is more effective if it dies after long period not if fails instantly)
periodic round trip rmi client to server calls with trivial operation to prove RMI server health
run multiple instances of rmi servers with deliberate shutdown restart cycles at different time to ensure 100% operation
process monitors on rmi servers => trigger alerts for manual intervention
Most importantly you need effective logging so you can diagnose why it died afterwards.

Java newFixedThreadPool explanation

I am trying to diagnose a problem I am having using WSO2 identity management.
package org.wso2.carbon.identity.mgt;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
/**
* email sender this creates a new task in thread pool for each email sending request
*/
public class NotificationSender {
private static ExecutorService threadPool = Executors.newFixedThreadPool(5);
NotificationSendingModule module;
/**
* creates and submits a task to the thread pool
*
* #param module email sending module as task
*/
public void sendNotification(NotificationSendingModule module) {
threadPool.submit(module);
}
public NotificationSendingModule getModule() {
return module;
}
}
I am load testing the user creation process, and at the point where WSO2 sends a User credentials configuration mail it is sending multiple to the same email address even though they are unique.
I have never used Java but am familiar with C#, so have been able to read through the code without issue, and my questions is:
In the Java docs it mentions "If a thread terminates due to failure during execution and prior to shutdown, a new thread is created to take its place."
Does this mean that if the email send encounters an error then a new thread will begin the process again?
I'm thinking perhaps the send email is erroring so a new thread is created, but the logging with isn't tied into a result is performed anyway.
Also, is it ok to never call
threadPool.shutdown()

When a thread that is part of a thread pool throws an exception it is indeed replaced with a new fresh thread. However it will not retry the same operation. The replacement only occurs so the thread pool can continue do its work when more tasks needs to be executed.
Normally when a thread is terminated in such a fashion, a stack trace is logged, but it is possible the exception is swallowed somewhere. You could try adding a try-catch block around the sending code and logging any exception explicitly to analyze the problem further.
Not calling shutdown is fine.

I am load testing the user creation process, and at the point where WSO2 sends a User credentials configuration mail it is sending multiple to the same email address even though they are unique.
Well, when I hear a Java framework/app server doing identity management + thread pools + strange behavior, what immediatly comes to mind is that most frameworks use a thread per user model (that is : the user identity is tied to the thread. if you switch threads, the user authentication data is lost). Now I do not know if that is the case with SO2, but refer to the documentation. It is the "usual suspect" : thread local authentication mechanisms are everywhere.
In the Java docs it mentions "If a thread terminates due to failure during execution and prior to shutdown, a new thread is created to take its place."
Does this mean that if the email send encounters an error then a new thread will begin the process again?
No. It means that a new thread will be created to handle other unit of works as they are / have been submitted. But the failed unit of work will not be attempted again. As far as the thread pool is concerned, the task completed (with an exception), and it is done with it.
Also, is it ok to never call threadPool.shutdown()
It is not. You should either make your NotificationSender class have a close() or end() method of some sort. Or maybe tie it with some of WSO2 lifecycle callbacks (e.g. in a servlet context, you have listeners for lifecycle events, in a Spring container, you have other create/destroy callbacks, ... whatever works in your context). Failure to shut down a thread pool implies that some threads will hang around, and their resources never freed. Threads are coming pretty cheap nowadays, but they may still pile up and bit you in the long run. It may only be kind of ok if you are sure you only create one NotificationSender in your whole app, and that the lifecycle of this object is the same as your app. Then, essentially, shutting it down is the same as shutting the app down and so nothing bad really happens.

How JMS work in Java?

How does async JMS work? I've below sample code:
public class JmsAdapter implements MessageListener, ExceptionListener
{
private ConnectionFactory connFactory = null;
private Connection conn = null;
private Session session = null;
public void receiveMessages()
{
try
{
this.session = this.conn.createSession(true, Session.SESSION_TRANSACTED);
this.conn.setExceptionListener(this);
Destination destination = this.session.createQueue("SOME_QUEUE_NAME");
this.consumer = this.session.createConsumer(destination);
this.consumer.setMessageListener(this);
this.conn.start();
}
catch (JMSException e)
{
//Handle JMS Exceptions Here
}
}
#Override
public void onMessage(Message message)
{
try
{
//Do Message Processing Here
//Message sucessfully processed... Go ahead and commit the transaction.
this.session.commit();
}
catch(SomeApplicationException e)
{
//Message processing failed.
//Do whatever you need to do here for the exception.
//NOTE: You may need to check the redelivery count of this message first
//and just commit it after it fails a predefined number of times (Make sure you
//store it somewhere if you don't want to lose it). This way you're process isn't
//handling the same failed message over and over again.
this.session.rollback()
}
}
}
But I'm new to Java & JMS. I'll probably consume messages in onMessage method. But I don't know how does it work exactly.
Do I need to add main method in JmsAdapter class? After adding main method, do I need to create a jar & then run the jar as "java -jar abc.jar"?
Any help is much appreciated.
UPDATE: What I want to know is that if I add main method, should I simply call receiveMessages() in main? And then after running, will the listener keep on running? And if there are messages, will it retrieve automatically in onMessage method?
Also, if the listener is continuously listening, doesn't it take CPU??? In case of threads, when we create a thread & put it in sleep, the CPU utilization is zero, how doe it work in case of listener?
Note: I've only Tomcat server & I'll not be using any jms server. I'm not sure if listener needs any specific jms server such as JBoss? But in any case, please assume that I'll not be having anything except tomcat.
Thanks!

You need to learn to walk before you start trying to run.
Read / do a tutorial on Java programming. This should explain (among other things) how to compile and run a Java program from the command line.
Read / do a tutorial on JMS.
Read the Oracle material on how to create an executable JAR file.
Figure out what it is you are trying to do ... and design your application.
Looking at what you've shown and told us:
You could add a main method to that class, but to make an executable JAR file, you've got to create your JAR file with a manifest entry that specifies the name of the class with the main method.
There's a lot more that you have to do before that code will work:
add code to (at least) log the exceptions that you are catching
add code to process the messages
add code to initialize the connection factory and connection objects
And like I said above, you probably need some kind of design ... so that you don't end up with everything in a "kitchen sink" class.
if I add main method, should I simply call receiveMessages() in main?
That is one approach. But like I said, you really need to design your application.
And then after running, will the listener keep on running?
It is not entirely clear. It should keep running as long as the main thread is alive, but it is not immediately obvious what happens when your main method returns. (It depends on whether the JMS threads are created as daemon threads, and that's not specified.)
And if there are messages, will it retrieve automatically in onMessage method?
It would appear that each message is retrieved (read from the socket) before your onMessage method is called.
Also, if the listener is continuously listening, doesn't it take CPU???
Not if it is implemented properly.
In case of threads, when we create a thread & put it in sleep, the CPU utilization is zero, how doe it work in case of listener?
At a certain level, a listener thread will make a system call that waits for data to arrive on a network socket. I don't know how it is exactly implemented, but this could be as simple as an read() call on the network socket's InoutStream. No CPU is used by a thread while it waits in a blocking system call.

This link looks like a pretty good place with examples using Oracle AQ. There's an examples section that tells you how to setup the examples and run them. Hopefully this can help.
Link to Oracle Advanced Queueing

Client thread hang emulation blocks server from accepting any I/O for the time client is set to wait

As the topic suggests I have a server and some clients.
The server accepts I/O connections concurrently (no queueing in socket connections) but I have this troubling issue and I do not know how to bypass it!
If I force a client to throw an I/O Exception the server detects it and terminates the client thread correctly (verified from Task Manager (Windows) and System Monitor (Ubuntu) ). But If I emulate an I/O that is "hanging" like i.e. Thread.sleep(60*1000);or
private static Object lock = new Object();
synchronized(lock) {
while (true) {
try {
lock.wait();
} catch (InterruptedException e) {
/* Foo */
}
}
}
then all subsequent I/O operations (connection & data transfer) seem to block or wait until the "hanging" client is terminated. The applications makes use of the ExecutorService so if the "hanging" client does not complete the operations in the suggested time limit then the task will time out and the client is forced to exit. The subsequent "blocked" I/Os will resume but I wonder why the server doesn't accept any I/O connections or performs any I/O operations when a client "hangs"?
NOTE:The client threading takes place in the server main like this:
while (true) {
accept client connection;
submit client task;
||
\ /
\/
// ExecutorService here in the form
// spService.submit(new Callable<Tuple<String[], BigDecimal[]>>() {
// ... code ... }}).get(taskTimeout, taskTimeUnit);
check task result & perform cleanup if result is null;
otherwise continue;
}

The Problem :
This may very well indicate that your server ACCEPTS client connections concurrently, however, it only handles these connections synchronously. That means that even if a million clients connect, successfully, at any given time, if anyone of them takes a long time (or hangs), it will hold up the others.
The TEST:
To verify this : I would toggle the amount of time a client takes to connect by adding Thread.sleep statments(1000) in your clients.
Expected result :
I believe you will see that even adding a single Thread.sleep(1000) statement in your client delays all other connecting clients by 1000.

I think I have found the source of my problems!
I do use one thread-per-client model but I run my tests locally i.e. in the same machine which means all of them have the same IP! So each client is assigned the same IP with the server! I guess that this leaves server and clients to differ only in port number but since each client is mapped to a different localport for each server connection then the server shouldn't block. I have confirmed that each client and server use different I/Os (compared references) and I wrap their sockets using <Input/Output>Streams to BufferedReaders & PrintWriters but still when a client hangs all other clients hang too (so maybe the I/O channels are indeed the same???)!I will test this on another machine and check the results back with you! :)
EDIT: Confirmed the erratic behaviour. It seems that even with remote clients if one hangs the other clients seem to hang too! :/
Don't know but I am determined to fix this. It's just that it's pretty weird since I am pretty sure I use one thread-per-client (I/Os differ, client sockets differ, IPs seem to be not a problem, I even map each client in the server to a localport of my choice ...)
May be I'll switch to NIO if I don't find a solution soon enough.
SOLUTION: Solved the problem! It seemed that the ExecutorService had to be run in a seperate thread otherwise if an I/O in a client blocked, all I/Os would block! That's strange given the fact that I've tried both an Executors.newFixedThreadPool(<nThreads>); and Executors.newCachedThreadPool(); and the client actions (aka I/Os) should take place in a new Thread for each client.
In any case, I used a method and wrapped the calls so each client instace would use a final ExecutorService baseWorker = Executors.newSingleThreadExecutor(); and created a new Thread explicitly each time using <Thread instance>.start(); so each thread would run in the background :)

Why does JVM does not terminate when using RMI

I just read the Trail on RMI from sun at http://java.sun.com/docs/books/tutorial/rmi/implementing.html
When I run the example, the JVM does not terminate although main has finished. Is RMI spawning a Thread somewhere internally?
What is the behaviour of multiple Threads spawned in main, after main exits?
Is it a clean way to let the Threads exit whenever they want or should you do a join on each Thread you spawn? I did not find any documentation on this question.
Thank you very much for your help!!
public class ComputeEngine implements Compute {
public ComputeEngine() {
super();
}
public <T> T executeTask(Task<T> t) {
return t.execute();
}
public static void main(String[] args) {
if (System.getSecurityManager() == null) {
System.setSecurityManager(new SecurityManager());
}
try {
String name = "Compute";
Compute engine = new ComputeEngine();
Compute stub = (Compute) UnicastRemoteObject.exportObject(engine, 0);
Registry registry = LocateRegistry.getRegistry();
registry.rebind(name, stub);
System.out.println("ComputeEngine bound");
} catch (Exception e) {
System.err.println("ComputeEngine exception:");
e.printStackTrace();
}
}
}

A thread is created for listening the socket and reply to requests to your object. One way to stop the JVM is to unbind the server:
Registry.unbind()
and unexport the objects:
UnicastRemoteObject.unexportObject()).

You can use the jstack utility program, included in the JDK to see which threads are running in a java program, and even what line those threads are on.
So if you're program is still running you just run jstack on the pid and it will tell you which threads are still running and what they're doing.

Concerning the "behaviour of multiple threads" part of your question: yes, the JVM will keep on running until all threads finished. The only exception are threads marked as daemon (see Thread.setDaemon()), the JVM will not wait for them.

Yes, when you are exposing objects through RMI it needs a thread to accept incoming requests for these objects. This thread could be a daemon thread which wouldn't stop the JVM from exiting but it isn't for several reasons and as long as there are still active exported objects it hinders the JVM from exiting normally. So you can use unexportObject for all objects or just use System.exit() to end the JVM although this would leave the clients uninformed about the shutdown.