Develop Reference

Java RMI connection broken

We have an java client-server RMI application. The client performs a call by:
public interface MyRemoteInterface extends java.rmi.Remote {
public byte[] action(byte[] b}
public static Remote lookUp(String ip, int port, String alias) {
return LocateRegistry.getRegistry(ip, port).lookup(alias)
}
serverImpl = (MyRemoteInterface)lookUp(ip, rmpiport, alias);
serverImpl.action(requestBytes);
..
It a classic rmi client-server appl. When the remote server receives the request (call) it performs some process P, which takes a time T and then returns. The problem is, suppose we have two clients C1 and C2, respectively, each are identical (same OS, same Java, no firewall, no antivirus etc). The difference of C1 and C2 is they are in different networks, different ISP provides etc. For example they are two users connecting from their home and use different service providers, modems and connecting to the centralized server S.
When the process P takes 4 minutes, there is no problem, both client receive the response. But if the process time takes for example 10 minutes, C2 receives no response without any exceptions on server or client side.
Question: What can be reason of this difference? How can the network architecture can effect the behaviour of rmi connection? Can we overcome this problem by setting some timeout parameters or jvm-parameters? We do not get any timeout exceptions on client or server. Simply one of the clients does not receive any response if the process time becomes longer.

Simply one of the clients does not receive any response if the process time becomes longer.
In our (enterprise) network we had some policy in force to close connections which were not active for long time. So there may be differences. Though the client should get a proper exception. For better answer you may gather more exact information on the network level (e. g. tcpdump)
But if the process time takes for example 10 minutes
I'm not really fond of long-running processes when executing remotely (regardless of the protocol). Mainly to prevent issues like the one in your question.
You may use some asynchronous processing, like a method to start a server process and then have the client to check for status/results.

Java socket does not timeout when severing the connection

I am seeing some strange issue with my automation tests.
There is a following setup:
Server: Centos 6
Client1: Windows 7
Client2: Centos 6
I'm writing the test that simulated connection disruption to server by blocking the outbound connection on server's iptables. But, the behavior of socket differs on Windows from one on Linux client.
One thing though, in both cases there is a line of Java code that does:
socket.setSoTimeout(0)
Scenario #1 (Windows):
send the ssh command to server iptables -A OUTPUT --dport XYZ -j DROP
After approximately 60 seconds my console says java.net.SocketTimeoutException: Read timed out
Connection drops
Scenario #2 (Centos)
send the same command as above
I tried waiting as much as 10 minutes but console never outputs the exception.
So, the question, is there a way to make the behavior of socket the same (or approximately the same)?
I read that Windows actually does not use SO_TIMEOUT but SO_RCVTIMEO instead.

From setSoTimeout(int timeout):
Enable/disable SO_TIMEOUT with the specified timeout, in milliseconds. With this option set to a non-zero timeout, a read() call on the InputStream associated with this Socket will block for only this amount of time. If the timeout expires, a java.net.SocketTimeoutException is raised, though the Socket is still valid. The option must be enabled prior to entering the blocking operation to have effect. The timeout must be > 0. A timeout of zero is interpreted as an infinite timeout.
Are you able to set a realistic timeout value?

After reading David's answer I observed the behavior of that socket in order to see how does it work.
As David stated, SO_TIMEOUT only applies to future read() calls and not the ones that have already been called.
The common pattern for my tests was:
Connect the user (socket)
Exchange some data
Server does not communicate with user for some time
At this point client has already entered the read method. Setting SO_TIMEOUT at this point is useless.
Block the server's OUTBOUND traffic on local port (some random 55000+ port)
Wait for SocketTimeoutException
If I was to set the SO_TIMEOUT before, my socket would throw the exceptions like mad. So, I was hoping and was wrong when excepting for exception to be thrown in case of network outage. On the contrary, exception is being throw in case of traffic outage.
Can this be an issue with Keep-Alive? My socket sets it to true.
Solution (empiric):
I measured (on Windows) that it takes approximately 60 seconds for socket to go down.
So, when I need to sever the connection, I create a thread that checks every 2 seconds if current time is greater from (creation time + 60s). When it reaches that time, it invokes socket.close which effectively causes SocketException.
This solution is by no means optimal but it will have to do, at least for now.
Hope this will be of any help to someone else out here..

Detecting Socket Disconnect Using TCP KeepAlive

I'm developing a server that hosts 3rd party devices over TCP/IP and have been experiencing sudden connection drops (the devices are connecting via cellular). I need to find a way to detect a disconnect without having to write data to the device itself.
I've looked at using the TCP keepalive functionality but Java doesn't appear to allow any adjustment of the timing of the keepalive operations.
Is there any suggested method for doing this?
My simplified socket code is as follows:
public class Test2Socket {
public static void main(String[] args) {
try {
ServerSocket skt = new ServerSocket(1111);
Socket clientSocket = skt.accept();
clientSocket.setKeepAlive(true);
System.out.println("Connected..");
BufferedReader input = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
String inputLine;
while((inputLine = input.readLine()) != null)
{
System.out.println(inputLine);
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
Any feedback would be greatly appreciated.

You will not get far with the built-in keep-alives of the TCP stack. That's because the keep-alive interval cannot be tuned by your application, it is set by the OS, and the defaults are rather high (hours). This is not specific to Java.
If you need to time out in a reasonable time, you have to implement some kind of keep alive in the protocol to be used. Most of the high-level protocols I have seen have some kind of NOP functionality, where you send an "Are you there?" message and the other party sends a "Yes, I'm here" reply without doing anything else.

For an in-depth discussion of TCP Keep-Alives see my answer here.
But basically TCP Keep-Alives are likely the best method for detecting a stale connection. The main problem is that OS defaults are set at 2 hours before the connection is checked with 11 more minutes of Keep-Alive packets before the connection will actually be dropped.
Don't write your own application-layer Keep Alive protocol when TCP already has it built in. All you have to do is set the TCP time out to something more reasonable like 2-3 minutes.
Unfortunately, since TCP timeouts are managed at the OS level and not from within the JVM, it is difficult (but not impossible) to configure TCP timeouts from within your code on a per-socket basis.

When you call setKeepalive() on a socket the system parameters (which are tunable) are used. (Checked it under Debian 8 with openjdk7.)
Because I needed exactly the same functionality, I wrote a small library called libdontdie that can be preloaded and works with Java.

Set a read timeout, with setSoTimeout(), to a reasonable value, say double the expected response time, and catch the resulting SocketTimeoutException.
NB there is no such thing as "Java TCP KeepAlive".

author of this library
here :-)
As has been stated before, unless you rely on difficult-to-configure functionality implemented directly in the TCP protocol (keep-alives), you can hardly detect breakdowns without actually sending data over a connection.
The referenced library encapsulates traditional Java Sockets, while adding easily configurable periodic connectivity checks and ACKing. (Those will stay invisible to the application programmer).
As soon as a breakdown is detected, all inscribed observers will be notified. (Depending on you configuration that can be close to real-time).
The upside is that the lib is rather easy to use (and at least in my own projects works quite reliable).
The downside is, the application layer is not meant to be used for connectivity checks. So basically that lib is using things in a way they are probably not meant to.
Side note:
You mentioned your clients are cellulars. While the linked library will work on android, you might have a look at Push Notification Services instead of dealing with connectivity issues yourself. That could improve e.g. battery consumption.

Faster detection of a broken socket in Java/Android

Background
My application gathers data from the phone and sends the to a remote server.
The data is first stored in memory (or on file when it's big enough) and every X seconds or so the application flushes that data and sends it to the server.
It's mission critical that every single piece of data is sent successfully, I'd rather send the data twice than not at all.
Problem
As a test I set up the app to send data with a timestamp every 5 seconds, this means that every 5 seconds a new line appear on the server.
If I kill the server I expect the lines to stop, they should now be written to memory instead.
When I enable the server again I should be able to confirm that no events are missing.
The problem however is that when I kill the server it takes about 20 seconds for IO operations to start failing meaning that during those 20 seconds the app happily sends the events and removes them from memory but they never reach the server and are lost forever.
I need a way to make certain that the data actually reaches the server.
This is possibly one of the more basic TCP questions but non the less, I haven't found any solution to it.
Stuff I've tried
Setting Socket.setTcpNoDelay(true)
Removing all buffered writers and just using OutputStream directly
Flushing the stream after every send
Additional info
I cannot change how the server responds meaning I can't tell the server to acknowledge the data (more than mechanics of TCP that is), the server will just silently accept the data without sending anything back.
Snippet of code
Initialization of the class:
socket = new Socket(host, port);
socket.setTcpNoDelay(true);
Where data is sent:
while(!dataList.isEmpty()) {
String data = dataList.removeFirst();
inMemoryCount -= data.length();
try {
OutputStream os = socket.getOutputStream();
os.write(data.getBytes());
os.flush();
}
catch(IOException e) {
inMemoryCount += data.length();
dataList.addFirst(data);
socket = null;
return false;
}
}
return true;
Update 1
I'll say this again, I cannot change the way the server behaves.
It receive data over TCP and UPD and does not send any data back to confirm the receive. This is a fact and sure in a perfect world the server would acknowledge the data but that will simply not happen.
Update 2
The solution posted by Fraggle works perfect (closing the socket and waiting for the input stream to be closed).
This however comes with a new set of problems.
Since I'm on a phone I have to assume that the user cannot send an infinite amount of bytes and I would like to keep all data traffic to a minimum if possible.
I'm not worried by the overhead of opening a new socket, those few bytes will not make a difference. What I am worried about however is that every time I connect to the server I have to send a short string identifying who I am.
The string itself is not that long (around 30 characters) but that adds up if I close and open the socket too often.
One solution is only to "flush" the data every X bytes, the problem is I have to choose X wisely; if too big there will be too much duplicate data sent if the socket goes down and if it's too small the overhead is too big.
Final update
My final solution is to "flush" the socket by closing it every X bytes and if all didn't got well those X bytes will be sent again.
This will possibly create some duplicate events on the server but that can be filtered there.

Dan's solution is the one I'd suggest right after reading your question, he's got my up-vote.
Now can I suggest working around the problem? I don't know if this is possible with your setup, but one way of dealing with badly designed software (this is your server, sorry) is to wrap it, or in fancy-design-pattern-talk provide a facade, or in plain-talk put a proxy in front of your pain-in-the-behind server. Design meaningful ack-based protocol, have the proxy keep enough data samples in memory to be able to detect and tolerate broken connections, etc. etc. In short, have the phone app connect to a proxy residing somewhere on a "server-grade" machine using "good" protocol, then have the proxy connect to the server process using the "bad" protocol. The client is responsible for generating data. The proxy is responsible for dealing with the server.
Just another idea.
Edit 0:
You might find this one entertaining: The ultimate SO_LINGER page, or: why is my tcp not reliable.

The bad news: You can't detect a failed connection except by trying to send or receive data on that connection.
The good news: As you say, it's OK if you send duplicate data. So your solution is not to worry about detecting failure in less than the 20 seconds it now takes. Instead, simply keep a circular buffer containing the last 30 or 60 seconds' worth of data. Each time you detect a failure and then reconnect, you can start the session by resending that saved data.
(This could get to be problematic if the server repeatedly cycles up and down in less than a minute; but if it's doing that, you have other problems to deal with.)

See the accepted answer here: Java Sockets and Dropped Connections
socket.shutdownOutput();
wait for inputStream.read() to return -1, indicating the peer has also shutdown its socket

Won't work: server cannot be modified
Can't your server acknowledge every message it receives with another packet? The client won't remove the messages that the server did not acknowledge yet.
This will have performance implications. To avoid slowing down you can keep on sending messages before an acknowledgement is received, and acknowledge several messages in one return message.
If you send a message every 5 seconds, and disconnection is not detected by the network stack for 30 seconds, you'll have to store just 6 messages. If 6 sent messages are not acknowledged, you can consider the connection to be down. (I suppose that logic of reconnection and backlog sending is already implemented in your app.)

What about sending UDP datagrams on a separate UDP socket while making the remote host respond to each, and then when the remote host doesn't respond, you kill the TCP connection? It detects a link breakage quickly enough :)

Use http POST instead of socket connection, then you can send a response to each post. On the client side you only remove the data from memory if the response indicates success.
Sure its more overhead, but gives you what you want 100% of the time.

"java.net.BindException: Address already in use" when trying to do rapid Socket creation and destruction for load testing

I'm trying to load test a Java server by opening a large number of socket connections to the server, authenticating, closing the connection, then repeating. My app runs great for awhile but eventually I get:
java.net.BindException: Address already in use: connect
According to documentation I read, the reason for this is that closed sockets still occupy the local address assigned to them for a period of time after close() was called. This is OS dependent but can be on the order of minutes. I tried calling setReuseAddress(true) on the socket with the hopes that its address would be reusable immediately after close() was called. Unfortunately this doesn't seem to be the case.
My code for socket creation is:
Socket socket = new Socket();
socket.setReuseAddress(true);
socket.connect(new InetSocketAddress(m_host, m_port));
But I still get this error:
java.net.BindException: Address already in use: connect after awhile.
Is there any other way to accomplish what I'm trying to do? I would like to for instance: open 100 sockets, close them all, open 200 sockets, close them all, open 300, etc. up to a max of 2000 or so sockets.
Any help would be greatly appreciated!

You are exhausing the space of outbound ports by opening that many outbound sockets within the TIME_WAIT period of two minutes. The first question you should ask yourself is does this represent a realistic load test at all? Is a real client really going to do that? If not, you just need to revise your testing methodology.
BTW SO_LINGER is the number of seconds the application will wait during close() for data to be flushed. It is normally zero. The port will hang around for the TIME_WAIT interval anyway if this is the end that issued the close. This is not the same thing. It is possible to abuse the SO_LINGER option to patch the problem. However that will also cause exceptional behaviour at the peer and again this is not the purpose of a test.

Not using bind() but setReuseAddress(true) is just weird, I hope you do understand the implications of setReuseAddress (and the point of). 100-2000 is not a great number of sockets to open, however the server you are attempting to connect to (since it looks the same addr/port pair), may just drop them w/ a normal backlog of 50.
Edit:
if you need to open multiple sockets quickly (ermm port scan?), I'd very strongly recommend using NIO and connect()/finishConnect() + Selector. Opening 1000 sockets in the same thread is just plain slow.
Forgot you may need finishConnect() either way in your code.

I think that you should plan on the port you want to use to connect to be in use. By that I mean try to connect using the given port. If the connect fails (or in your case throws an exception), try to open the connection using the next port number.
Try wrapping the connect statement in a try/catch.
Here's some pseudo-code that conveys what I think will work:
portNumber = x //where x is the first port number you will try
numConnections = 200 // or however many connections you want to open
while(numConnections > 0){
try{
connect(host, portNumber)
numConnections--
}catch(){}
portNumber++
}
This code doesn't cover corner cases such as "what happens when all ports are in use?"