Develop Reference

httpClient: How to check whether the connection was already closed by the server side

We connect to a data provider and keep reading XML data in an endless loop using HttpClient. We already set connection timeout and socket timeout in the Java code. However, when the connection is closed on the HTTP server side, our application does not throw any exception and just hangs there doing nothing. Because of this behavior, when the server is up the Java code will not reconnect. Is there any way to see whether the socket connection was already closed by the server side?
In addition, https://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html mentioned how to close STALE connections, but it still does not solve our problem.
Thank you very much in advance!
Code snippet:
private static final HttpClientConnectionManager CONN_MGR = new BasicHttpClientConnectionManager();
public boolean connectToServer() {
disconnect();
CredentialsProvider provider = new BasicCredentialsProvider();
UsernamePasswordCredentials credentials = new UsernamePasswordCredentials(username, password);
provider.setCredentials(AuthScope.ANY, credentials);
RequestConfig requestConfig = RequestConfig.custom().setConnectTimeout(15 * 1000)
.setSocketTimeout(15 * 1000).build();
HttpClient client = HttpClientBuilder.create().setDefaultCredentialsProvider(provider)
.setSSLSocketFactory(sslsf)
.setConnectionManager(CONN_MGR)
.setDefaultRequestConfig(requestConfig).build();
HttpGet request = new HttpGet(url);
try {
HttpResponse response = client.execute(request);
in = response.getEntity().getContent();
reader = new BufferedReader(new InputStreamReader(in));
connected = true;
return true;
} catch (Exception re) {
LOGGER.error("error", re);
}
return false;
}
public void process() {
String xml = null;
while (!shuttingDown) {
try {
/* if we know the server side closed the connection already here
we can simply return and scheduler will take care of anything else. */
xml = reader.readLine();
lastSeen = System.currentTimeMillis();
if (StringUtils.isBlank(xml)) {
continue;
}
xml = xml.trim();
// processing XML here
} catch (IOException | NullPointerException ie) {
/* We see SocketTimeoutException relatively often and
* sometimes NullPointerException in reader.readLine(). */
if (!shuttingDown) {
if(!connectToServer()) {
break;
}
}
} catch (RuntimeException re) {
LOGGER.error("other RuntimeException", re);
break;
}
}
// the scheduler will start another processing.
disconnect();
}
// called by the scheduler periodically
public boolean isConnected() {
CONN_MGR.closeExpiredConnections();
CONN_MGR.closeIdleConnections(15, TimeUnit.SECONDS);
long now = System.currentTimeMillis();
if (now - lastSeen > 15000L) {
LOGGER.info("call disconnect() from isConnected().");
disconnect();
}
return connected;
}

I think with the introduction of Automatic Resource management in java 1.7
https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html
cleaning up of stale connections has been taken care and even on the worst case it has been taken care by the implementation providers carefully by monitoring the connection
Apache HttpClient uses HttpClientConnectionManager and other utility classes that does the job of cleaning the stale connections.

After the last catch block add a finally block for your cleanup code.
finally {
CONN_MGR.shutdown();
}
As per documentation it shuts down connection manager and releases allocated resources

We avoided the problem by closing the connection from another thread when no data coming in for more than 15 seconds. More specifically now isConnected() method is as follows:
// called by the scheduler periodically
public boolean isConnected() {
long now = System.currentTimeMillis();
if (now - lastSeen > 15000L) {
disconnect();
}
return connected;
}
Of course, the reading part get will java.net.SocketException: Socket closed or java.lang.IllegalStateException: Connection is still allocated when this happens.
After this change everything works fine.

How can I check if a hazelcast cluster is alive from a java client?

We use hazelcast in client-server mode. The hazelcast cluster contains 2 hazelcast nodes and we have about 25 clients connected to the cluster.
What I am lookin for now is a simple check that tries to figure out if the cluster is still alive. It should be a rather cheap operation because this check will occure on every client quite frequently (once every second I could imagine).
What is the best way to do so?

The simplest way would be the register a LifecycleListener to the client HazelcastInstance:
HazelcastInstance client = HazelcastClient.newHazelcastClient();
client.getLifecycleService().addLifecycleListener(new LifecycleListener() {
#Override
public void stateChanged(LifecycleEvent event) {
}
})
The client uses a periodic heartbeat to detect if the cluster is still running.

You can use the LifecycleService.isRunning() method as well:
HazelcastInstance hzInstance = HazelcastClient.newHazelcastClient();
hzInstance.getLifecycleService().isRunning()

As isRunning() may be true even if cluster is down, I'd go for the following approach (a mixture of #konstantin-zyubin's answer and this). This doesn't need an event-listener, which is an advantage in my setup:
if (!hazelcastInstance.getLifecycleService().isRunning()) {
return Health.down().build();
}
int parameterCount;
LocalTopicStats topicStats;
try {
parameterCount = hazelcastInstance.getMap("parameters").size();
topicStats = hazelcastInstance.getTopic("myTopic").getLocalTopicStats();
} catch (Exception e) {
// instance may run but cluster is down:
Health.Builder builder = Health.down();
builder.withDetail("Error", e.getMessage());
return builder.build();
}
Health.Builder builder = Health.up();
builder.withDetail("parameterCount", parameterCount);
builder.withDetail("receivedMsgs", topicStats.getReceiveOperationCount());
builder.withDetail("publishedMsgs", topicStats.getPublishOperationCount());
return builder.build();

I have found a more reliable way to check hazelcast availability, because
client.getLifecycleService().isRunning()
when you use async reconnection mode is always return true, as was mentioned.
#Slf4j
public class DistributedCacheServiceImpl implements DistributedCacheService {
private HazelcastInstance client;
#Autowired
protected ConfigLoader<ServersConfig> serversConfigLoader;
#PostConstruct
private void initHazelcastClient() {
ClientConfig config = new ClientConfig();
if (isCacheEnabled()) {
ServersConfig.Hazelсast hazelcastConfig = getWidgetCacheSettings().getHazelcast();
config.getGroupConfig().setName(hazelcastConfig.getName());
config.getGroupConfig().setPassword(hazelcastConfig.getPassword());
for (String address : hazelcastConfig.getAddresses()) {
config.getNetworkConfig().addAddress(address);
}
config.getConnectionStrategyConfig()
.setAsyncStart(true)
.setReconnectMode(ClientConnectionStrategyConfig.ReconnectMode.ASYNC);
config.getNetworkConfig()
.setConnectionAttemptLimit(0) // infinite (Integer.MAX_VALUE) attempts to reconnect
.setConnectionTimeout(5000);
client = HazelcastClient.newHazelcastClient(config);
}
}
#Override
public boolean isCacheEnabled() {
ServersConfig.WidgetCache widgetCache = getWidgetCacheSettings();
return widgetCache != null && widgetCache.getEnabled();
}
#Override
public boolean isCacheAlive() {
boolean aliveResult = false;
if (isCacheEnabled() && client != null) {
try {
IMap<Object, Object> defaultMap = client.getMap("default");
if (defaultMap != null) {
defaultMap.size(); // will throw Hazelcast exception if cluster is down
aliveResult = true;
}
} catch (Exception e) {
log.error("Connection to hazelcast cluster is lost. Reason : {}", e.getMessage());
}
}
return aliveResult;
}
}

Check client is alive with HttpServletRequest object

I'm writing a Spring web application and I'm mapping the "/do" URL path to the following Controller's method
#Controller
public class MyController
{
#RequestMapping(value="/do", method=RequestMethod.GET)
public String do()
{
File f = new File("otherMethodEnded.tmp");
while (!f.exists())
{
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
}
}
// ok, let's continue
}
}
The otherMethodEnded.tmp file is written by one another Controller's method, so when the client calls the second URL I expect the first method to exit the while loop.
Everything works, except when the client calls the "/do" URL and then closes the connection before the response was received. The problem is that the server remains in the while (!f.exists()) loop even though the client is down and cannot call the second URL to unlock the while loop.
I would try to retrieve the connection status of the "/do" URL and exit the loop when the connection is closed by the client, but I cannot find any way to do so.
I tried with the HttpServletRequest.getSession(false) method but the returned HttpSession object is always not null, so the HttpServletRequest object is not updated in case of connection close of the client.
How can I check whether the client is still waiting for the risponse or not?

The simplest way to verify something is not right is to define a timeout value and then during your loop test if your time spent waiting has exceeded the timeout.
something like:
#Controller
public class MyController
{
private static final long MAX_LOOP_TIME = 1000 * 60 * 5; // 5 minutes? choose a value
#RequestMapping(value="/do", method=RequestMethod.GET)
public String do()
{
File f = new File("otherMethodEnded.tmp");
long startedAt = System.currentTimeMillis()
boolean forcedExit = false;
while (!forcedExit && !f.exists())
{
try {
Thread.sleep(5000);
if (System.currentTimeMillis() - startedAt > MAX_LOOP_TIME) {
forcedExit = true;
}
} catch (InterruptedException e) {
forcedExit = true;
}
}
// ok, let's continue
// if forcedExit , handle error scenario?
}
}
Additionally: InterruptedException is not something to blindly catch and ignore. see this discussion
In your case I would really exit the while loop if you're interrupted.
You only know if the client is no longer waiting on your connection when you notice the output stream you write to (response.outputstream) is closed. But there isn't a way to detect it.
(see this question for details)
Seeing as you've indicated your client does occasional callbacks, you could on the clientside poll if the other call has been completed. If this other call has completed, do the operation, otherwise return directly and have the client do the call again. (assuming you are sending json, but adapt as you require)
something like
public class MyController
{
#RequestMapping(value="/do", method=RequestMethod.GET)
public String do()
{
File f = new File("otherMethodEnded.tmp");
if (f.exists()) {
// do what you set out to do
// ok, let's continue
// and return with a response that indicates the call did what it did
// view that returns json { "result" : "success" }
return "viewThatSIgnalsToClientThatOperationSucceeded";
} else {
// view that returns json: { "result" : "retry" }
return "viewThatSignalsToClientToRetryIn5Seconds";
}
}
}
Then the clientside would run something like: (pseudojavascript as it's been a while)
val callinterval = setInterval(function() { checkServer() }, 5000);
function checkServer() {
$.ajax({
// ...
success: successFunction
});
}
function successFunction(response) {
// connection succeeded
var json = $.parseJSON(response);
if (json.result === "retry") {
// interval will call this again
} else {
clearInterval(callinterval);
if (json.result === "success") {
// do the good stuff
} else if (json.result === "failure") {
// report that the server reported an error
}
}
}
Ofcourse this is just semi-serious code but it's roughly how i'd try it if I were to have the dependency. If this is regarding afile upload, keep in mind that this file may not contain all of the bytes yet. file exists != file = completely uploaded, unless you use move it. cp / scp / etc. is not atomic.

Why I am seeing lot of TimeoutException if any one server goes down?

Here is my DataClientFactory class.
public class DataClientFactory {
public static IClient getInstance() {
return ClientHolder.INSTANCE;
}
private static class ClientHolder {
private static final DataClient INSTANCE = new DataClient();
static {
new DataScheduler().startScheduleTask();
}
}
}
Here is my DataClient class.
public class DataClient implements IClient {
private ExecutorService service = Executors.newFixedThreadPool(15);
private RestTemplate restTemplate = new RestTemplate();
// for initialization purpose
public DataClient() {
try {
new DataScheduler().callDataService();
} catch (Exception ex) { // swallow the exception
// log exception
}
}
#Override
public DataResponse getDataSync(DataKey dataKeys) {
DataResponse response = null;
try {
Future<DataResponse> handle = getDataAsync(dataKeys);
response = handle.get(dataKeys.getTimeout(), TimeUnit.MILLISECONDS);
} catch (TimeoutException e) {
// log error
response = new DataResponse(null, DataErrorEnum.CLIENT_TIMEOUT, DataStatusEnum.ERROR);
} catch (Exception e) {
// log error
response = new DataResponse(null, DataErrorEnum.ERROR_CLIENT, DataStatusEnum.ERROR);
}
return response;
}
#Override
public Future<DataResponse> getDataAsync(DataKey dataKeys) {
Future<DataResponse> future = null;
try {
DataTask dataTask = new DataTask(dataKeys, restTemplate);
future = service.submit(dataTask);
} catch (Exception ex) {
// log error
}
return future;
}
}
I get my client instance from the above factory as shown below and then make a call to getDataSync method by passing DataKey object. DataKey object has userId and Timeout values in it. Now after this, call goes to my DataTask class to call method as soon as handle.get is called.
IClient dataClient = DataClientFactory.getInstance();
long userid = 1234l;
long timeout_ms = 500;
DataKey keys = new DataKey.Builder().setUserId(userid).setTimeout(timeout_ms)
.remoteFlag(false).secondaryFlag(true).build();
// call getDataSync method
DataResponse dataResponse = dataClient.getDataSync(keys);
System.out.println(dataResponse);
Here is my DataTask class which has all the logic -
public class DataTask implements Callable<DataResponse> {
private DataKey dataKeys;
private RestTemplate restTemplate;
public DataTask(DataKey dataKeys, RestTemplate restTemplate) {
this.restTemplate = restTemplate;
this.dataKeys = dataKeys;
}
#Override
public DataResponse call() {
DataResponse dataResponse = null;
ResponseEntity<String> response = null;
int serialId = getSerialIdFromUserId();
boolean remoteFlag = dataKeys.isRemoteFlag();
boolean secondaryFlag = dataKeys.isSecondaryFlag();
List<String> hostnames = new LinkedList<String>();
Mappings mappings = ClientData.getMappings(dataKeys.whichFlow());
String localPrimaryAdress = null;
String remotePrimaryAdress = null;
String localSecondaryAdress = null;
String remoteSecondaryAdress = null;
// use mappings object to get above Address by using serialId and basis on
// remoteFlag and secondaryFlag populate the hostnames linked list
if (remoteFlag && secondaryFlag) {
hostnames.add(localPrimaryHostIPAdress);
hostnames.add(localSecondaryHostIPAdress);
hostnames.add(remotePrimaryHostIPAdress);
hostnames.add(remoteSecondaryHostIPAdress);
} else if (remoteFlag && !secondaryFlag) {
hostnames.add(localPrimaryHostIPAdress);
hostnames.add(remotePrimaryHostIPAdress);
} else if (!remoteFlag && !secondaryFlag) {
hostnames.add(localPrimaryHostIPAdress);
} else if (!remoteFlag && secondaryFlag) {
hostnames.add(localPrimaryHostIPAdress);
hostnames.add(localSecondaryHostIPAdress);
}
for (String hostname : hostnames) {
// If host name is null or host name is in local block host list, skip sending request to this host
if (hostname == null || ClientData.isHostBlocked(hostname)) {
continue;
}
try {
String url = generateURL(hostname);
response = restTemplate.exchange(url, HttpMethod.GET, dataKeys.getEntity(), String.class);
// make DataResponse
break;
} catch (HttpClientErrorException ex) {
// make DataResponse
return dataResponse;
} catch (HttpServerErrorException ex) {
// make DataResponse
return dataResponse;
} catch (RestClientException ex) {
// If it comes here, then it means some of the servers are down.
// Add this server to block host list
ClientData.blockHost(hostname);
// log an error
} catch (Exception ex) {
// If it comes here, then it means some weird things has happened.
// log an error
// make DataResponse
}
}
return dataResponse;
}
private String generateURL(final String hostIPAdress) {
// make an url
}
private int getSerialIdFromUserId() {
// get the id
}
}
Now basis on userId, I will get the serialId and then get the list of hostnames, I am suppose to make a call depending on what flag is passed. Then I iterate the hostnames list and make a call to the servers. Let's say, if I have four hostnames (A, B, C, D) in the linked list, then I will make call to A first and if I get the data back, then return the DataResponse back. But suppose if A is down, then I need to add A to block list instantly so that no other threads can make a call to A hostname. And then make a call to hostname B and get the data back and return the response (or repeat the same thing if B is also down).
I have a background thread as well which runs every 10 minutes and it gets started as soon we get the client instance from the factory and it parses my another service URL to get the list of block hostnames that we are not supposed to make a call. Since it runs every 10 minutes so any servers which are down, it will get the list after 10 minutes only, In general suppose if A is down, then my service will provide A as the block list of hostnames and as soon as A becomes up, then that list will be updated as well after 10 minutes.
Here is my background thread code DataScheduler-
public class DataScheduler {
private RestTemplate restTemplate = new RestTemplate();
private static final Gson gson = new Gson();
private final ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1);
public void startScheduleTask() {
scheduler.scheduleAtFixedRate(new Runnable() {
public void run() {
try {
callDataService();
} catch (Exception ex) {
// log an error
}
}
}, 0, 10L, TimeUnit.MINUTES);
}
public void callDataService() throws Exception {
String url = null;
// execute the url and get the responseMap from it as a string
parseResponse(responseMap);
}
private void parseResponse(Map<FlowsEnum, String> responses) throws Exception {
// .. some code here to calculate partitionMappings
// block list of hostnames
Map<String, List<String>> coloExceptionList = gson.fromJson(response.split("blocklist=")[1], Map.class);
for (Map.Entry<String, List<String>> entry : coloExceptionList.entrySet()) {
for (String hosts : entry.getValue()) {
blockList.add(hosts);
}
}
if (update) {
ClientData.setAllMappings(partitionMappings);
}
// update the block list of hostnames
if (!DataUtils.isEmpty(responses)) {
ClientData.replaceBlockedHosts(blockList);
}
}
}
And here is my ClientData class which holds all the information for block list of hostnames and partitionMappings details (which is use to get the list of valid hostnames).
public class ClientData {
private static final AtomicReference<ConcurrentHashMap<String, String>> blockedHosts = new AtomicReference<ConcurrentHashMap<String, String>>(
new ConcurrentHashMap<String, String>());
// some code here to set the partitionMappings by using CountDownLatch
// so that read is blocked for first time reads
public static boolean isHostBlocked(String hostName) {
return blockedHosts.get().contains(hostName);
}
public static void blockHost(String hostName) {
blockedHosts.get().put(hostName, hostName);
}
public static void replaceBlockedHosts(List<String> blockList) {
ConcurrentHashMap<String, String> newBlockedHosts = new ConcurrentHashMap<>();
for (String hostName : blockList) {
newBlockedHosts.put(hostName, hostName);
}
blockedHosts.set(newBlockedHosts);
}
}
Problem Statement:-
When all the servers are up (A,B,C,D as an example) above code works fine and I don't see any TimeoutException happening at all from the handle.get but if let's say one server (A) went down which I was supposed to make a call from the main thread then I start seeing lot of TimeoutException, by lot I mean, huge number of client timeouts happening.
And I am not sure why this is happening? In general this won't be happening right since as soon as the server goes down, it will get added to blockList and then no thread will be making a call to that server, instead it will try another server in the list? So it should be smooth process and then as soon as those servers are up, blockList will get updated from the background thread and then you can start making a call.
Is there any problem in my above code which can cause this problem? Any suggestions will be of great help.
In general, what I am trying to do is - make a hostnames list depending on what user id being passed by using the mappings object. And then make a call to the first hostname and get the response back. But if that hostname is down, then add to the block list and make a call to the second hostname in the list.
Here is the Stacktrace which I am seeing -
java.util.concurrent.TimeoutException\n\tat java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:258)
java.util.concurrent.FutureTask.get(FutureTask.java:119)\n\tat com.host.client.DataClient.getDataSync(DataClient.java:20)\n\tat
NOTE: For multiple userId's, we can have same server, meaning server A can get resolve to multiple userId's.

In DataClient class, at the below line:
public class DataClient implements IClient {
----code code---
Future<DataResponse> handle = getDataAsync(dataKeys);
//BELOW LINE IS PROBLEM
response = handle.get(dataKeys.getTimeout(), TimeUnit.MILLISECONDS); // <--- HERE
} catch (TimeoutException e) {
// log error
response = new DataResponse(null, DataErrorEnum.CLIENT_TIMEOUT, DataStatusEnum.ERROR);
} catch (Exception e) {
// log error
response = new DataResponse(null, DataErrorEnum.ERROR_CLIENT, DataStatusEnum.ERROR);
----code code-----
You have assigned a timeout to handle.get(...), which is timing out before your REST connections can respond. The rest connections themselves may or may not be timing out, but since you are timing out of get method of future before the completion of the execution of the thread, the blocking of hosts has no visible effect, while the code inside the call method of DataTask may be performing as expected. Hope this helps.

You asked about suggestions, so here are some suggestions:
1.) Unexpected return value
Method returns unexpectedly FALSE
if (ClientData.isHostBlocked(hostname)) //this may return always false! please check
2.) Exception-Handling
Are you really sure, that a RestClientException occurs?
Only when this exception occured, the host will be added to blocked list!
Your posted code seems to ignore logging (it is commented out!)
...catch (HttpClientErrorException ex) {
// make DataResponse
return dataResponse;
} catch (HttpServerErrorException ex) {
// make DataResponse
return dataResponse;
} catch (RestClientException ex) {
// If it comes here, then it means some of the servers are down.
// Add this server to block host list
ClientData.blockHost(hostname);
// log an error
} catch (Exception ex) {
// If it comes here, then it means some weird things has happened.
// log an error
// make DataResponse
}

MINA: Performing synchronous write requests / read responses

I'm attempting to perform a synchronous write/read in a demux-based client application with MINA 2.0 RC1, but it seems to get stuck. Here is my code:
public boolean login(final String username, final String password) {
// block inbound messages
session.getConfig().setUseReadOperation(true);
// send the login request
final LoginRequest loginRequest = new LoginRequest(username, password);
final WriteFuture writeFuture = session.write(loginRequest);
writeFuture.awaitUninterruptibly();
if (writeFuture.getException() != null) {
session.getConfig().setUseReadOperation(false);
return false;
}
// retrieve the login response
final ReadFuture readFuture = session.read();
readFuture.awaitUninterruptibly();
if (readFuture.getException() != null) {
session.getConfig().setUseReadOperation(false);
return false;
}
// stop blocking inbound messages
session.getConfig().setUseReadOperation(false);
// determine if the login info provided was valid
final LoginResponse loginResponse = (LoginResponse)readFuture.getMessage();
return loginResponse.getSuccess();
}
I can see on the server side that the LoginRequest object is retrieved, and a LoginResponse message is sent. On the client side, the DemuxingProtocolCodecFactory receives the response, but after throwing in some logging, I can see that the client gets stuck on the call to readFuture.awaitUninterruptibly().
I can't for the life of me figure out why it is stuck here based upon my own code. I properly set the read operation to true on the session config, meaning that messages should be blocked. However, it seems as if the message no longer exists by time I try to read response messages synchronously.
Any clues as to why this won't work for me?

The reason this wasn't working for me was because of an issue elsewhere in my code where I stupidly neglected to implement the message response encoder/decoder. Ugh. Anyway, the code in my question worked as soon as I fixed that.

I prefer this one (Christian Mueller : http://apache-mina.10907.n7.nabble.com/Mina-Client-which-sends-receives-messages-synchronous-td35672.html)
public class UCPClient {
private Map<Integer, BlockingQueue<UCPMessageResponse>> concurrentMap = new ConcurrentHashMap<Integer, BlockingQueue<UCPMessageResponse>>();
// some other code
public UCPMessageResponse send(UCPMessageRequest request) throws Throwable {
BlockingQueue<UCPMessageResponse> queue = new LinkedBlockingQueue<UCPMessageResponse>(1);
UCPMessageResponse res = null;
try {
if (sendSync) {
concurrentMap.put(Integer.valueOf(request.getTransactionReference()), queue);
}
WriteFuture writeFuture = session.write(request);
if (sendSync) {
boolean isSent = writeFuture.await(transactionTimeout, TimeUnit.MILLISECONDS);
if (!isSent) {
throw new TimeoutException("Could not sent the request in " + transactionTimeout + " milliseconds.");
}
if (writeFuture.getException() != null) {
throw writeFuture.getException();
}
res = queue.poll(transactionTimeout, TimeUnit.MILLISECONDS);
if (res == null) {
throw new TimeoutException("Could not receive the response in " + transactionTimeout + " milliseconds.");
}
}
} finally {
if (sendSync) {
concurrentMap.remove(Integer.valueOf(request.getTransactionReference()));
}
}
return res;
}
}
and the IoHandler:
public class InnerHandler implements IoHandler {
// some other code
public void messageReceived(IoSession session, Object message) throws Exception {
if (sendSync) {
UCPMessageResponse res = (UCPMessageResponse) message;
BlockingQueue<UCPMessageResponse> queue = concurrentMap.get(res.getTransactionReference());
queue.offer(res);
}
}
}

I had this exact problem. It turns out that it's because I was doing reads/writes in my IoHandler.sessionCreated() implementation. I moved the processing onto the thread that established the connection, instead of just waiting for the close future.

You must not use your login() function in IoHandler Thread :
If you call IoFuture.awaitUninterruptibly() in the override event function of IoHandler,
IoHandler don't work and get stuck.
You can call login() in other Thread and it will be work properly.