Develop Reference

Netty NIO: Read received messages

I am developing a client and server communication system using Netty NIO in Java. My code can be found in the following repository. Currently I am having one server and two clients and I am sending information from server to the clients and the opposite.
What I am trying to figure out, when I am receiving a message form the first client to the server, how can i send that message to the second client (and the opposite from client 2 to client 1). How can I send a message to a specific client?
I have noticed that my issues arised because of the way that I am trying to send the messages from the server. My code in serverHandler is the following:
for (Channel ch : channels1) {
responseData.setIntValue(channels1.size());
remoteAddr.add(ch.remoteAddress().toString());
future = ch.writeAndFlush(responseData);
//future.addListener(ChannelFutureListener.CLOSE);
System.out.println("the requested data from the clients are: "+requestData);
responseData1.setStringValue(requestData.toString());
future = ch.writeAndFlush(responseData1);
System.out.println(future);
}
By default am sending a message about the number of the connections, but also when I am receiving message from the client 1 or 2 I want to send it back to 2 and 1. So I want to perform the communication between the two components. How can I send from the server to a specific client? I am not sure how can I send the messages back to the clients.

General approach
Let's describe an approach to the problem.
When receiving data on the server side, use the remote address of the channel (the java.net.SocketAddress Channel.remoteAddress() method) to identify the client.
Such identification may be done using a map like: Map<SocketAddress, Client>, where the Client class or interface should contain the appropriate client connection (channel) associated context, including its Channel. Be sure to keep the map up-to-date: handle the «client connected» and «client disconnected» events appropriately.
After a client is identified, you may just send the appropriate messages to the clients, except the current sending client, using the client connection (channel) map.
Additionally, I would like to recommend you to find a good implementation of a chat application using Netty and to take a look at it.
Netty-specific solution
Let's consider the server side implementation, in particular, the implementation of the ProcessingHandler class.
It already manages the active channels by representing them as the channel group:
static final ChannelGroup channels1 =
new DefaultChannelGroup(GlobalEventExecutor.INSTANCE);
Keeping the channel group up-to-date
The current implementation handles the «channel becomes active» event to keep the channel group up-to-date:
#Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
channels1.add(ctx.channel());
// ...
}
But this is only a half: it is necessary to handle the «channel becomes inactive» event symmetrically as well. The implementation should look like:
#Override
public void channelInactive(final ChannelHandlerContext ctx) throws Exception {
channels1.remove(ctx.channel());
}
Broadcasting: Sending the received message to all channels, except the current one
To implement the desired behaviour, just update the implementation by introducing the appropriate check as follows:
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
// ...
for (Channel ch : channels1) {
// Does `ch` represent the channel of the current sending client?
if (ch.equals(ctx.channel())) {
// Skip.
continue;
}
// Send the message to the `ch` channel.
// ...
}
// ...
}
Sending and receiving string problem
Currently, the functionality around the ResponseData class is not present (not implemented).
The following draft changes are required to make both the client and the server work.
The ResponseData class: the getStringValue and toString methods should be corrected:
String getStringValue() {
return this.strValue;
}
#Override
public String toString() {
return intValue + ";" + strValue;
}
The ResponseDataEncoder class: it should use the string value:
private final Charset charset = Charset.forName("UTF-8");
#Override
protected void encode(final ChannelHandlerContext ctx, final ResponseData msg, final ByteBuf out) throws Exception {
out.writeInt(msg.getIntValue());
out.writeInt(msg.getStringValue().length());
out.writeCharSequence(msg.getStringValue(), charset);
}
The ResponseDataDecoder class: it should use the string value:
private final Charset charset = Charset.forName("UTF-8");
#Override
protected void decode(final ChannelHandlerContext ctx, final ByteBuf in, final List<Object> out) throws Exception {
ResponseData data = new ResponseData();
data.setIntValue(in.readInt());
int strLen = in.readInt();
data.setStringValue(in.readCharSequence(strLen, charset).toString());
out.add(data);
}
The ClientHandler class: it should correctly receive and handle the message:
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
final ResponseData responseData = (ResponseData) msg;
System.out.println("The message sent from the server " + responseData);
update.accept(responseData.getIntValue());
}
Additional references
«SecureChat ‐ an TLS-based chat server, derived from the Telnet example», Netty Documentation. In particular, the implementation of the SecureChatServerHandler class.
«Netty in Action», Norman Maurer, Marvin Allen Wolfthal (ISBN-13: 978-1617291470), «Part 3 — Network protocols», the «12.2 Our example WebSocket application» subchapter. Covers implementation of «a browser-based chat application».

Java RabbitMQ consumer.nextMessage always gets same message

We are using Java rabbitMq with spring boot in a distributed service architecture. One service gets an HTTP request and forwards it to an unkown queue for processing. At the same time it has to wait for a response on another queue before it can terminate the HTTP request. (It's a preview request that gets its work done by a renderer).
There can be more than one instance of ServiceA (the HTTP Interface) and ServiceB (the renderer) so with every preview message we also send a unique ID to be used as routing key.
I'm having trouble with the BlockingConsumer. Whenever I call consumer.nextMessage() I get the same message over and over again. This is doubly weird, as for one it should be ACKed and removed from the queue and for another the consumer shouldn't even bother with it as the unique ID we used is no longer bound to the queue. nextMessage even returns before the renderer service is done and has sent its done message back.
Here's the simplified setup:
general
All services use a global DirectExchange for all messages
#Bean
public DirectExchange globalDirectExchange() {
return new DirectExchange(EXCHANGE_NAME, false, true);
}
ServiceA (handles the HTTP request):
private Content requestPreviewByKey(RenderMessage renderMessage, String previewKey) {
String renderDoneRoutingKey= UUID.randomUUID().toString();
renderMessage.setPreviewDoneKey(renderDoneId);
Binding binding = BindingBuilder.bind(previewDoneQueue).to(globalDirectExchange)
.with(renderDoneRoutingKey);
try {
amqpAdmin.declareBinding(binding);
rabbitProducer.sendPreviewRequestToKey(renderMessage, previewKey);
return getContentBlocking();
} catch (Exception e) {
logErrorIfDebug(type, e);
throw new ApiException(BaseErrorCode.COMMUNICATION_ERROR, "Could not render preview");
} finally {
amqpAdmin.removeBinding(binding);
}
}
private Content getContentBlocking() {
BlockingQueueConsumer blockingQueueConsumer = new BlockingQueueConsumer(rabbitMqConfig.connectionFactory(), new DefaultMessagePropertiesConverter(), new ActiveObjectCounter<>(), AcknowledgeMode.AUTO, true, 1, PREVIEW_DONE_QUEUE);
try {
blockingQueueConsumer.start();
Message message = blockingQueueConsumer.nextMessage(waitForPreviewMs);
if (!StringUtils.isEmpty(message)) {
String result = new String(message.getBody());
return JsonUtils.stringToObject(result, Content.class);
}
throw new ApiException("Could not render preview");
} catch (Exception e) {
logError(e);
throw new ApiException("Could not render preview");
} finally {
blockingQueueConsumer.stop();
}
}
Service B
I'll spare you most of the code. My log says everything is going well and as soon as its done the service sends the correct message to the UUID key that was sent with the initial render request.
public void sendPreviewDoneMessage(Content content, String previewDoneKey) {
String message = JsonUtils.objectToString(content);
rabbitTemplate.convertAndSend(globalDirectExchange, previewDoneKey, message);
}
The whole thing works... Once...
The real issues seems to be the consumer setup. Why do I keep getting the same (first) message from the queue when I use nextMessage().
Doesn't creating and removing a Bindung ensure, that only messages bound to that routingKey are even received in that instance? And doesn't nextMessage() acknowledge the message and remove it from the queue?!
Thank's a lot for bearing with me and even more for any helpful answer!

BlockingQueueConsumer is not designed to be used directly; it is a component of the SimpleMessageListenerContainer, which will take care of acking the message after it has been consumed by a listener (the container calls commitIfNecessary).
There may be other unexpected side effects of using this consumer directly.
I strongly advise using the listener container to consume messages.
If you just want to receive messages on demand, use a RabbitTemplate receive() or receiveAndConvert() method instead.

Recovering from HBase server failure using Async HBase client

I'm currently trying to find a way to deal with unexpected HBase failures in my application. More specifically, what I'm trying to solve is a case where my application inserts data to HBase and then HBase fails and restarts.
In order to check how my application reacts to that scenario I wrote an application that uses HBase Async client by doing a tight loop and saving the results in HBase. When I start the application I can see rows are saved into the table, if during this time I intentionally fail my HBase server and restart it the client seems to reconnect but new insertions are not saved into the table
The code looks like this:
HConnection connection = HConnectionManager.createConnection();
HBaseClient hbaseClient = new HBaseClient(connection);
IntStream.range(0, 10000)
.forEach(new IntConsumer() {
#Override
public void accept(int value) {
try {
System.out.println("in value: " + value);
Thread.sleep(2000);
Get get = new Get(Bytes.toBytes("key"));
hbaseClient.get(TableName.valueOf("testTable"), get, new ResponseHandler<Result>() {
#Override
public void onSuccess(Result response) {
System.out.println("SUCCESS");
}
#Override
public void onFailure(IOException e) {
System.out.println("FAILURE");
}
});
urlsClient.save("valuekey", "w" + value, new FailureHandler<IOException>() {
#Override
public void onFailure(IOException failure) {
System.out.println("FAILURE");
}
});
} catch (InterruptedException e) {
e.printStackTrace();
}
}
});
This is obviously just a simple test but what I'm trying to achieve is that the async client will successfully save new rows after I restarted my HBase server. What the asynchronous HBase clients prints to me if I actually print the stacktrace in the "onFailure" method is:
org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=303, waitTime=60096, rpcTimeout=60000
at org.apache.hadoop.hbase.ipc.AsyncRpcChannel.cleanupCalls(AsyncRpcChannel.java:612)
at org.apache.hadoop.hbase.ipc.AsyncRpcChannel$1.run(AsyncRpcChannel.java:119)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:581)
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:655)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:367)
at java.lang.Thread.run(Thread.java:745)
And so my questions are:
How should one deal with a situation like I mentioned using the specified async client?
If this async client is no longer relevant could someone suggest a different async client that can perform asynchronous puts? I tried the BufferedMutator but it does not seem to actually flush any contents but just fails with the following java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator (but this gets a little off topic so I wont expand anymore)
Thanks

It's been quite a long time since I asked this question but I ended up using the HBase high availability instead of finding a way to solve it with code

Implementing xml rpc server on Android

I'm developing an app to exchange data between an Android phone and a pc. I've chosen xml-rpc as a way to communicate, and as a library i downloaded android-xmlrpc (https://code.google.com/p/android-xmlrpc/).
When I call remote procedures from my phone, I use the XMLRPCClient and it works almost fine.
But then I have to synchronize data from my server to the application, so I need to listen on a given port and wait for a xmlrpc request.
I tried the following code:
protected Void doInBackground(Void... params) {
try { ServerSocket socket = new ServerSocket(8888);
XMLRPCServer server = new XMLRPCServer();
while (true) {
Socket client = socket.accept();
MethodCall call = server.readMethodCall(client);
String name = call.getMethodName();
}
} catch (Exception e) { Log.v(LOG_TAG, "Server: Error", e); }
return null;
}
But it seems it never receives any procedure call. I mean, to debug it I'm trying to use my XMLRPCClient as follow:
XMLRPCClient client = new XMLRPCClient("http://127.0.0.1:8888/");
Object o = client.call("add",1,3);
Log.v(LOG_TAG, o.toString());
And I always get no response. It's stuck on readMethodCall and never reads that the method I requested is "add"... It's strange it gets stuck there. Can't figure the reason why it is stucked here. Do you have any idea ?

What can be the best approach to handle java.net.UnknownHostException for AWS users?

My application sends message to Amazon Simple Notification Service (SNS) topic but sometime (6/10) I get java.net.UnknownHostException:sqs.ap-southeast-1.amazonaws.com. The reason of exception is described in the amazon web services discussion forums, please look: https://forums.aws.amazon.com/thread.jspa?messageID=499290&#499290.
My problem is similar to what described in forums of amazon but my rate of publishing messages to topic is very dynamic. It can be 1 message/second or 1 message/minute or no message in an hour. I am looking for a cleaner, better and safe approach, which guaranties sending of message to SNS topic.
Description of problem in detail:
Topic_Arn= arn of SNS topic where application wants to publish message
msg = Message to send in topic
// Just a sample example which publish message to Amazon SNS topic
class SimpleNotificationService {
AmazonSNSClient mSnsClient = null;
static {
createSnsClient()
}
private void static createSnsClient() {
Region region = Region.getRegion(Regions.AP_SOUTHEAST_1);
AWSCredentials credentials = new
BasicAWSCredentials(AwsPropertyLoader.getInstance().getAccessKey(),
AwsPropertyLoader.getInstance().getSecretKey());
mSqsClient = new AmazonSQSClient(credentials);
mSqsClient.setRegion(region);
}
public void static publishMessage(String Topic_Arn, String msg) {
PublishRequest req = new PublishRequest(Topic_Arn, msg);
mSnsClient.publish(req);
}
}
class which calls SimpleNotificationService
class MessagingManager {
public void sendMessage(String message) {
String topic_arn = "arn:of:amazon:sns:topic";
SimpleNotificationService.publishMessage(topic_arn, message);
}
}
Please note that this is a sample code, not my actual code. Here can be class design issue but please ignore those if they are not related to problem.
My thought process says to have try-catch block inside sendMessage, so when we catch UnknownHostException then again retry but I am not sure how to write this in safer, cleaner and better way.
So MessagingManager class will look something like this:
class MessagingManager {
public void sendMessage(String message) {
String topic_arn = "arn:of:amazon:sns:topic";
try {
SimpleNotificationService.publishMessage(topic_arn, message);
} catch (UnknownHostException uhe) {
// I need to catch AmazonClientException as aws throws
//AmazonClientException when sees UnknownHostException.
// I am mentioning UnknownHostException for non-aws user to understand
// my problem in better way.
sendMessage(message); // Isn't unsafe? - may falls into infinite loop
}
}
}
I am open for answers like this: java.net.UnknownHostException: Invalid hostname for server: local but my concern is to dependent on solution at application code-level and less dependent on changes to machine. As my server application is going to run in many boxes (developer boxes, testing boxes or production boxes). If changes in machine host-files or etc is only guaranted solution then I prefer that to include with code level changes.

Each AWS SDK implements automatic retry logic. The AWS SDK for Java automatically retries requests, and you can configure the retry settings using the ClientConfiguration class.
Below is the sample example to create SNS client. It retries for 25 times if encounters UnKnownHostException. It uses default BackOff and retry strategy. If you want to have your own then you need to implement these two interfaces: http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/retry/RetryPolicy.html
private void static createSnsClient() {
Region region = Region.getRegion(Regions.AP_SOUTHEAST_1);
AWSCredentials credentials = new
BasicAWSCredentials(AwsPropertyLoader.getInstance().getAccessKey(),
AwsPropertyLoader.getInstance().getSecretKey());
ClientConfiguration clientConfiguration = new ClientConfiguration();
clientConfiguration.setMaxErrorRetry(25);
clientConfiguration.setRetryPolicy(new RetryPolicy(null, null, 25, true));
mSnsClient = new AmazonSNSClient(credentials, clientConfiguration);
mSnsClient.setRegion(region);
}

Have you considering looking into the JVM TTL for the DNS Cache?
http://docs.aws.amazon.com/AWSSdkDocsJava/latest//DeveloperGuide/java-dg-jvm-ttl.html