Develop Reference

Sending messages on same socket connection while a file is being sent

My app can transfer files and messages between server and client. Server is multithreaded and clients simply connects to it. While file is being transferred, if sender sends a message, it will be consumed as bytes of file.
I don't want to open more ports,
Can I establish a new connection to the server for file transfer? Or I
should open a separate port for files.
I don't want to block communication while a file is being transferred.
The question was marked as a duplicate but its not, i am trying to send messages and files simultaneously not one by one. I can already receive files one by one. Read again.
Also, as server is multithreaded, I cannot call server socket.accept() again to receive files in new connection because main thread listening for incoming will try to handle it instead. Is there a way around?

Seems to me like trying to multiplex files and messages onto the same socket stream is an XYProblem.
I am not an expert on this, but it sounds like you should do some reading on "ports vs sockets". My understanding is that ip:port is the address of the listening service. Once a client connects, the server will open a socket to actually do the communication.
The trick is that every time a client connects, spawn a new thread (on a new socket) to handle the request. This instantly frees up the main thread to go back to listening for new connections. Your file transfer and your messages can come into the same port, but each new request will get its own socket and its own server thread --> no collision!
See this question for a java implementation:
Multithreading Socket communication Client/Server

you could use some system of all the lines of a file start with a string like this (file:linenum) and then on the other side it puts that in a file then to send text you could do the same thing but with a tag like (text)
Server:
Scanner in = new Scanner(s.getInputStream());
while(true) {
String message = in.nextLine();
if(message.length > 14 && message.substring(0,6).equalsIgnoreCase("(file:") {
int line = Integer.valueOf(message.substring(6).replaceall(")", ""));
saveToFile(message.substring(6).replaceAll(")","").replaceAll("<1-9>",""));
} else {
System.out.println(message);
}
}
I think that code works but I haven't checked it so it might need some slight modifications

You could introduce a handshake protocol where clients can state who they are (probably happening already) and what they want from the given connection. The first connection they make could be about control, and perhaps the messages, and remain in use all the time. File transfer could happen via secondary connections, which may come and go during a session. Having several parallel connections between a client and a server is completely normal, that is what #MikeOunsworth was explaining too.
A shortcut you can take is issuing short-living, one-time tokens which clients can present when opening the secondary connection and then the server will immediately know which file it should start sending. Note that this approach easily can raise various security (if token encodes actual request data) and/or scalability issues (if token is something completely random and has to be looked up in some table).

Disabling HTTP protocol requests within Netty?

I'm writing a game server in Java, and I'm using Netty 3.6.2. The game servers should accept no HTTP requests, as they simply handle game client data (which is purely bytes over TCP). When I load http://server-ip:game-servers-port in Chrome, I download a file with the game's handshake packet (which should not happen).
I bind to the game server's port like so:
ChannelFactory factory = new NioServerSocketChannelFactory(Executors.newCachedThreadPool(), Executors.newCachedThreadPool());
clientAcceptor = new ServerBootstrap(factory);
clientAcceptor.setOption("child.tcpNoDelay", true);
clientAcceptor.setOption("child.keepAlive", false);
clientAcceptor.setPipelineFactory(() -> Channels.pipeline(new PacketDecoder(), new ClientHandler()));
clientAcceptor.bind(new InetSocketAddress(Configurations.CHANNEL_GAME_PORT));
And I process requests in a SimpleChannelHandler, like so
#Override
public void channelConnected(ChannelHandlerContext ctx, ChannelStateEvent e) {
// Encryption key code.
Packets.sendHello(ctx.getChannel(), ivRecv, ivSend);
ctx.getChannel().setAttachment(client);
}
How can I go about deciphering if an incoming request is using the HTTP protocol?
Edit: I should also say, it should also block any FTP, WebSocket, etc. protocol (essentially anything that isn't the game's protocol) on channel connect.

You can't expect to be able to disable HTTP requests if the first thing you do with an accepted connection is to send something.
Have the client do the first send in the handshake. Have the server start by doing a receive, and if it isn't the correct initial handshake packet, close the connection.

Apache Thrift tutorial client stuck when using 2 clients - how to make the server multitask?

I'm executing Apache Thrift tutorial for Java.
When running 2 client processes at the same time, the server doesn't accept the 2nd client. Only after the first client finishes, the second one is accepted by the server.
Can anyone explain what's going on?
How can I make the server accept several connections in several threads?

Can anyone explain what's going on?
You already found it out: The TSimpleServer allows only for one connection at a time. It will be available again when the first client disconnects.
How can I make the server accept several connections in several threads?
Use one of the threading servers, whichever fits your use case best.
TThreadPoolServer
TThreadedSelectorServer
TNonBlockingServer
The half-sync/half-async server
Please note, that some of the servers require the client to use TFramedTransport.

Based on other answers, below is the code to enable executing multiple clients simultaneously.
Server (simple):
CalculatorHandler handler = new CalculatorHandler();
Calculator.Processor processor = new Calculator.Processor(handler);
TNonblockingServerSocket serverTransport = new TNonblockingServerSocket(9090);
THsHaServer.Args args = new THsHaServer.Args(serverTransport);
args.processor(processor);
args.transportFactory(new TFramedTransport.Factory());
TServer server = new THsHaServer(args);
server.serve();
Client:
transport = new TSocket("localhost", 9090);
transport.open();
TProtocol protocol = new TBinaryProtocol(new TFramedTransport(transport));
Calculator.Client client = new Calculator.Client(protocol);
perform(client);

Preventing RabbitMQ from blocking upstream services

I have a Spring application that consumes messages on a specific port (say 9001), restructures them and then forwards to a Rabbit MQ server. The code segment is:
private void send(String routingKey, String message) throws Exception {
String exchange = applicationConfiguration.getAMQPExchange();
String exchangeType = applicationConfiguration.getAMQPExchangeType();
Connection connection = myConnection.getConnection();
Channel channel = connection.createChannel();
channel.exchangeDeclare(exchange, exchangeType);
channel.basicPublish(exchange, routingKey, null, message.getBytes());
log.debug(" [CORE: AMQP] Sent message with key {} : {}",routingKey, message);
}
If the Rabbit MQ server fails (crashes, runs out of RAM, turned off etc) the code above blocks, preventing the upstream service from receiving messages (a bad thing). I am looking for a way of preventing this behaviour whilst not losing mesages so that at some time in the future they can be resent.
I am not sure how best to address this. One option may be to queue the messages to a disk file and then use a separate thread to read and forward to the Rabbit MQ server?

If I understand correctly, the issue you are describing is a known JDK socket behaviour when the connection is lost mid-write. See this mailing list thread: http://markmail.org/thread/3vw6qshxsmu7fv6n.
Note that if RabbitMQ is shut down, the TCP connection should be closed in a way that's quickly observable by the client. However, it is true that stale TCP connections can take
a while to be detected, that's why RabbitMQ's core protocol has heartbeats. Set heartbeat
interval to a low value (say, 6-8) and the client itself will notice unresponsive peer
in that amount of time.
You need to use Publisher confirms [1] but also account for the fact that the app itself
can go down right before sending a message. As you rightly point out, having a disk-based
WAL (write-ahead log) is a common solution for this problem. Note that it is both quite
tricky to get right and still leaves some time window where your app process shutting down can result in an unpublished and unlogged message.
No promises on the time frame but the idea of adding WAL to the Java client has been discussed.
http://www.rabbitmq.com/confirms.html

Apache Mina Idle Monitor

I have been developing my first TCP/Socket based application with Apache Mina, it looks great and easy to do things. I just want to ask a question here about Mina.
The server impose an idle time of 5 second will terminate the socket connection, so we have to send periodic heartbeat (echo message / keepalive) to make sure connection is alive. Sort of keepalive mechanism.
There's one way that we send blindly echo/heartbeat message just before every 5 seconds. I am thinking, there should be smart/intelligent way "Idle Monitor" if I am sending my business message and do not come to idle time i.e. 5 second, I should not issue heartbeat message. Heartbeat message will be sent if whole connection is idle, so that we save bandwidth and fast reading & writing on socket.

You can achieve it by using Keep Alive Filter (already present in mina).
Alternatively, you can achieve a smarter way of sending echo/heart beat by setting session idle timeout of client a bit smaller than idle timeout of server. For example:
For server side
NioSocketAcceptor.getSessionConfig().setIdleTime(IdleStatus.BOTH_IDLE, 5);
and for client side it would be
NioSocketConnector.getSessionConfig().setIdleTime(IdleStatus.BOTH_IDLE, 3);
Now, if there is no communication for lets say 3 seconds, a sessionIdle will be triggred at the client side ( and it will not be triggered at server side as timeout there is 5 seconds) and you can send an echo. This will keep the session alive. The echo will be sent only if the session is idle.
Note: I am assuming that at session idle, session is being closed at the server side. If it is other way around you will need to switch values of session idle timeout(e.g. 3 seconds for server and 5 seconds for client) and echo will be sent from server.

(I hope I'm understanding the question correctly)
I was having trouble keeping my session alive and this question came up on Google search results so I'm hoping someone else will find it useful:
#Test
public void testClientWithHeartBeat() throws Exception {
SshClient client = SshClient.setUpDefaultClient();
client.getProperties().put(ClientFactoryManager.HEARTBEAT_INTERVAL, "500");
client.start();
ClientSession session = client.connect("localhost", port).await().getSession();
session.authPassword("smx", "smx").await().isSuccess();
ClientChannel channel = session.createChannel(ClientChannel.CHANNEL_SHELL);
int state = channel.waitFor(ClientChannel.CLOSED, 2000);
assertTrue((state & ClientChannel.CLOSED) == 0);
channel.close(false);
client.stop();
}
(Source: https://issues.apache.org/jira/browse/SSHD-185)
In newer versions (e.g. version 2.8.0), enabling heartbeats changed to CoreModuleProperties.HEARTBEAT_INTERVAL.set(client, Duration.ofMillis(500));

I'm not sure I totally understand your question, but you can send a heartbeat in an overridden sessionIdle method of the IoHandlerAdapter. You don't need to necessarily close a session just because Mina on the server side calls Idle. As far as a more intelligent way of maintaining an active connection between and Server and Client without this type of heartbeat communication I have never heard of one.
Here is an interesting read of how microsoft handles their heartbeat in ActiveSync. I personally used this methodology when using mina in my client/server application. Hope this helps you some.