Develop Reference

Sending json to logstash over tcp, keeping connection open, never show up

The code below runs in a thread, takes messages off of a queue, and sends them to logstash. I've a unit test that proves that the json (which is the content of the messages) makes it into the socket and gets to the other end.
However, when I point it at actual logstash, nothing ever turns up. I can 'nc' the same content and it shows up. Of course, nc will close the connection at the end of sending the one message. There's a bit of complexity here to reconnect if we ever lose the connection.
The logstash config is pretty simple:
input {
tcp {
port => 9191
codec => json
}
}
output {
stdout {
codec => rubydebug
}
}
while (true) {
byte[] message = messages.take();
executor.getWithRetry(() -> {
try {
if (socket == null) {
socket = new Socket(sinkAddress, port);
}
socket.getOutputStream().write(message);
return null;
} catch (IOException e) {
socket.close();
socket = null;
throw e;
}
}).join();
}

Make sure you are sending a newline character at the end of the message. I found that sending "\n" works fine even from a windows box to logstash running on linux.

Java message is sent to a closed socket

I have a client program that sends messages typed in console to the server. Following some advices, I introduced a check for a closed socket with Socket.checkError(). Nevertheless, for some reason it indicates error only after second failed attempt to send a message.
My code:
BufferedReader stdIn = new BufferedReader(new InputStreamReader(System.in));
while (true)
try (
Socket clientSocket = new Socket(hostname, port);
PrintWriter socketOut = new PrintWriter(clientSocket.getOutputStream(), true);
) {
String input;
while ((input=stdIn.readLine())!=null) {
socketOut.println(input);
if (socketOut.checkError()) {
System.out.println("Got socket error");
break;
}
}
} catch (IOException e) {
try {
Thread.sleep(5000);
} catch (InterruptedException e1) {}
}
I shut down (manually) my server side after receiving 'message1'. Therefore, I expect to get the error while trying to send the very next message. Nevertheless it occurs only one message after:
message1
message2
message3
Got socket error
Can anyone explain this behavior and advise me a method to get notification right on the first attempt to send a message in void?

Following some advices, I introduced a check for a closed socket with Socket.checkError().
There is no such method. Clearly you are referring to PrintWriter.checkError().
Nevertheless, for some reason it indicates error only after second failed attempt to send a message.
The reason is that there is both a socket send buffer at the sender and a socket receive buffer at the receiver, and that sending is asynchronous: it therefore isn't possible for the first send to detect an error.
Can anyone explain this behavior and advise me a method to get notification right on the first attempt to send a message in void?
There isn't one. That's the nature of TCP. What you are attempting indicates an application protocol error, and the answer lies in the realm of the application protocol as well: don't have the peer close the socket while this end could still be sending data, OR don't allow this end to send data after the peer has indicated, via the application protocol, that it won't be reading any more data.
Don't use PrintWriter over the network. It suppresses the actual exception. Use BufferedWriter and the write() and newLine() methods.

In ths java library there is no method to check if connection is opened or not. Method like isConnected() and isClosed() check only one side of the connection (where you invoked the method).
From javadoc:
Note: Closing a socket doesn't clear its connection state, which means
this method will return true for a closed socket (see isClosed()) if
it was successfuly connected prior to being closed.
To check if the connection has been really closed simply invoke the read() method (or equivalent) and check if it returns -1.
Note: also if isConnected will work as you like (giving false if the other side of the socket closed the connection or if there is a network problem or similar) the sequence:
if (socket.isConnected()) {
int x = socked.read();
}
will not grant that the x has a value different from -1 or throws an IOException, because the connection could be closed after the isConnected test and before the read operation.
The following code to show how any kind of check on the socket cannot guarantee that a subsequent read will give a valid result.
// Return true because socket communication is enabled
if (myFunctionToCheckIfSocketIsOpen(socket)) {
// Here the peer closed the socket or the network shutdown
// This read will give -1 or throws IOException also if the previous check returned true
int x = socket.read();
}

From the answer of #Davide Lorenzo MARINO I got the idea of employing read(). The only problem that it is blocking. However, one can always run it in another thread, which would modify a class global variable, when read() finally returns -1:
static boolean socketIsAlive;
...
BufferedReader stdIn = new BufferedReader(new InputStreamReader(System.in));
while (true)
try (
Socket clientSocket = new Socket(hostname, port);
PrintWriter socketOut = new PrintWriter(clientSocket.getOutputStream(), true);
) {
socketIsAlive=true;
new ConnectionChecker(clientSocket).start();
String input;
while (true) {
if ((input=stdIn.readLine())!=null)
socketOut.println(input);
if (!socketIsAlive) {
System.out.println("Got socket error");
break;
}
}
} catch (IOException e) {
try {
Thread.sleep(5000);
} catch (InterruptedException e1) {}
}
}
...
static public class ConnectionChecker extends Thread{
Socket socket;
public ConnectionChecker(Socket socket) {
this.socket=socket;
}
#Override
public void run() {
try {
if (socket.getInputStream().read()==-1)
socketIsAlive=false;
} catch (IOException e) {}
}
}

Permanent and persistent Socket connection in java

I've created a client-server connection, something like a chat system. Previously I was using a while loop on the client side, and it was waiting to read a message from the console every time (of course server has a while loop as well to serve forever). But now, I'm trying to first create a connection at the beginning of the session, and then occasionally send a message during the session, so to maintain a permanent and persistent connection.
Currently, without the while loop, the client closes the connection and I don't know how to find a workaround.
Here is the client code:
import java.net.*;
import java.io.*;
public class ControlClientTest {
private Socket socket = null;
// private BufferedReader console = null;
private DataOutputStream streamOut = null;
public static void main(String args[]) throws InterruptedException {
ControlClientTest client = null;
String IP="127.0.0.1";
client = new ControlClientTest(IP, 5555);
}
public ControlClientTest(String serverName, int serverPort) throws InterruptedException {
System.out.println("Establishing connection. Please wait ...");
try {
socket = new Socket(serverName, serverPort);
System.out.println("Connected: " + socket);
start();
} catch (UnknownHostException uhe) {
System.out.println("Host unknown: " + uhe.getMessage());
} catch (IOException ioe) {
System.out.println("Unexpected exception: " + ioe.getMessage());
}
String line = "";
// while (!line.equals(".bye")) {
try {
Thread.sleep(1000);
//TODO get data from input
// line = console.readLine();
line="1";
if(line.equals("1"))
line="1,123";
streamOut.writeUTF(line);
streamOut.flush();
} catch (IOException ioe) {
System.out.println("Sending error: " + ioe.getMessage());
}
// }
}
public void start() throws IOException {
// console = new BufferedReader(new InputStreamReader(System.in));
streamOut = new DataOutputStream(socket.getOutputStream());
}
}
And here is the Server code:
import java.awt.*;
import java.io.*;
import java.net.ServerSocket;
import java.net.Socket;
public class ControlServer {
private Socket socket = null;
private ServerSocket server = null;
private DataInputStream streamIn = null;
public static void main(String args[]) {
ControlServer server = null;
server = new ControlServer(5555);
}
public ControlServer(int port) {
try {
System.out
.println("Binding to port " + port + ", please wait ...");
server = new ServerSocket(port);
System.out.println("Server started: " + server);
System.out.println("Waiting for a client ...");
socket = server.accept();
System.out.println("Client accepted: " + socket);
open();
boolean done = false;
while (!done) {
try {
String line = streamIn.readUTF();
// TODO get the data and do something
System.out.println(line);
done = line.equals(".bye");
} catch (IOException ioe) {
done = true;
}
}
close();
} catch (IOException ioe) {
System.out.println(ioe);
}
}
public void open() throws IOException {
streamIn = new DataInputStream(new BufferedInputStream(
socket.getInputStream()));
}
public void close() throws IOException {
if (socket != null)
socket.close();
if (streamIn != null)
streamIn.close();
}
}

I would like to summarize some good practices regarding the stability of TCP/IP connections which I apply on a daily basis.
Good practice 1 : Built-in Keep-Alive
socket.setKeepAlive(true);
It automatically sends a signal after a period of inactivity and checks for a reply. The keep-alive interval is operating system dependent though, and has some shortcomings. But all by all, it could improve the stability of your connection.
Good practice 2 : SoTimeout
Whenver you perform a read (or readUTF in your case), your thread will actually block forever. In my experience this is bad practice for the following reasons: It's difficult to close your application. Just calling socket.close() is dirty.
A clean solution, is a simple read time-out (e.g. 200ms). You can do this with the setSoTimeoutmethod. When the read() method timeouts it will throw a SocketTimeoutException. (which is a subclass of IOException).
socket.setSoTimeout(timeoutInterval);
Here is an example to implement the loop. Please note the shutdown condition. Just set it to true, and your thread will die peacefully.
while (!shutdown)
{
try
{
// some method that calls your read and parses the message.
code = readData();
if (code == null) continue;
}
catch (SocketTimeoutException ste)
{
// A SocketTimeoutExc. is a simple read timeout, just ignore it.
// other IOExceptions will not be stopped here.
}
}
Good practice 3 : Tcp No-Delay
Use the following setting when you are often interfacing small commands that need to be handled quickly.
try
{
socket.setTcpNoDelay(true);
}
catch (SocketException e)
{
}
Good practice 4 : A heartbeat
Actually there are a lot of side scenario's that are not covered yet.
One of them for example are server applications that are designed to only communicate with 1 client at a time. Sometimes they accept connections and even accept messages, but never reply to them.
Another one: sometimes when you lose your connection it actually can take a long time before your OS notices this. Possibly due to the shortcomings described in good practice 3, but also in more complex network situations (e.g. using RS232-To-Ethernet converters, VMware servers, etc) this happens often.
The solution here is to create a thread that sends a message every x seconds and then waits for a reply. (e.g. every 15 seconds). For this you need to create a second thread that just sends a message every 15 seconds. Secondly, you need to expand the code of good practice 2 a little bit.
try
{
code = readData();
if (code == null) continue;
lastRead = System.currentTimeMillis();
// whenever you receive the heart beat reply, just ignore it.
if (MSG_HEARTBEAT.equals(code)) continue;
// todo: handle other messages
}
catch (SocketTimeoutException ste)
{
// in a typical situation the soTimeout is about 200ms
// the heartbeat interval is usually a couple of seconds.
// and the heartbeat timeout interval a couple of seconds more.
if ((heartbeatTimeoutInterval > 0) &&
((System.currentTimeMillis() - lastRead) > heartbeatTimeoutInterval))
{
// no reply to heartbeat received.
// end the loop and perform a reconnect.
break;
}
}
You need to decide if your client or server should send the message. That decision is not so important. But e.g. if your client sends the message, then your client will need an additional thread to send the message. Your server should send a reply when it receives the message. When your client receives the answer, it should just continue (i.e. see code above). And both parties should check: "how long has it been?" in a very similar way.

You could wrap a thread around the connection and have it periodically send a status to keep the line open, say every 30 seconds or whatever. Then, when it actually has data to send it would reset the keep alive to be 30 seconds after the last transmission. The status could be helpful to see if the client is still alive anyway, so at least it can be a useful ping.
Also, you should change your server code, you appear to only handle one connection at the moment. You should loop and when a socket connection comes in spawn a thread to handle the client request and go back to listening. I may be reading to much into what may just be your test code, though.

Make the client socket connection wrapped around a thread. Use a blocking queue to wait for messages. There should only be a single sender queue throughout your application, so use a singleton pattern.
e.g.
QueueSingleton queue = QueueSingleton.getSenderQueue();
Message message = queue.take() // blocks thread
send(message); //send message to server
When you need to send a message to the server, you can use the blocking queue to send the message.
QueueSingleton queue = QueueSingleton.getSenderQueue();
queue.put(message)
The client thread will wake up and process the message.
For maintaining the connection, use a timer task. This is special type of thread that calls a run method repetitively at specified periods. You can use this to post a message, a ping message, every so often.
For processing the received message, you could have another thread, waiting for messages on another blocking queue (receiver queue). The client thread will put the received message on this queue.

Async reading in server-client: separate client messages

I'm implementing a simple server using AsynchronousServerSocketChannel. For testing purposes, I created a tiny client prototype that sends two messages, "hi" and "stackoverflow", then disconnects. On server side, I read the arrived messages and print them to standard output. When the client executed, I'm expecting to receive:
message [hi], bytecount 2
message [stackoverflow], bytecount 13
The problem is, that sometimes both messages already arrived when server invokes reading callback so I get
message [histackoverflow], bytecount 15
instead.
The question is, if it is possible to ensure on server side that the messages arrive separately and if yes, how to do it?
Here's my CompletionHandler prototype that handles client connections:
class CommunicationHandler implements CompletionHandler<AsynchronousSocketChannel, Void> {
private final AsynchronousServerSocketChannel server;
public CommunicationHandler(final AsynchronousServerSocketChannel server) {
this.server = server;
}
#Override
public void failed(Throwable ex, Void attachment) {}
#Override
public void completed(final AsynchronousSocketChannel client, Void attachment) {
// handle client messages
final ByteBuffer buffer = ByteBuffer.allocateDirect(Server.BUFFER_SIZE);
final Session session = new Session();
try {
client.read(buffer, session, new CompletionHandler<Integer, Session>() {
#Override
public void completed(Integer byteCount, final Session currSession) {
if (byteCount == -1) {
return;
}
buffer.flip();
// TODO forward buffer to message handler (probably protocol?)
System.out.println("message [" + convertToString(buffer) + "], byteCount " + byteCount);
buffer.clear();
// read next message
client.read(buffer, currSession, this);
}
#Override
public void failed(Throwable ex, final Session currSession) {}
});
}
// accept the next connection
server.accept(null, this);
}
ByteBuffer to String conversion:
public static String convertToString(ByteBuffer bb) {
final byte[] bytes = new byte[bb.remaining()];
bb.duplicate().get(bytes);
return new String(bytes);
}
Here is a test client prototype:
public class Client {
public final void start() {
try (AsynchronousSocketChannel client = AsynchronousSocketChannel.open();) {
Future<Void> connCall = client.connect(InetAddress.getByName("127.0.0.1"), 8060));
connCall.get();
// client is now connected
// send greeting message
Future<Integer> writeCall = client.write(Charset.forName("utf-8").encode(CharBuffer.wrap("hi")));
writeCall.get();
// Thread.sleep(5000L);
writeCall = client.write(Charset.forName("utf-8").encode(CharBuffer.wrap("stackoverflow")));
writeCall.get();
client.close();
} catch (IOException e) {
} catch (InterruptedException ex) {
} catch (ExecutionException ex) {
}
}

In addition to the possibility of getting two (or even more) writes in one read, for larger messages (usually about 3k or more) you can get one write split over several reads. TCP is a stream protocol and does not preserve record boundaries, unless by chance: What is a message boundary? There are two solutions that work in general, although with async channel I think you'll need to do your own buffer management which may be confusing and hard to test:
add an explicit length field before each record
add a delimiter after each record when there is a byte not otherwise used, or an escape can be used to distinguish data from the delimiter
and several others that have been tried:
as your comment suggests, wait long enough that the first request has always been read before the second is sent. On the local networks and test systems used by developers this usually is a few milliseconds or even less; on the real Internet it is fairly often several seconds, sometimes minutes, and in theory can be hours or even days.
if records are never longer than a few fragments (maybe 10k or so) use UDP (available in Java as DatagramSocket but not as a NIO channel AFAICS) and implement your own protocols to handle message loss, duplication and reordering (which is hard to do and often ends up failing in some obscure cases that were discovered and avoided or fixed in TCP 30 years ago)
use SCTP (not available in Java at all AFAICS, and not too many other systems either)
Aside: your test client sends data in UTF-8, but new String (byte[]) uses the default encoding which is platform-dependent and not necessarily UTF-8. I'm not sure it's guaranteed but in practice all usable encodings include ASCII as a subset, and your example data is ASCII. But if you want to support actual UTF-8 data code for it.

Java Process "The pipe has been ended" problem

I am using Java Process API to write a class that receives binary input from the network (say via TCP port A), processes it and writes binary output to the network (say via TCP port B). I am using Windows XP. The code looks like this. There are two functions called run() and receive(): run is called once at the start, while receive is called whenever there is a new input received via the network. Run and receive are called from different threads.
The run process starts an exe and receives the input and output stream of the exe. Run also starts a new thread to write output from the exe on to the port B.
public void run() {
try {
Process prc = // some exe is `start`ed using ProcessBuilder
OutputStream procStdIn = new BufferedOutputStream(prc.getOutputStream());
InputStream procStdOut = new BufferedInputStream(prc.getInputStream());
Thread t = new Thread(new ProcStdOutputToPort(procStdOut));
t.start();
prc.waitFor();
t.join();
procStdIn.close();
procStdOut.close();
} catch (Exception e) {
e.printStackTrace();
printError("Error : " + e.getMessage());
}
}
The receive forwards the received input from the port A to the exe.
public void receive(byte[] b) throws Exception {
procStdIn.write(b);
}
class ProcStdOutputToPort implements Runnable {
private BufferedInputStream bis;
public ProcStdOutputToPort(BufferedInputStream bis) {
this.bis = bis;
}
public void run() {
try {
int bytesRead;
int bufLen = 1024;
byte[] buffer = new byte[bufLen];
while ((bytesRead = bis.read(buffer)) != -1) {
// write output to the network
}
} catch (IOException ex) {
Logger.getLogger().log(Level.SEVERE, null, ex);
}
}
}
The problem is that I am getting the following stack inside receive() and the prc.waitfor() returns immediately afterwards. The line number shows that the stack is while writing to the exe.
The pipe has been ended
java.io.IOException: The pipe has been ended
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
at java.io.FilterOutputStream.write(FilterOutputStream.java:80)
at xxx.receive(xxx.java:86)
Any advice about this will be appreciated.

This means you are writing to the pipe after the other end has already closed it.
That indicates a major error in your application protocol.

I have had the same problem recently and I have found a solution.
First of all, "The pipe has been ended" error is not a Java error - it comes from Windows system. According to MSDN:
The using process has closed the pipe or, if you are trying to write
to the pipe, there are no available readers.
Not very informative. However, if process has closed the pipe itself, it may mean that some errors occurred in process.
To check this, redirect errors coming from process, for instance, to a file:
File f = new File("errors.txt");
pb.redirectError(f);
In my case (I've been trying to execute SrcML parser) file contained this:
.\libs\srcML-Win\src2srcml.exe: unrecognised option `--language Java'
Try 'src2srcml --help' for more information.
Fixing this solved the problem.