I'm new to Scala so the question may be quite simple, though I have spent some time trying to resolve it. I have a simple Scala TCP server (no actors, single thread):
import java.io._
import java.net._
object Application {
def readSocket(socket: Socket): String = {
val bufferedReader = new BufferedReader(new InputStreamReader(socket.getInputStream))
var request = ""
var line = ""
do {
line = bufferedReader.readLine()
if (line == null) {
println("Stream terminated")
return request
}
request += line + "\n"
} while (line != "")
request
}
def writeSocket(socket: Socket, string: String) {
val out: PrintWriter = new PrintWriter(new OutputStreamWriter(socket.getOutputStream))
out.println(string)
out.flush()
}
def main(args: Array[String]) {
val port = 8000
val serverSocket = new ServerSocket(port)
while (true) {
val socket = serverSocket.accept()
readSocket(socket)
writeSocket(socket, "HTTP/1.1 200 OK\r\n\r\nOK")
socket.close()
}
}
}
The server listens on localhost:8000 for incomming requests and sends HTTP response with single OK word in the body. Then I run Apache Benchmark like this:
ab -c 1000 -n 10000 http://localhost:8000/
which works nicely for the first time. The second time I start ab it hangs producing the following output in netstat -a | grep 8000:
....
tcp 0 0 localhost.localdo:43709 localhost.localdom:8000 FIN_WAIT2
tcp 0 0 localhost.localdo:43711 localhost.localdom:8000 FIN_WAIT2
tcp 0 0 localhost.localdo:43717 localhost.localdom:8000 FIN_WAIT2
tcp 0 0 localhost.localdo:43777 localhost.localdom:8000 FIN_WAIT2
tcp 0 0 localhost.localdo:43722 localhost.localdom:8000 FIN_WAIT2
tcp 0 0 localhost.localdo:43725 localhost.localdom:8000 FIN_WAIT2
tcp6 0 0 [::]:8000 [::]:* LISTEN
tcp6 83 0 localhost.localdom:8000 localhost.localdo:43724 CLOSE_WAIT
tcp6 83 0 localhost.localdom:8000 localhost.localdo:43786 CLOSE_WAIT
tcp6 1 0 localhost.localdom:8000 localhost.localdo:43679 CLOSE_WAIT
tcp6 83 0 localhost.localdom:8000 localhost.localdo:43735 CLOSE_WAIT
tcp6 83 0 localhost.localdom:8000 localhost.localdo:43757 CLOSE_WAIT
tcp6 83 0 localhost.localdom:8000 localhost.localdo:43754 CLOSE_WAIT
tcp6 83 0 localhost.localdom:8000 localhost.localdo:43723 CLOSE_WAIT
....
Since that no more requests are served by the server. One more detail: The same ab script with the same parameters works smoothly testing a simple Node.js server on the same machine. So this issue is not related to a number of opened TCP connections which I have set to be reusable with
sudo sysctl -w net.ipv4.tcp_tw_recycle=1
sudo sysctl -w net.ipv4.tcp_tw_reuse=1
Could anyone give me a clue on what I'm missing?
Edit: Termination of stream handling has been added to the code above:
if (line == null) {
println("Stream terminated")
return request
}
I'm posting the (partial) answer to my own question for those who will stumble upon the same issue one day. First, the nature of the problem lies not in the source code but in the system itself which restricts numerious connections. The problem is that the socket passed to readSocket function appears corrupted under some conditions, i.e. it can not be read and bufferedReader.readLine() either returns null on first call or hangs indefinitely. The following two steps make the code working on some machines:
Increase the number of concurrent connections to a socket with
sysctl -w net.core.somaxconn=65535
Provide the second parameter to ServerSocket constructor which will explicitly set the length of connection queue:
val maxQueue = 50000
val serverSocket = new ServerSocket(port, maxQueue)
The steps above solve the problem on EC2 m1.large instances, however I'm still getting issues on my local machine. The better way would be to use Akka for the stuff of that kind:
import akka.actor._
import java.net.InetSocketAddress
import akka.util.ByteString
class TCPServer(port: Int) extends Actor {
override def preStart {
IOManager(context.system).listen(new InetSocketAddress(port))
}
def receive = {
case IO.NewClient(server) =>
server.accept()
case IO.Read(rHandle, bytes) => {
val byteString = ByteString("HTTP/1.1 200 OK\r\n\r\nOK")
rHandle.asSocket.write(byteString)
rHandle.close()
}
}
}
object Application {
def main(args: Array[String]) {
val port = 8000
ActorSystem().actorOf(Props(new TCPServer(port)))
}
}
First, I'd suggest trying this without ab. You can do something like:
echo "I'm\nHappy\n" | nc -vv localhost 8000
Second, I'd suggest handling end-of-stream. This is where BufferedReader.readLine() returns null. The code above only checks for an empty String. After you fix this, I'd try again. Then test with ab, after everything looks good. Let us know if the problem persists.
Related
In my Java 8 application (RHEL 6.x, Wildfly 10.1.0.Final) the first time a user prints a document, application gets stuck while getting the list of printers from the system.
Here is the stacktrace of the blocking thread :
"Thread-211" #799 daemon prio=5 os_prio=0 tid=0x00007fca543a6800 nid=0x10755 runnable [0x00007fca02820000]
java.lang.Thread.State: RUNNABLE
at sun.print.CUPSPrinter.canConnect(Native Method)
at sun.print.CUPSPrinter.isCupsRunning(CUPSPrinter.java:444)
at sun.print.UnixPrintServiceLookup.getDefaultPrintService(UnixPrintServiceLookup.java:650)
- locked <0x00000006d2c7fff8> (a sun.print.UnixPrintServiceLookup)
at sun.print.UnixPrintServiceLookup.refreshServices(UnixPrintServiceLookup.java:277)
- locked <0x00000006d2c7fff8> (a sun.print.UnixPrintServiceLookup)
at sun.print.UnixPrintServiceLookup$PrinterChangeListener.run(UnixPrintServiceLookup.java:947)
Other users trying to print documents and relatives threads are blocked by this one.
I looked at the source code of CUPSPrinter.canConnect() (native code) and at this point we try to connect to the cups server :
/*
* Checks if connection can be made to the server.
*
*/
JNIEXPORT jboolean JNICALL
Java_sun_print_CUPSPrinter_canConnect(JNIEnv *env,
jobject printObj,
jstring server,
jint port)
{
const char *serverName;
serverName = (*env)->GetStringUTFChars(env, server, NULL);
if (serverName != NULL) {
http_t *http = j2d_httpConnect(serverName, (int)port);
(*env)->ReleaseStringUTFChars(env, server, serverName);
if (http != NULL) {
j2d_httpClose(http);
return JNI_TRUE;
}
}
return JNI_FALSE;
}
In my case CUPS is on the same host listening on port 631.
I checked the logs & everything seems to be fine.
I also checked active connections for cups with netstat :
tcp 0 0 0.0.0.0:631 0.0.0.0:* LISTEN 76107/cupsd
tcp 0 0 127.0.0.1:45652 127.0.0.1:631 TIME_WAIT -
tcp 0 0 :::631 :::* LISTEN 76107/cupsd
tcp 0 0 ::1:35982 ::1:631 TIME_WAIT -
tcp 0 0 ::1:35981 ::1:631 TIME_WAIT -
tcp 0 0 ::1:35978 ::1:631 TIME_WAIT -
tcp 0 0 ::1:35979 ::1:631 TIME_WAIT -
udp 0 0 0.0.0.0:631 0.0.0.0:* 76107/cupsd
Important notes :
If I restart Cups service, thread is not deblocked. It seems to live endessly until application restarts.
I found a bug similar to this on Open JDK : https://bugs.openjdk.java.net/browse/JDK-6290446 But the workaround of setting -Dsun.java2d.print.polling=false does not work for me (the property seems to be cleared at some point for an obscure reason, so PrinterChangeListener gets instantiated and though polling is not desactivated).
I can't reproduce the problem with a test application (clone of production) on the same server
Please HELP !!
-Djava.awt.printerjob=sun.print.PSPrinterJob
I have the following code to start a netty server:
Application application = ApplicationTypeFactory.getApplication(type);
resteasyDeployment = new ResteasyDeployment();
// Explicitly setting the Application should prevent scanning
resteasyDeployment.setApplication(application);
// Need to set the provider factory to the default, otherwise the
// providers we need won't be registered, such as JSON mapping
resteasyDeployment.setProviderFactory(ResteasyProviderFactory.getInstance());
netty = new NettyJaxrsServer();
netty.setHostname(HOST);
netty.setPort(port);
netty.setDeployment(resteasyDeployment);
// Some optional extra configuration
netty.setKeepAlive(true);
netty.setRootResourcePath("/");
netty.setSecurityDomain(null);
netty.setIoWorkerCount(16);
netty.setExecutorThreadCount(16);
LOGGER.info("Starting REST server on: " + System.getenv("HOSTNAME") + ", port:" + port);
// Start the server
//("Starting REST server on " + System.getenv("HOSTNAME"));
netty.start();
LOGGER.info("Started!");
When I do:
netty.stop()
It doesn't appear to close the connection:
[john#dub-001948-VM01:~/workspace/utf/atf/src/test/resources ]$ netstat -anp | grep 8888
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 ::ffff:10.0.21.88:8888 ::ffff:10.0.21.88:60654 TIME_WAIT -
tcp 0 0 ::ffff:10.0.21.88:8888 ::ffff:10.0.21.88:60630 TIME_WAIT -
tcp 0 0 ::ffff:10.0.21.88:8888 ::ffff:10.0.21.88:60629 TIME_WAIT -
tcp 0 0 ::ffff:10.0.21.88:8888 ::ffff:10.0.21.88:60637 TIME_WAIT -
tcp 0 0 ::ffff:10.0.21.88:8888 ::ffff:10.0.21.88:60640 TIME_WAIT -
even after the program exits. In other posts I have read that netty does not close client connections on a stop. How do I shut it down cleanly?
Time_Wait is one of socket state. application can not do anything about it. more information here: http://www.isi.edu/touch/pubs/infocomm99/infocomm99-web/. For a busy server, you could tune some tcp/ip parameter to alleviate its impact.
I am use Netty ( a java network framework) to server UDP request .
I found the
SO_RCVBUF
and
/proc/sys/net/core/rmem_default
and
/proc/net/udp
output confusing:
By the manual of socket( man 7 socket) It explains :
rmem_default
contains the default setting in bytes of the socket receive
buffer.
rmem_max
contains the maximum socket receive buffer size in bytes which
a user may set by using the SO_RCVBUF socket option.
and the /proc/net/udp show's the udp receive queue size :
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode
60: 00000000:2F3C 00000000:0000 07 00000000:0003E094 00:00000000 00000000 500 0 2224362 2 ffff810066908cc0
I can get the SO_RCVBUF = 12904, but the rx_queue size is 254100(3E094), and rmem_default is 262142
my confuse is, why the SO_RCVBUF not equals to the rmem_default, and why the queue size not equals the SO_RCVBUF (but larger than it ) ?
It seams the ctx.channel().config().getOption(ChannelOption.SO_RCVBUF)'s value is half of /proc/sys/net/core/rmem_default and the real receive buffer size( queue size ) is also the value of /proc/sys/net/core/rmem_default
so why ctx.channel().config().getOption(ChannelOption.SO_RCVBUF) not equals to the system's config ?
The accepted answer is wrong. The correct alignment is:
tx_queue rx_queue
00000000:0003E094
So rx_queue is 0x0003E094 or 254100. This can be confirmed using netstat -unlp | grep pid , Recv-Q is second column. Example output:
cat /proc/net/udp
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops
8101: 41FD080A:06A5 00000000:0000 07 00000000:00005A00 00:00000000 00000000 1000 0 3892411 2 0000000000000000 0
netstat --udp -nl
Proto Recv-Q Send-Q Local Address Foreign Address State
udp 23040 0 10.8.253.65:1701 0.0.0.0:*
Here, 0x00005A00==23040
You seem to have misread the output. When you line up the columns properly, rx_queue is zero.
I have been experiencing this issue for a while now. A quick explanation of the system:
A simple application will read in data over a tcp connection. The application uses Socketchannel object to establish a data communication line. The connection to hardware is established, and the application processes roughly between 400 - 700 packets before throwing the error:
at sun.nio.ch.SocketDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)
at sun.nio.ch.IOUtil.read(IOUtil.java:186)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
I suspect it has to do with the operating system (windows 7) buffer, although I have tried a few work arounds (increasing JVM heap size, and creating additional registry variables for TCP settings), but with no luck.
Inspecting packet data using wireshark, the following information can been seen just as the error occurs
[TCP Window Full] 5001 > 49995 [PSH, ACK] Seq=123177 Ack=1 Win=200 Len=176
[TCP ZeroWindow] 49995 > 5001 [ACK] Seq=1 Ack=123353 Win=0 Len=0
Which is then followed by a series of
[TCP Dup ACK 144#1] 5001 > 49995 [ACK] Seq=123353 Ack=1 Win=200 Len=0
until the connection is terminated.
I am hoping perhaps someone has had a similar issue in the past,or could offer some guidance on something I may be overlooking.
Here is sample code where data is read, and the error is thrown.
try{
do{
socketChannel.read(syncBuf);
start = syncBuf.get(0);
int b = start;
System.out.println("start byte is " + b);
syncBuf.clear();
}while(start != INIT_BYTE);
}catch(Exception e){
e.printStackTrace();
}
packetBuf.order(ByteOrder.LITTLE_ENDIAN);
packetBuf.clear();
try{
while(socketChannel.read(packetBuf) != 206){
nrBytesRead += socketChannel.read(packetBuf);
}
}catch(Exception e){
ApplicationLogger.log("Error at INIT_BYTE loop ", "ERROR");
}
Many thanks.
I think you have a problem here:
while(socketChannel.read(packetBuf) != 206)
{
nrBytesRead += socketChannel.read(packetBuf);
}
socketChannel.read(packetBuf) will return the number read this time, not the total, and I guess that's what 206 is, the total read.
You probably want to do that:
do
{
nrBytesRead += socketChannel.read(packetBuf);
}
while (nrBytesRead != 206)
I have a java app program run on centos 6.3 and tomcat 7 as the app container, currently I meet one error : java.io.socketexception Maximum number of datagram sockets reached
we use MulticastSocket class to send message. when this error happened, I check the current server UDP socket count with command: ss -s
Total: 212 (kernel 248)
TCP: 70 (estab 15, closed 44, orphaned 0, synrecv 0, timewait 40/0), ports 22
Transport Total IP IPv6
* 248 - -
RAW 0 0 0
UDP 40 40 0
TCP 26 26 0
INET 66 66 0
FRAG 0 0 0
and I also check the
ulimits -n
The default setting is 32768, seem UDP socket count not exceed max count.
Any ideas for this error?
we use MulticastSocket class to send message.
Why? You only need a MulticastSocket to receive multicasts.
Obviously you are leaking MulticastSockets. Presumably you are creating a new one per message and never closing it.