Develop Reference

Play Framework Stop responding after 2-3 days

I am using play framework and it stops responding after 2-3 days and when I restart server then everything works fine.
Please let me know what I am doing wrong.
Thanks
Stack trace:
Caused by: io.netty.channel.ChannelException: Failed to open a socket.
at io.netty.channel.socket.nio.NioSocketChannel.newSocket(NioSocketChannel.java:62)
at io.netty.channel.socket.nio.NioSocketChannel.<init>(NioSocketChannel.java:72)
at sun.reflect.GeneratedConstructorAccessor42.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at io.netty.bootstrap.AbstractBootstrap$BootstrapChannelFactory.newChannel(AbstractBootstrap.java:454)
... 64 common frames omitted
Caused by: java.net.SocketException: Too many open files
at sun.nio.ch.Net.socket0(Native Method)
at sun.nio.ch.Net.socket(Net.java:411)
at sun.nio.ch.Net.socket(Net.java:404)
at sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:105)
at sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60)
at io.netty.channel.socket.nio.NioSocketChannel.newSocket(NioSocketChannel.java:60)
... 70 common frames omitted

Looks like you are hitting the ulimit for your user. This is likely a function of some or all of the following:
Your user having the default ulimit (probably 256 or 1024 depending on the OS)
The amount/type of activity on your Play application
You can identify what open file handles your Play application by:
Running lsof -p PID on a *nix OS
Running something like Filemon (from sysinternals) on Windows
You'll likely see everything on your Play applications' classpath plus any files which your application opens e.g. log files, configuration files. In addition, if you are running on a *nix OS then open sockets will also consume file handles so you might see open file handles relating to database connection pools or indeed anything which your Play application communicates with via sockets.
Once you understand what your application is doing w.r.t open file handles you can consider what to do next. Likely to be one of:
Change your application such that it opens fewer file handles (and, if on *nix, uses fewer socket connections)
Change your application such that it closes open file handles when finished with them
Increase the number of open files allowed for your user by invoking ulimit -n <some number> to increase the limit for your current shell. The number you choose cannot exceed the hard limit configured on your host. You can also change the ulimit permanently, more details here.

EARs are automatically undeployed in jboss-as-7.1.1.Final

I am not getting while EARs are undeployed automatically in jboss-as-7.1.1.Final.
I can see these logs:
ERROR org.apache.tomcat.util.net.JIoEndpoint$Acceptor [run] Socket accept failed: java.net.SocketException: Too many open files
WARN com.kpn.tie.ejbs.dao.webservice.tt.WebServiceProcessor [invoke] WebService unavailable. The request could not be completed due to technical problems. ; nested exception is: java.net.SocketException: Too many open files
Can somebody tell me root cause of this behavior and also suggest solution for this.
For workaround, restarting jboss in particular time interval will resolve this issue?

The reason could be that the application is overloaded or the file descriptor settings is too low. Due to this, the JVM can not open any new file handle, so you are getting Socket accept failed for incoming requests.
After a while the Deployment-Scanner comes into play (5 sec is default) and tries to check the deployments folder, which is not possible as it can not open any file-handle. So it gets confused and stops the deployed apps.
First solution could be:
Deactivate the scanner so that it only checks once during boot or remove the deployment scanner subsystem and use only CLI to deploy.
Second solution could be:
Increase the file-handler limit (open files size)

java.net.SocketException: Too many open files
On Linux you can increase the number of concurrently open files with
ulimit -n 2048
This would allow 2048 open at the same time in the current session. The command should be either inserted in the session configuration (e.g. .bashrc or similar, depends on your used shell) or in the JBoss start script.
To show the current limit you can use
ulimit -n

Apache Tomcat Exception - Too many open files

We are running a web service in Apache Tomcat in Amazon Linux. Initially web-service is running properly. We are getting too many open files exception after making more than 1000 web request. Again this issue will be resolved when we re start the tomcat server.
Please find below the Exception
25-Apr-2016 10:05:52.628 SEVERE [http-nio-8080-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept failed
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:686)
at java.lang.Thread.run(Thread.java:745)
PS : we are not doing any file related operations in the web service .

It looks like, that there is some limit on open files. As you are running on Linux I suspect you are running out of file descriptors.
Check out ulimit command to see the number of allowed opened files.
ulimit -n
You can change the number of open files by editing:
/etc/security/limits.conf
and adding something like this:
* soft nofile 4096
* hard nofile 4096
You can check more about limits.conf here.
The default limit is 1024 and can be too low for some Java applications.
More information about increasing the maximum number of open files in this article: http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/

Although if "ulimit" is raised at some point down the line tomcat stops causing same error.
So in order to avoid this you can check list of open files for the application user on Linux using command "lsof -u username" or simply "lsof" and see if code related files are open ( eg..properties files ) if so kill those specific files using # kill -9 lsof -t -u username command for that specific tomcat user.
You need to fix your code to load those files writing simply in a static block of your classes. So that only one file loads even if multiple hits are made by any number of users.
Now you can re check after deploying new changes with the same lsof command and see. Only one file will be seen. This will permanently fix your issue without raising the ulimit each time

That is because socket connections are treated as files, so that means you have too many connections opened. Check the limitations (each OS has different policy about it - same goes for each server), how many ports you can open at same time, etc. You can use NIO to limit those things.

java.io.IOException: Bad file descriptor Jetty 9.2.10.v20150310

I started jetty in nonstop server on port 18095 and it was running fine, few days later suddenly noticed it consumes more CPU and when I check the log noticed the following log writing continiously
2015-07-08 13:25:48.606:WARN:oejs.ServerConnector:qtp26807578-18-acceptor-0#182e42f-ServerConnector#1f02fde {HTTP/1.1}{0.0.0.0:18095}:
java.io.IOException: Bad file descriptor (errno:4009)
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:724)
Is there any way to fix this. Thanks

The "errno: 4009" is from outside of Java itself.
Something in the OS (or FileSystem) is preventing that particular incoming socket from being accepted.
If you are a unix system, consider evaluating your various ulimit values and bumping up the appropriate values to suit your needs better.
If you are on a Windows environment, don't run on Windows ME/2000 (as those have a long history of JVM/ServerSocket issues)

Oracle 11g connection reset error

Am seeing the below error while trying to connect to Oracle 11g on Red Hat Linux, 64-bit using thin jdbc drivers. Would highly appreciate if anyone can throw more light on how to go about troubleshooting this.
Caused by: java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at oracle.net.ns.DataPacket.send(DataPacket.java:199)
at oracle.net.ns.NetOutputStream.flush(NetOutputStream.java:211)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:227)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:175)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:100)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:85)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:123)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:79)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1122)
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1099)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:288)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
at oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:366)
at oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:752)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:366)

We've had something very similar, moved a program from 32-bit to 64bit OS & a number of concurrent processes would throw the IO Error: Connection Reset.
Stumbled across this which fixed it:
https://community.oracle.com/message/3701989
Basically add the -Djava.security.egd=file:/dev/./urandom parameter and it's good to go :)

You didn't include any details of the problem like what changed? Is this a new configuration for you or did it suddenly stop working? Do you know if you have enough connections available? Does this happen to every connection or is it intermittent?
Considering the error is occurring during the logon process, a few possibilities are:
Network fault
You have exhausted the maximum # of connections, so Oracle hangs up on you.
Firewall restrictions
A problem with the database server or the listener. The processing serving your session could be crashing after it's opened.
Check the following stackoverflow thread about how to check the number of active connections and the max. I would expect an "ORA-00018: maximum number of sessions exceeded" error if that were the problem, so it may not be. But it's worth checking.
How to check the maximum number of allowed connections to an Oracle database?

I faced similar issue with Sqoop Oracle import and I have implemented the fix suggested by this link
This setting helped to resolve issue:
-Dmapred.child.java.opts="-Djava.security.egd=file:///dev/urandom"
and I've changed mapred-site.xml configuration property name value as:
mapreduce.admin.map.child.java.opts -Djava.security.egd=file:///dev/urandom
This is a bit sqoop specific, but I think setting JVM option
-Djava.security.egd=file:///dev/urandom"
will help you to resolve issue.

Since the stack does not indicate any ORAs, you have underlying network problems.

Other thing that was causing me this problem was having the HOSTNAME settings wrong. My connection attempt was hanged at:
"main" prio=10 tid=0x00007f7cc8009000 nid=0x2f3a runnable [0x00007f7cce69e000]
java.lang.Thread.State: RUNNABLE
at java.net.Inet4AddressImpl.getLocalHostName(Native Method)
at java.net.InetAddress.getLocalHost(InetAddress.java:1444)
at sun.security.provider.SeedGenerator$1.run(SeedGenerator.java:176)
at sun.security.provider.SeedGenerator$1.run(SeedGenerator.java:162)
at java.security.AccessController.doPrivileged(Native Method)
So make sure you have an entry for your hostname in /etc/hosts/.
If you issue a hostname command like this:
$ hostname
my.server.com
You need a line in your /etc/hosts:
127.0.0.1 my my.server.com

In my application, I was creating BasicDataSource twice & it was failing with java 1.8 32 bit. Issue got resolved when I moved to java 1.8 64 bit