Develop Reference

Play Framework Stop responding after 2-3 days

I am using play framework and it stops responding after 2-3 days and when I restart server then everything works fine.
Please let me know what I am doing wrong.
Thanks
Stack trace:
Caused by: io.netty.channel.ChannelException: Failed to open a socket.
at io.netty.channel.socket.nio.NioSocketChannel.newSocket(NioSocketChannel.java:62)
at io.netty.channel.socket.nio.NioSocketChannel.<init>(NioSocketChannel.java:72)
at sun.reflect.GeneratedConstructorAccessor42.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at io.netty.bootstrap.AbstractBootstrap$BootstrapChannelFactory.newChannel(AbstractBootstrap.java:454)
... 64 common frames omitted
Caused by: java.net.SocketException: Too many open files
at sun.nio.ch.Net.socket0(Native Method)
at sun.nio.ch.Net.socket(Net.java:411)
at sun.nio.ch.Net.socket(Net.java:404)
at sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:105)
at sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60)
at io.netty.channel.socket.nio.NioSocketChannel.newSocket(NioSocketChannel.java:60)
... 70 common frames omitted

Looks like you are hitting the ulimit for your user. This is likely a function of some or all of the following:
Your user having the default ulimit (probably 256 or 1024 depending on the OS)
The amount/type of activity on your Play application
You can identify what open file handles your Play application by:
Running lsof -p PID on a *nix OS
Running something like Filemon (from sysinternals) on Windows
You'll likely see everything on your Play applications' classpath plus any files which your application opens e.g. log files, configuration files. In addition, if you are running on a *nix OS then open sockets will also consume file handles so you might see open file handles relating to database connection pools or indeed anything which your Play application communicates with via sockets.
Once you understand what your application is doing w.r.t open file handles you can consider what to do next. Likely to be one of:
Change your application such that it opens fewer file handles (and, if on *nix, uses fewer socket connections)
Change your application such that it closes open file handles when finished with them
Increase the number of open files allowed for your user by invoking ulimit -n <some number> to increase the limit for your current shell. The number you choose cannot exceed the hard limit configured on your host. You can also change the ulimit permanently, more details here.

EARs are automatically undeployed in jboss-as-7.1.1.Final

I am not getting while EARs are undeployed automatically in jboss-as-7.1.1.Final.
I can see these logs:
ERROR org.apache.tomcat.util.net.JIoEndpoint$Acceptor [run] Socket accept failed: java.net.SocketException: Too many open files
WARN com.kpn.tie.ejbs.dao.webservice.tt.WebServiceProcessor [invoke] WebService unavailable. The request could not be completed due to technical problems. ; nested exception is: java.net.SocketException: Too many open files
Can somebody tell me root cause of this behavior and also suggest solution for this.
For workaround, restarting jboss in particular time interval will resolve this issue?

The reason could be that the application is overloaded or the file descriptor settings is too low. Due to this, the JVM can not open any new file handle, so you are getting Socket accept failed for incoming requests.
After a while the Deployment-Scanner comes into play (5 sec is default) and tries to check the deployments folder, which is not possible as it can not open any file-handle. So it gets confused and stops the deployed apps.
First solution could be:
Deactivate the scanner so that it only checks once during boot or remove the deployment scanner subsystem and use only CLI to deploy.
Second solution could be:
Increase the file-handler limit (open files size)

java.net.SocketException: Too many open files
On Linux you can increase the number of concurrently open files with
ulimit -n 2048
This would allow 2048 open at the same time in the current session. The command should be either inserted in the session configuration (e.g. .bashrc or similar, depends on your used shell) or in the JBoss start script.
To show the current limit you can use
ulimit -n

AIX java.net.SocketException: A system call received a parameter that is not valid

I am really stuck with an AIX java issue. I have an issue here that doesn't match other issues on SO and on the Web.
My application code runs fine on another AIX server with the same exact JRE - IBM AIX Java 1.8, but does not run on the server that I need it to.
Both servers are AIX 7.1, running the same JAR and same JRE from the same tarball.
I'm getting the following error when using a Spring RestTemplate.exchange() to retrieve and unmarshal some JSON.
It must be a server configuration issue, but I'm very stuck and would appreciate any help!
Caused by: java.net.SocketException: A system call received a parameter that is not valid.
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:127)
at java.net.SocketInputStream.read(SocketInputStream.java:181)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at com.ibm.jsse2.a.a(a.java:209)
at com.ibm.jsse2.a.b(a.java:41)
at com.ibm.jsse2.a.a(a.java:193)
at com.ibm.jsse2.as.a(as.java:268)
at com.ibm.jsse2.as.a(as.java:745)
at com.ibm.jsse2.e.read(e.java:56)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:257)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:297)
at java.io.BufferedInputStream.read(BufferedInputStream.java:356)
at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:564)
at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:621)
at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:708)
at java.io.FilterInputStream.read(FilterInputStream.java:144)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3352)
at java.io.FilterInputStream.read(FilterInputStream.java:144)
at java.io.PushbackInputStream.read(PushbackInputStream.java:197)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.loadMore(UTF8StreamJsonParser.java:178)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseEscapedName(UTF8StreamJsonParser.java:1749)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.slowParseName(UTF8StreamJsonParser.java:1654)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._parseName(UTF8StreamJsonParser.java:1484)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:700)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:120)
at com.fasterxml.jackson.databind.deser.std.ObjectArrayDeserializer.deserialize(ObjectArrayDeserializer.java:149)
at com.fasterxml.jackson.databind.deser.std.ObjectArrayDeserializer.deserialize(ObjectArrayDeserializer.java:18)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:2993)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2158)
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:222)

It's probably a firewall issue.
Communication is cut abruptly by some sort of firewall, so the socket is closed by the OS, which gives error when you try to read from it.

java.lang.IllegalStateException while reloading Tomcat

I'm working on Spring with Hibernate application, It's working fine but while starting or reloading tomcat server, i'm getting java.lang.IllegalStateException Exception.
Can anybody explain, why this exception occurring and how to resolve it??
INFO: Illegal access: this web application instance has been stopped already. Could not load java.net.BindException. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
at com.mysql.jdbc.SQLError.createLinkFailureMessageBasedOnHeuristics(SQLError.java:1220)
at com.mysql.jdbc.exceptions.jdbc4.CommunicationsException.<init>(CommunicationsException.java:57)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1074)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3270)
at com.mysql.jdbc.MysqlIO.quit(MysqlIO.java:1659)
at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4296)
at com.mysql.jdbc.ConnectionImpl.cleanup(ConnectionImpl.java:1265)
at com.mysql.jdbc.ConnectionImpl.finalize(ConnectionImpl.java:2667)
at java.lang.System$2.invokeFinalize(Unknown Source)
at java.lang.ref.Finalizer.runFinalizer(Unknown Source)
at java.lang.ref.Finalizer.access$100(Unknown Source)

I want to give you couple of choices. You can try it. Any option can fulfill your demand.
Restart your tomcat and apache server because a long time of using,
it keeps older version of your application.
Clean your tomcat temp directory and restart
As stated in error,
The eventual following stack trace is caused by an error thrown for
debugging purposes as well as to attempt to terminate the thread which
caused the illegal access, and has no functional impact.
It may be the real cause that at this moment you are in debugging mode and it doesn't clear the running thread. so remove your break point and run it not debugging
If your code contains any kind of thread which are not properly terminated, then this type of error may occur.
That you have init() method but no destroy() method, then this type of error may occur. For details, you can follow the link - http://www.javaspecialists.eu/archive/Issue056.html
if the webapp has stopped, or is stopping, that means that the .war file or WEB-INF/web.xml timestamp changed, and the webapp might be reloading. Please check timestamp is OK or not.
To turn it off, set reloadable="false" in the context definition of
your app. It might be tomcat's server.xml.
In details solution:
The tomcat's server.xml of the reloadable Context is set to false.
For example:
Context path="/expert" docBase="expert" debug="0" reloadable ="false"/>
The solution is easy, as long as the tomcat's server.xml in reloadable = "true" into false on the line, but to do so would lose the advantage of hot deployment, and for the development is not very convenient, simply change it or not. This error does not matter.
Error principle:
The reason is because the tomcat restart, because the previous tomcat thread has not completely shut down, restart tomcat will report this exception, but this does not affect the normal use, just jump abnormal annoying people. Used hibernate, spring or other large components, when a WEB application system has a lot of class,
if you turned the Tomcat reloadable = true, then whenever the relevant documents change, Tomcat stops web app and frees up memory, and then reload web app. it may be a huge project. So we always think if there is only a certain class of overloaded functions, will greatly meet our debugger.
UPDATE: For code related issue
First, I want to tell you that I have given you some solutions for tomcat basis. Now I want to give you a solution for code basis. Would you please cross check your code with this issue? Please follow the URL.
http://www.coderanch.com/t/660126/Wiki/Illegal-State-Exception
UPDATE: For MySQL related issue
There are 2 issues.
This web application instance has been stopped already. Could not load java.net.BindException.
This web application instance has been
stopped already.Could not load com.mysql.jdbc.
This is because the MySQL JDBC driver on the application under the WEB-INF/lib directory, in the re-release of its loaded twice, so long as it can be copied to %TOMCAT_HOME%/lib can solve the problem.
We can solve these two anomalies MySQL drivers from the WEB-INF/lib folder moved to %TOMCAT_HOME%/lib.
I suspect this might be happening after the web application is restarted, where it's down for a short period of time. Then some finalize() method in the code is probably trying to do some cleanup too late. Whether or not that's in your code or the MySQL driver I can't say. You definitely should only have one version of a jar in a directory at a time. You might want to upgrade it to the latest (5.1.38 right now) in case something has been fixed that might be affecting you.(Number 9 is copied from #WhiteFang34)
Related link for 9: tomcat 6.0.24 Exception: Could not load com.mysql.jdbc.SQLError

The issue was because of the server cache. Follow the below steps to fix this issue
Find the directory called "work" under your tomcat's root directory
Delete it completely
Restart the server
Issue will be fixed.
Note: In case you have default tomcat installation in linux machine, you have to delete the folder /var/cache/tomcat7/Catalina

I think that the root cause of this problem is that something is leaking JDBC Connection objects.
These objects are actually instances of com.mysql.jdbc.ConnectionImpl ... which is a class with a finalize(). Normally, object finalization tidies up the leaked connections. In this case, it looks like the sequence of events is as follows:
the GC runs after the webapp has shutdown
a ConnectionImpl object is found to be unreachable
the object's finalize() method attempts to shutdown the MySQL database connection ... which entails sending a message to the server
when the message send fails for some reason, and the comms-level code tries to create an exception by calling com.mysql.jdbc.SQLError.createCommunicationsException
that tries to create an exception reflectively ...
The last step is failing because it is trying to use the webapp's classloader for the webapp ... which has been shut down. Apparently, this is detected by a check that is intended to help programmers diagnose webapps that don't shutdown cleanly.
The best solution is to track down the source of the Connection leak(s) and fix it. If Connection objects did not leak, then they would be closed while the webapp's classloader was still active.
Alternatively, you could simply turn off the check, as described in #Skywalker's answer.

The problem could be a simple SQL Error. tThe
java.lang.IllegalStateException
means that an object or class was called on when it was not ready to be called on. These errors at the bottom of the stack
at java.lang.ref.Finalizer.runFinalizer(Unknown Source)
at java.lang.ref.Finalizer.access$100(Unknown Source)
at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
look like they come from the Java Garbage Collector
attempting to clean up MySQL Object Pointers
that were temporarily unreferenced during a reboot. It seems to me that your error here is just a few MySQL Object Pointers being halted in the middle of execution when the GC calls their finalize methods and then pauses the application to attempt a clean up the object. (leading to a fail and a throw of this exception)
This error is most likely harmless, and could maybe be fixed by adding a
JNI_DestroyJavaVM();
to your main thread to be executed on shutdown.

Too many open files Exception in JBoss AS 4.2.3-GA

I want to fix this issue, I actually don't have a clear Idea of what is happening when my application runs over a CentOs environment and after some days I start getting the following exceptions:
2011-07-12 21:58:03,598 12155907 ERROR [org.jboss.naming.Naming] (JBoss System Threads(1)-2:) Naming accept handler stopping
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at org.jnp.server.Main$AcceptHandler.run(Main.java:481)
at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
at java.lang.Thread.run(Thread.java:662)
2011-07-12 21:58:03,600 12155909 ERROR [org.jboss.naming.Naming] (JBoss System Threads(1)-2:) Naming accept handler stopping
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at org.jnp.server.Main$AcceptHandler.run(Main.java:481)
at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
at java.lang.Thread.run(Thread.java:662)
2011-07-12 21:58:03,600 12155909 ERROR [org.jboss.naming.Naming] (JBoss System Threads(1)-2:) Naming accept handler stopping
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at org.jnp.server.Main$AcceptHandler.run(Main.java:481)
at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
at java.lang.Thread.run(Thread.java:662)
2011-07-12 21:58:03,601 12155910 ERROR [org.jboss.naming.Naming] (JBoss System Threads(1)-2:) Naming accept handler stopping
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at org.jnp.server.Main$AcceptHandler.run(Main.java:481)
at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
at java.lang.Thread.run(Thread.java:662)
The logs start growing quick... I am using some JBOSS QUEUES in my application, as well some JMX connections, I want to know how to detect the problem quickly or if this is caused by other component in the computer. Any suggestions, please I am totally worried about this issue.

Here are some tips on debugging a Too Many Open Files situation. Two unix commands that can be of help are ulimit and lsof
I suggest you man ulimit first to understand that you can alter the maximum number of open files for a process. Just typing ulimit on the commandline will give you the default value for ulimit. For example, for me:
$ ulimit
unlimited
I'm running on a vanilla Ubuntu 11.04 distro, so looks like unlimited is the default. On most of my production boxes, the default is 1024.
Next, lsof -p <pid> which will list all open files for process with id <pid>.
Perhaps you will find that you are either not closing files when thought you were or you are just opening too many files (based on the limit set up ulimit)?
Next step, you are running a Java process so you can get a thread dump of your process while the Too Many Open Files situation is occurring. To get a thread dump, either send the process a kill -3 <pid> or if you have started the Java process in your current shell, you can type CTRL-Break to get a thread dump. It's especially helpful to gather 3 or more thread dumps within, say a minute or two, and from this collection of thread dumps, so the threads that exist in a thread dumps are worth taking a look at.
If none of this helps you, search SO for "too many open files". I did and found this link, which the accepted answer may help you
Java Too Many Open Files

I think you are hitting this issue - AJP connection is left as CLOSE_WAIT