I am using play framework and it stops responding after 2-3 days and when I restart server then everything works fine.
Please let me know what I am doing wrong.
Thanks
Stack trace:
Caused by: io.netty.channel.ChannelException: Failed to open a socket.
at io.netty.channel.socket.nio.NioSocketChannel.newSocket(NioSocketChannel.java:62)
at io.netty.channel.socket.nio.NioSocketChannel.<init>(NioSocketChannel.java:72)
at sun.reflect.GeneratedConstructorAccessor42.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at io.netty.bootstrap.AbstractBootstrap$BootstrapChannelFactory.newChannel(AbstractBootstrap.java:454)
... 64 common frames omitted
Caused by: java.net.SocketException: Too many open files
at sun.nio.ch.Net.socket0(Native Method)
at sun.nio.ch.Net.socket(Net.java:411)
at sun.nio.ch.Net.socket(Net.java:404)
at sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:105)
at sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60)
at io.netty.channel.socket.nio.NioSocketChannel.newSocket(NioSocketChannel.java:60)
... 70 common frames omitted
Looks like you are hitting the ulimit for your user. This is likely a function of some or all of the following:
Your user having the default ulimit (probably 256 or 1024 depending on the OS)
The amount/type of activity on your Play application
You can identify what open file handles your Play application by:
Running lsof -p PID on a *nix OS
Running something like Filemon (from sysinternals) on Windows
You'll likely see everything on your Play applications' classpath plus any files which your application opens e.g. log files, configuration files. In addition, if you are running on a *nix OS then open sockets will also consume file handles so you might see open file handles relating to database connection pools or indeed anything which your Play application communicates with via sockets.
Once you understand what your application is doing w.r.t open file handles you can consider what to do next. Likely to be one of:
Change your application such that it opens fewer file handles (and, if on *nix, uses fewer socket connections)
Change your application such that it closes open file handles when finished with them
Increase the number of open files allowed for your user by invoking ulimit -n <some number> to increase the limit for your current shell. The number you choose cannot exceed the hard limit configured on your host. You can also change the ulimit permanently, more details here.
Related
I am not getting while EARs are undeployed automatically in jboss-as-7.1.1.Final.
I can see these logs:
ERROR org.apache.tomcat.util.net.JIoEndpoint$Acceptor [run] Socket accept failed: java.net.SocketException: Too many open files
WARN com.kpn.tie.ejbs.dao.webservice.tt.WebServiceProcessor [invoke] WebService unavailable. The request could not be completed due to technical problems. ; nested exception is: java.net.SocketException: Too many open files
Can somebody tell me root cause of this behavior and also suggest solution for this.
For workaround, restarting jboss in particular time interval will resolve this issue?
The reason could be that the application is overloaded or the file descriptor settings is too low. Due to this, the JVM can not open any new file handle, so you are getting Socket accept failed for incoming requests.
After a while the Deployment-Scanner comes into play (5 sec is default) and tries to check the deployments folder, which is not possible as it can not open any file-handle. So it gets confused and stops the deployed apps.
First solution could be:
Deactivate the scanner so that it only checks once during boot or remove the deployment scanner subsystem and use only CLI to deploy.
Second solution could be:
Increase the file-handler limit (open files size)
java.net.SocketException: Too many open files
On Linux you can increase the number of concurrently open files with
ulimit -n 2048
This would allow 2048 open at the same time in the current session. The command should be either inserted in the session configuration (e.g. .bashrc or similar, depends on your used shell) or in the JBoss start script.
To show the current limit you can use
ulimit -n
We are running a web service in Apache Tomcat in Amazon Linux. Initially web-service is running properly. We are getting too many open files exception after making more than 1000 web request. Again this issue will be resolved when we re start the tomcat server.
Please find below the Exception
25-Apr-2016 10:05:52.628 SEVERE [http-nio-8080-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept failed
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:686)
at java.lang.Thread.run(Thread.java:745)
PS : we are not doing any file related operations in the web service .
It looks like, that there is some limit on open files. As you are running on Linux I suspect you are running out of file descriptors.
Check out ulimit command to see the number of allowed opened files.
ulimit -n
You can change the number of open files by editing:
/etc/security/limits.conf
and adding something like this:
* soft nofile 4096
* hard nofile 4096
You can check more about limits.conf here.
The default limit is 1024 and can be too low for some Java applications.
More information about increasing the maximum number of open files in this article: http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/
Although if "ulimit" is raised at some point down the line tomcat stops causing same error.
So in order to avoid this you can check list of open files for the application user on Linux using command "lsof -u username" or simply "lsof" and see if code related files are open ( eg..properties files ) if so kill those specific files using # kill -9 lsof -t -u username command for that specific tomcat user.
You need to fix your code to load those files writing simply in a static block of your classes. So that only one file loads even if multiple hits are made by any number of users.
Now you can re check after deploying new changes with the same lsof command and see. Only one file will be seen. This will permanently fix your issue without raising the ulimit each time
That is because socket connections are treated as files, so that means you have too many connections opened. Check the limitations (each OS has different policy about it - same goes for each server), how many ports you can open at same time, etc. You can use NIO to limit those things.
I started jetty in nonstop server on port 18095 and it was running fine, few days later suddenly noticed it consumes more CPU and when I check the log noticed the following log writing continiously
2015-07-08 13:25:48.606:WARN:oejs.ServerConnector:qtp26807578-18-acceptor-0#182e42f-ServerConnector#1f02fde {HTTP/1.1}{0.0.0.0:18095}:
java.io.IOException: Bad file descriptor (errno:4009)
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:724)
Is there any way to fix this. Thanks
The "errno: 4009" is from outside of Java itself.
Something in the OS (or FileSystem) is preventing that particular incoming socket from being accepted.
If you are a unix system, consider evaluating your various ulimit values and bumping up the appropriate values to suit your needs better.
If you are on a Windows environment, don't run on Windows ME/2000 (as those have a long history of JVM/ServerSocket issues)
I am trying to execute this jnlp application.
However, I keep receiving this security exception:
java.lang.ExceptionInInitializerError
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:399)
at java.lang.Class.newInstance0(Class.java:370)
at java.lang.Class.newInstance(Class.java:322)
at com.sun.javafx.applet.FXApplet2.init(FXApplet2.java:63)
at com.sun.deploy.uitoolkit.impl.fx.FXApplet2Adapter.init(FXApplet2Adapter.java:207)
at sun.plugin2.applet.Plugin2Manager$AppletExecutionRunnable.run(Unknown Source)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.security.AccessControlException: access denied ("java.util.PropertyPermission" "java.net.useSystemProxies" "write")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:364)
at java.security.AccessController.checkPermission(AccessController.java:560)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at java.lang.System.setProperty(System.java:783)
at ensemble.Ensemble2.<clinit>(Ensemble2.java:77)
... 10 more
I tried changing the security settings in the Java control panel (OSX 10.8.3), but they seem to be disabled and I cannot modify them (I do can check the "Enable Java content in the browser" option, but it is resetted when I click apply). This is shown in the following figure:
Is there another mechanism for bypassing the Java security manager and being able to execute a jnlp application ?
UPDATE:
After reinstalling Java I was finally able to change the security options in the Java Control panel (I did not do anything special, just reinstall it from scratch). Unfortunately, when I execute the jnlp application it dies silently without showing any errors. This problem could be related to the fact that I am using the jdk 1.8 (since the jnlp application requires javafx) and it is just a preview version (and OSX does not seem to like a lot Java these days...), it is my best guess but I cannot be completely sure.
Possible reasons for this:
I have heard that Java 7 on Mac is still not stable.
By default JNLP files run in a "sandbox" to prevent malicious code from wreaking havoc on your system. This sandbox restricts access to the local filesystem and local network. I imagine that system properties are restricted too, and it looks like the app is trying to write to them. To allow unrestricted access, you need to sign the JAR file and configure the JNLP file with <security><all-permissions/></security>
I want to fix this issue, I actually don't have a clear Idea of what is happening when my application runs over a CentOs environment and after some days I start getting the following exceptions:
2011-07-12 21:58:03,598 12155907 ERROR [org.jboss.naming.Naming] (JBoss System Threads(1)-2:) Naming accept handler stopping
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at org.jnp.server.Main$AcceptHandler.run(Main.java:481)
at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
at java.lang.Thread.run(Thread.java:662)
2011-07-12 21:58:03,600 12155909 ERROR [org.jboss.naming.Naming] (JBoss System Threads(1)-2:) Naming accept handler stopping
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at org.jnp.server.Main$AcceptHandler.run(Main.java:481)
at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
at java.lang.Thread.run(Thread.java:662)
2011-07-12 21:58:03,600 12155909 ERROR [org.jboss.naming.Naming] (JBoss System Threads(1)-2:) Naming accept handler stopping
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at org.jnp.server.Main$AcceptHandler.run(Main.java:481)
at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
at java.lang.Thread.run(Thread.java:662)
2011-07-12 21:58:03,601 12155910 ERROR [org.jboss.naming.Naming] (JBoss System Threads(1)-2:) Naming accept handler stopping
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at org.jnp.server.Main$AcceptHandler.run(Main.java:481)
at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
at java.lang.Thread.run(Thread.java:662)
The logs start growing quick... I am using some JBOSS QUEUES in my application, as well some JMX connections, I want to know how to detect the problem quickly or if this is caused by other component in the computer. Any suggestions, please I am totally worried about this issue.
Here are some tips on debugging a Too Many Open Files situation. Two unix commands that can be of help are ulimit and lsof
I suggest you man ulimit first to understand that you can alter the maximum number of open files for a process. Just typing ulimit on the commandline will give you the default value for ulimit. For example, for me:
$ ulimit
unlimited
I'm running on a vanilla Ubuntu 11.04 distro, so looks like unlimited is the default. On most of my production boxes, the default is 1024.
Next, lsof -p <pid> which will list all open files for process with id <pid>.
Perhaps you will find that you are either not closing files when thought you were or you are just opening too many files (based on the limit set up ulimit)?
Next step, you are running a Java process so you can get a thread dump of your process while the Too Many Open Files situation is occurring. To get a thread dump, either send the process a kill -3 <pid> or if you have started the Java process in your current shell, you can type CTRL-Break to get a thread dump. It's especially helpful to gather 3 or more thread dumps within, say a minute or two, and from this collection of thread dumps, so the threads that exist in a thread dumps are worth taking a look at.
If none of this helps you, search SO for "too many open files". I did and found this link, which the accepted answer may help you
Java Too Many Open Files
I think you are hitting this issue - AJP connection is left as CLOSE_WAIT