We are running a web service in Apache Tomcat in Amazon Linux. Initially web-service is running properly. We are getting too many open files exception after making more than 1000 web request. Again this issue will be resolved when we re start the tomcat server.
Please find below the Exception
25-Apr-2016 10:05:52.628 SEVERE [http-nio-8080-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept failed
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:686)
at java.lang.Thread.run(Thread.java:745)
PS : we are not doing any file related operations in the web service .
It looks like, that there is some limit on open files. As you are running on Linux I suspect you are running out of file descriptors.
Check out ulimit command to see the number of allowed opened files.
ulimit -n
You can change the number of open files by editing:
/etc/security/limits.conf
and adding something like this:
* soft nofile 4096
* hard nofile 4096
You can check more about limits.conf here.
The default limit is 1024 and can be too low for some Java applications.
More information about increasing the maximum number of open files in this article: http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/
Although if "ulimit" is raised at some point down the line tomcat stops causing same error.
So in order to avoid this you can check list of open files for the application user on Linux using command "lsof -u username" or simply "lsof" and see if code related files are open ( eg..properties files ) if so kill those specific files using # kill -9 lsof -t -u username command for that specific tomcat user.
You need to fix your code to load those files writing simply in a static block of your classes. So that only one file loads even if multiple hits are made by any number of users.
Now you can re check after deploying new changes with the same lsof command and see. Only one file will be seen. This will permanently fix your issue without raising the ulimit each time
That is because socket connections are treated as files, so that means you have too many connections opened. Check the limitations (each OS has different policy about it - same goes for each server), how many ports you can open at same time, etc. You can use NIO to limit those things.
Related
So apologies if what I ask is trivial but I am experimenting with Memcached and Jmeter. I have a Memcached server setup (as far as I can tell) and am able to make telnet requests to it via telnet IP PORT and additionally set and get using commands set and get appropriately.
Now point me to a different application if perhaps this is the wrong choice; but my understanding was that Jmeter should allow me to pound the server with equivalent Set and Get requests.
Unfortunately the experimental platform is a remote linux PC running Rockylinux which is similar to CentOS / RedHat to my understanding (I didn't set this part up); and as a result I do not have a GUI to launch while on the Linux PC. I have however opened Jmeter up on my local PC on windows and understand I should be able to send the test file over and run it.
I followed these instructions to try to setup a TCP sampler and set the "text to send" field as below; after doing the additional step in the link regarding the precompiler.
set tutorialspoint 0 900 9${CR}${LF}
memcached${CR}${LF}
quit${CR}${LF}
Running the above as a headless jmeter session doesn't generate any errors called [./jmeter -n -t "Sample.jmx" -l testresults.jtl"] but when I connect via telnet I'm also not seeing the value for the key "tutorialspoint" get updated. When manually doing the get and set I am seeing updates. Any ideas what I could be doing wrong? Checking the log indicates ResponseCode 200 OK as expected. Is there a good method to debug something in a Headless setup?
Thanks for your time.
I believe the easiest way is using Memcached Java Client library
Download spymemcached-2.12.3.jar and drop it to "lib" folder of your JMeter installation (or any other location in JMeter Classpath)
Restart JMeter to pick the .jar up
Add JSR223 Sampler to your test plan and use the following code snippets:
def client = new net.spy.memcached.MemcachedClient(new InetSocketAddress('your-memcached-host', your-memcached-port)) - for connecting to the server
client.set('tutorialspoint', 900, 'memcached').done to write memcached to the tutorialspoint key for 15 minutes
client.get('tutorialspoint') - to read the value of tutorialspoint key
client.shutdown() - to disconnect
More information on Groovy scripting in JMeter: Apache Groovy - Why and How You Should Use It
Demo:
I am using play framework and it stops responding after 2-3 days and when I restart server then everything works fine.
Please let me know what I am doing wrong.
Thanks
Stack trace:
Caused by: io.netty.channel.ChannelException: Failed to open a socket.
at io.netty.channel.socket.nio.NioSocketChannel.newSocket(NioSocketChannel.java:62)
at io.netty.channel.socket.nio.NioSocketChannel.<init>(NioSocketChannel.java:72)
at sun.reflect.GeneratedConstructorAccessor42.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at io.netty.bootstrap.AbstractBootstrap$BootstrapChannelFactory.newChannel(AbstractBootstrap.java:454)
... 64 common frames omitted
Caused by: java.net.SocketException: Too many open files
at sun.nio.ch.Net.socket0(Native Method)
at sun.nio.ch.Net.socket(Net.java:411)
at sun.nio.ch.Net.socket(Net.java:404)
at sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:105)
at sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60)
at io.netty.channel.socket.nio.NioSocketChannel.newSocket(NioSocketChannel.java:60)
... 70 common frames omitted
Looks like you are hitting the ulimit for your user. This is likely a function of some or all of the following:
Your user having the default ulimit (probably 256 or 1024 depending on the OS)
The amount/type of activity on your Play application
You can identify what open file handles your Play application by:
Running lsof -p PID on a *nix OS
Running something like Filemon (from sysinternals) on Windows
You'll likely see everything on your Play applications' classpath plus any files which your application opens e.g. log files, configuration files. In addition, if you are running on a *nix OS then open sockets will also consume file handles so you might see open file handles relating to database connection pools or indeed anything which your Play application communicates with via sockets.
Once you understand what your application is doing w.r.t open file handles you can consider what to do next. Likely to be one of:
Change your application such that it opens fewer file handles (and, if on *nix, uses fewer socket connections)
Change your application such that it closes open file handles when finished with them
Increase the number of open files allowed for your user by invoking ulimit -n <some number> to increase the limit for your current shell. The number you choose cannot exceed the hard limit configured on your host. You can also change the ulimit permanently, more details here.
I am not getting while EARs are undeployed automatically in jboss-as-7.1.1.Final.
I can see these logs:
ERROR org.apache.tomcat.util.net.JIoEndpoint$Acceptor [run] Socket accept failed: java.net.SocketException: Too many open files
WARN com.kpn.tie.ejbs.dao.webservice.tt.WebServiceProcessor [invoke] WebService unavailable. The request could not be completed due to technical problems. ; nested exception is: java.net.SocketException: Too many open files
Can somebody tell me root cause of this behavior and also suggest solution for this.
For workaround, restarting jboss in particular time interval will resolve this issue?
The reason could be that the application is overloaded or the file descriptor settings is too low. Due to this, the JVM can not open any new file handle, so you are getting Socket accept failed for incoming requests.
After a while the Deployment-Scanner comes into play (5 sec is default) and tries to check the deployments folder, which is not possible as it can not open any file-handle. So it gets confused and stops the deployed apps.
First solution could be:
Deactivate the scanner so that it only checks once during boot or remove the deployment scanner subsystem and use only CLI to deploy.
Second solution could be:
Increase the file-handler limit (open files size)
java.net.SocketException: Too many open files
On Linux you can increase the number of concurrently open files with
ulimit -n 2048
This would allow 2048 open at the same time in the current session. The command should be either inserted in the session configuration (e.g. .bashrc or similar, depends on your used shell) or in the JBoss start script.
To show the current limit you can use
ulimit -n
I have a Java application which is running as non root mode.
My App will create a TFTP server (using apache commons tftp). TFTP server is bind to port 69(Default TFTP port). When running the app from IDE everything works fine since the IDE running as root. But if the app is run from other user i get the error
java.net.BindException: Permission denied
It is clear that for non root user i can not open the port. Is there a workaround for this issue?
For binding on Linux of ports less that 1024 you need to application to run a root. There is no way around this. If you need to do this you have you run as root. sudo might be the command to look into.
BTW - Running your IDE as root is not a very good idea.
To resolve this issue. You can use setuid() and setfid() system calls. So that you can temporarily elevate the permissions and then drop the permission back to user permissions.
In my Case, this problem happened in Solaris 11 OS. I added privileges to user to use the ports under 1024.
https://technicalsanctuary.wordpress.com/2014/06/03/allowing-a-user-to-use-ports-under-1024-on-solaris-11/
I have a Tomcat installation where I suspect the thread pool may be decreasing over time due to threads not being properly released. I get an error in catalina.out when maxthreads is reached, but I would like to log the number of threads in use to a file every five minutes so I can verify this hypothesis. Would anyone please be able to advise how this can be be done?
Also in this installation there is no Tomcat manager, it appears whoever did the original installation deleted the manager webapp for some reason. I'm not sure if manager would be able to do the above or if I can reinstall it without damaging the existing installation? All I really want to do is keep track of the thread pool.
Also, I noticed that maxthreads for Tomcat is 200, but the max number of concurrent connections for Apache is lower (Apache is using mod_proxy and mod_proxy_ajp (AJP 1.3) to feed Tomcat). That seems wrong too, what is the correct relationship between these numbers?
Any help much appreciated :D
Update: Just a quick update to say the direct JMX access worked. However I also had to set Dcom.sun.management.jmxremote.host. I set it to localhost and it worked, however without it no dice. If anyone else has a similar problem trying to enable JMX I recommend you set this value also, even if you are connecting from the local machine. Seems it is required with some versions of Tomcat.
Just a quick update to say the direct JMX access worked. However I also had to set Dcom.sun.management.jmxremote.host. I set it to localhost and it worked, however without it no dice. If anyone else has a similar problem trying to enable JMX I recommend you set this value also, even if you are connecting from the local machine. Seems it is required with some versions of Tomcat.
Direct JMX access
Try adding this to catalina.sh/bat:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=5005
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
UPDATE: Alex P suggest that the following settings might also be required in some situations:
-Dcom.sun.management.jmxremote.host=localhost
This enables remote anonymous JMX connections on port 5005. You may also consider JVisualVM which is much more please and allows to browse JMX via plugin.
What you are looking for is Catalina -> ThreadPool -> http-bio-8080 -> various interesting metrics.
JMX proxy servlet
Easier method might be to use Tomcat's JMX proxy servlet under: http://localhost:8080/manager/jmxproxy. For instance try this query:
$ curl --user tomcat:tomcat http://localhost:8080/manager/jmxproxy?qry=Catalina:name=%22http-bio-8080%22,type=ThreadPool
A little bit of grepping and scripting and you can easily and remotely monitor your application. Note that tomcat:tomcat is the username/password of user having manager-jmx role in conf/tomcat-users.xml.
You can deploy jolokia.war and then retrieve mbeans values in JSON (without the manager):
http://localhost:8080/jolokia/read/Catalina:name=*,type=ThreadPool?ignoreErrors=true
If you want only some values (currentThreadsBusy, maxThreads, currentThreadCount, connectionCount):
http://localhost:8080/jolokia/read/Catalina:name=*,type=ThreadPool/currentThreadsBusy,maxThreads,currentThreadCount,connectionCount?ignoreErrors=true
{
request: {
mbean: "Catalina:name="http-nio-8080",type=ThreadPool",
attribute: [
"currentThreadsBusy",
"maxThreads",
"currentThreadCount",
"connectionCount"
],
type: "read"
},
value: {
currentThreadsBusy: 1,
connectionCount: 4,
currentThreadCount: 10,
maxThreads: 200
},
timestamp: 1490396960,
status: 200
}
Note: This example works on Tomcat7 +.
For a more enterprise solution. I have been using New Relic in our production environment.
This provides a graph of the changes to the threadpool over time.
There are cheaper tools out meanwhile: I am using this jar here: https://docs.cyclopsgroup.org/jmxterm
You can automate it via shell/batch scripts. I regexed the output and let prometheus poll it for displaying it in grafana.