Develop Reference

Play Framework Stop responding after 2-3 days

I am using play framework and it stops responding after 2-3 days and when I restart server then everything works fine.
Please let me know what I am doing wrong.
Thanks
Stack trace:
Caused by: io.netty.channel.ChannelException: Failed to open a socket.
at io.netty.channel.socket.nio.NioSocketChannel.newSocket(NioSocketChannel.java:62)
at io.netty.channel.socket.nio.NioSocketChannel.<init>(NioSocketChannel.java:72)
at sun.reflect.GeneratedConstructorAccessor42.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at io.netty.bootstrap.AbstractBootstrap$BootstrapChannelFactory.newChannel(AbstractBootstrap.java:454)
... 64 common frames omitted
Caused by: java.net.SocketException: Too many open files
at sun.nio.ch.Net.socket0(Native Method)
at sun.nio.ch.Net.socket(Net.java:411)
at sun.nio.ch.Net.socket(Net.java:404)
at sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:105)
at sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60)
at io.netty.channel.socket.nio.NioSocketChannel.newSocket(NioSocketChannel.java:60)
... 70 common frames omitted

Looks like you are hitting the ulimit for your user. This is likely a function of some or all of the following:
Your user having the default ulimit (probably 256 or 1024 depending on the OS)
The amount/type of activity on your Play application
You can identify what open file handles your Play application by:
Running lsof -p PID on a *nix OS
Running something like Filemon (from sysinternals) on Windows
You'll likely see everything on your Play applications' classpath plus any files which your application opens e.g. log files, configuration files. In addition, if you are running on a *nix OS then open sockets will also consume file handles so you might see open file handles relating to database connection pools or indeed anything which your Play application communicates with via sockets.
Once you understand what your application is doing w.r.t open file handles you can consider what to do next. Likely to be one of:
Change your application such that it opens fewer file handles (and, if on *nix, uses fewer socket connections)
Change your application such that it closes open file handles when finished with them
Increase the number of open files allowed for your user by invoking ulimit -n <some number> to increase the limit for your current shell. The number you choose cannot exceed the hard limit configured on your host. You can also change the ulimit permanently, more details here.

Apache Tomcat Exception - Too many open files

We are running a web service in Apache Tomcat in Amazon Linux. Initially web-service is running properly. We are getting too many open files exception after making more than 1000 web request. Again this issue will be resolved when we re start the tomcat server.
Please find below the Exception
25-Apr-2016 10:05:52.628 SEVERE [http-nio-8080-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept failed
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:686)
at java.lang.Thread.run(Thread.java:745)
PS : we are not doing any file related operations in the web service .

It looks like, that there is some limit on open files. As you are running on Linux I suspect you are running out of file descriptors.
Check out ulimit command to see the number of allowed opened files.
ulimit -n
You can change the number of open files by editing:
/etc/security/limits.conf
and adding something like this:
* soft nofile 4096
* hard nofile 4096
You can check more about limits.conf here.
The default limit is 1024 and can be too low for some Java applications.
More information about increasing the maximum number of open files in this article: http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/

Although if "ulimit" is raised at some point down the line tomcat stops causing same error.
So in order to avoid this you can check list of open files for the application user on Linux using command "lsof -u username" or simply "lsof" and see if code related files are open ( eg..properties files ) if so kill those specific files using # kill -9 lsof -t -u username command for that specific tomcat user.
You need to fix your code to load those files writing simply in a static block of your classes. So that only one file loads even if multiple hits are made by any number of users.
Now you can re check after deploying new changes with the same lsof command and see. Only one file will be seen. This will permanently fix your issue without raising the ulimit each time

That is because socket connections are treated as files, so that means you have too many connections opened. Check the limitations (each OS has different policy about it - same goes for each server), how many ports you can open at same time, etc. You can use NIO to limit those things.

SFTP via JSch is throwing error 4: Failure

I am facing a peculiar issue while trying to SFTP a file from Windows to Unix server. The error "stack trace" is -
4: Failure
at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2846)
at com.jcraft.jsch.ChannelSftp.checkStatus(ChannelSftp.java:2459)
at com.jcraft.jsch.ChannelSftp._sendCLOSE(ChannelSftp.java:2465)
at com.jcraft.jsch.ChannelSftp._put(ChannelSftp.java:683)
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:475)
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:365)
I have searched a lot on other forums but could not get to the root cause. I have observed one more thing - the file name, I am trying to SFTP, is *.xml when I change it to *.XML the SFTP is "successful". Very weird could not identify the issue. I am trying this using JSch Java API. But I have seen the same issue while trying the WinSCP front end.

It is common for OpenSSH SFTP server to return code 4 (Failure) for many different reasons, without giving any details.
You would have to check SFTP server log to find more.
Though the most usual cause is a lack of free disk or user quota space – Which can usually be identified by the file being created with zero size.

i was facing the same issue. was puzzled then i realized there was a space issue on the server folder. Please do check the server available space.
command: df path to the folder

I was facing the same issue. Please make sure that your SFTP service is running. And If it is already running then restarting this service should do the trick.

I had the same issue. Tried everything, turns out I had set the timeout to what I thought was 30s but it was actually 30ms. Changing it to 30000ms solved the issue.
session.setTimeout(30000);

I've seen this error and the cause in my case was, the uploaded file names weren't supported by the file system used by the SFTP server.

Binding a port < 1024 for non root user in Java

I have a Java application which is running as non root mode.
My App will create a TFTP server (using apache commons tftp). TFTP server is bind to port 69(Default TFTP port). When running the app from IDE everything works fine since the IDE running as root. But if the app is run from other user i get the error
java.net.BindException: Permission denied
It is clear that for non root user i can not open the port. Is there a workaround for this issue?

For binding on Linux of ports less that 1024 you need to application to run a root. There is no way around this. If you need to do this you have you run as root. sudo might be the command to look into.
BTW - Running your IDE as root is not a very good idea.

To resolve this issue. You can use setuid() and setfid() system calls. So that you can temporarily elevate the permissions and then drop the permission back to user permissions.

In my Case, this problem happened in Solaris 11 OS. I added privileges to user to use the ports under 1024.
https://technicalsanctuary.wordpress.com/2014/06/03/allowing-a-user-to-use-ports-under-1024-on-solaris-11/

How can I monitor/log Tomcat's thread pool?

I have a Tomcat installation where I suspect the thread pool may be decreasing over time due to threads not being properly released. I get an error in catalina.out when maxthreads is reached, but I would like to log the number of threads in use to a file every five minutes so I can verify this hypothesis. Would anyone please be able to advise how this can be be done?
Also in this installation there is no Tomcat manager, it appears whoever did the original installation deleted the manager webapp for some reason. I'm not sure if manager would be able to do the above or if I can reinstall it without damaging the existing installation? All I really want to do is keep track of the thread pool.
Also, I noticed that maxthreads for Tomcat is 200, but the max number of concurrent connections for Apache is lower (Apache is using mod_proxy and mod_proxy_ajp (AJP 1.3) to feed Tomcat). That seems wrong too, what is the correct relationship between these numbers?
Any help much appreciated :D
Update: Just a quick update to say the direct JMX access worked. However I also had to set Dcom.sun.management.jmxremote.host. I set it to localhost and it worked, however without it no dice. If anyone else has a similar problem trying to enable JMX I recommend you set this value also, even if you are connecting from the local machine. Seems it is required with some versions of Tomcat.
Just a quick update to say the direct JMX access worked. However I also had to set Dcom.sun.management.jmxremote.host. I set it to localhost and it worked, however without it no dice. If anyone else has a similar problem trying to enable JMX I recommend you set this value also, even if you are connecting from the local machine. Seems it is required with some versions of Tomcat.

Direct JMX access
Try adding this to catalina.sh/bat:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=5005
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
UPDATE: Alex P suggest that the following settings might also be required in some situations:
-Dcom.sun.management.jmxremote.host=localhost
This enables remote anonymous JMX connections on port 5005. You may also consider JVisualVM which is much more please and allows to browse JMX via plugin.
What you are looking for is Catalina -> ThreadPool -> http-bio-8080 -> various interesting metrics.
JMX proxy servlet
Easier method might be to use Tomcat's JMX proxy servlet under: http://localhost:8080/manager/jmxproxy. For instance try this query:
$ curl --user tomcat:tomcat http://localhost:8080/manager/jmxproxy?qry=Catalina:name=%22http-bio-8080%22,type=ThreadPool
A little bit of grepping and scripting and you can easily and remotely monitor your application. Note that tomcat:tomcat is the username/password of user having manager-jmx role in conf/tomcat-users.xml.

You can deploy jolokia.war and then retrieve mbeans values in JSON (without the manager):
http://localhost:8080/jolokia/read/Catalina:name=*,type=ThreadPool?ignoreErrors=true
If you want only some values (currentThreadsBusy, maxThreads, currentThreadCount, connectionCount):
http://localhost:8080/jolokia/read/Catalina:name=*,type=ThreadPool/currentThreadsBusy,maxThreads,currentThreadCount,connectionCount?ignoreErrors=true
{
request: {
mbean: "Catalina:name="http-nio-8080",type=ThreadPool",
attribute: [
"currentThreadsBusy",
"maxThreads",
"currentThreadCount",
"connectionCount"
],
type: "read"
},
value: {
currentThreadsBusy: 1,
connectionCount: 4,
currentThreadCount: 10,
maxThreads: 200
},
timestamp: 1490396960,
status: 200
}
Note: This example works on Tomcat7 +.

For a more enterprise solution. I have been using New Relic in our production environment.
This provides a graph of the changes to the threadpool over time.

There are cheaper tools out meanwhile: I am using this jar here: https://docs.cyclopsgroup.org/jmxterm
You can automate it via shell/batch scripts. I regexed the output and let prometheus poll it for displaying it in grafana.