I have a Tomcat Java application running on OpenShift (1 small gear) that consists of two main parts: A cron job that runs every minute, parses information from the web and saves it into a MongoDB database, and some servlets to access that data.
After deploying the app, it runs fine, but sooner or later the server will stop and I cannot access the servlets anymore (the HTTP request takes very long, and if it finishes, it returns a Proxy Error). I can only force stop the app using the rhc command line and restart it.
When I look at the jbossews.log file, I see multiple occurences of this error:
Exception in thread "http-bio-127.5.35.129-8080-Acceptor-0" java.lang.OutOfMemoryError:
unable to create new native thread
Is there anything I can do to prevent this error without needing to upgrade to a larger gear with more memory?
According your description I can understand that some memory leak issue is their with your app . That may be because that you are not stooping your threads.
Sometimes what happen is thread will not stop automatically then we need to stop the thread explicitly.
I guess, itss not a memory problem, but OS resource problem. You are running out of native threads, max threads your JVM can have.
You can increase it by this way
ulimit -s newvalue
Related
Can we automatically restart a websphere application server v6.1 on OOM exception after heap dump is created?we have an enterprise application hosted on websphere application server,recently we are facing OOM exceptions,and from time to time the app server gets automatically restarted after the heap dump is generated.But recently the app server restart is not happening automatically but has to be done manually.Can you please let me know what may be the issue
There is no in built/parameter based option in WAS 6.1 answering your question. It comes in v.7.0.
Better way I/Many follow is write a basic java program to monitor the sysout.log/ syserr.log for the particular String "OutOfMemory" or "in total in the server that may be hung". If the log has any of those string, then (i) stop the server, (ii) rotate the logs (iii)start the server.
Schedule this java program for every 2 or 5 mins.
I wont recommend this method, This is not a good practice as well. I would recommend WASADM should inform the data related team to fix the issue and providing the logs, threads, hprofs, etc..
But most of the time, it is difficult and time consuming for data/application team to fix it immediately. So WAS administrator has to follow these kind of methods.
We are trying to access an application from the tomcat which is on a different host, but it is not loading even though the tomcat is running. It was running fine for the past 3 months. We restarted the tomcat now it is working fine.
But, we could not able to zero in on what happened.
Any idea how to trace / what might have caused this?
The CPU usage was normal and the tomcat memory was 1205640.
the memory setting of tomcat are 1024- 2048(min-max)
We are using tomcat 7.
Help much appreciated....thanks in advance.....cheers!!
...also - not sure on Windows - you may be running out of file descriptors. This typically happens when streams are not properly closed in finally blocks.
In addition, check with netstat if you have a lot of sockets remaining open or accumulating in wait state.
Less likely, the application is creating threads and never releasing them.
The application is leaking something (memory, file descriptors, sockets, threads,...) and running over a limit.
There are different ways to track this. A profiler may help or more simply, running JVM dumps at regular intervals and checking what is accumulating. The excellent MAT will help you analyze the dumps.
Memory leak problems are not uncommon. If your Tomcat instance was running for three months and suddenly the contained application became unresponsive maybe that was the case. One solution (and if your resources allow you to do so) could be monitoring that Tomcat instance though JMX using jconsole to see how it behaves
I'm trying to run jWebSocket server on CentOS 5.8 (1and1 VPS). Just after start of the server and few requests from client (reloading webpage) I get this error:
Exception in thread "jWebSocket TCP-Connector 01.33719.16" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:691)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1555)
at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:546)
at java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:646)
at org.jwebsocket.tcp.TimeoutOutputStreamNIOWriter.sendPacket(TimeoutOutputStreamNIOWriter.java:215)
at org.jwebsocket.tcp.TCPConnector.sendPacket(TCPConnector.java:279)
at org.jwebsocket.server.BaseServer.sendPacket(BaseServer.java:186)
at org.jwebsocket.server.TokenServer.sendPacketData(TokenServer.java:405)
at org.jwebsocket.server.TokenServer.sendTokenData(TokenServer.java:388)
at org.jwebsocket.server.TokenServer.sendToken(TokenServer.java:312)
at org.jwebsocket.plugins.TokenPlugIn.sendToken(TokenPlugIn.java:174)
at org.jwebsocket.plugins.system.SystemPlugIn.sendWelcome(SystemPlugIn.java:397)
at org.jwebsocket.plugins.system.SystemPlugIn.connectorStarted(SystemPlugIn.java:261)
at org.jwebsocket.plugins.BasePlugInChain.connectorStarted(BasePlugInChain.java:126)
at org.jwebsocket.server.TokenServer.connectorStarted(TokenServer.java:170)
at org.jwebsocket.engines.BaseEngine.connectorStarted(BaseEngine.java:93)
at org.jwebsocket.tcp.TCPEngine.connectorStarted(TCPEngine.java:320)
at org.jwebsocket.tcp.TCPConnector$ClientProcessor.run(TCPConnector.java:502)
at java.lang.Thread.run(Thread.java:722)
But when I run jWebSocket on my computer everything is working ok. I made my own virtual server using virtualbox and CentOS 5.8 fresh install and it is working there too.
I noticed that java on 1and1 VPS uses a lot of memory ~1GB (10 times more then on my computer or virtualbox). On 1and1 VPS I have 2GB of RAM (there is an error) and on Virtualbox jWebSocket is running just fine with only 512MB of RAM.
What may be the cause of this out of memory error? Please share if you have any suggestions. I don't know what to do with this any more.
check the link: http://devgrok.blogspot.sk/2012/03/resolving-outofmemoryerror-unable-to.html
sounds like they had to increase in linux max process per user limit.
At least the similarity with your problem I see is that you get the same exception + both run linux :)
It is 1and1 (http://www.1and1.pl/) virtual hosting problem. They use Parallels Virtuozzo Containers with so called User Beancounters or UBC parameters. It is a set of limits witch was the cause of my problems. You can see this limits in /proc/user_beancounters. When one of the limits called "numproc" is reached my application can't create new threads.
EDIT
At that time jWebSocket was designed in way that it was creating one thread for every incoming token and not destroying it when this thread was no more needed.
I'm not using jWebSocket any more, I replaced it with websocket serwer written in PHP.
I have few java application running 24/7. I'd like to write another java application, which checks if any of this applications crashes or not. If some application has crashed i'd like to restart it. Is there any solutions for this?
I see some ways but there should be better ways to do it :
If you have control over your running applications you can make each of them listen to a given port on the machine and reply to any request on it. Then you can simply ping that port to see if the app is running.
If you launch the apps by yourself you can store their pid and then check if the process with that pid is still alive.
Each app can create a file and update it regularly. Then your monitoring app can check if the file has been updated. if not, then the app is dead.
What i can think is you can define a new custom exception in those 24/7 running programs, add that exception to blocks where you think crash can occur, and then call some method of the another java application when this exception is called by the crash.
You can use wmic utility.
Like this : Process proc = Runtime.getRuntime().exec("wmic.exe");
Then you may check for the different params associated with your java app (javaw.exe) application.
You can perhaps use the ( http://quartz-scheduler.org/ ) Quartz scheduler, or the java.util.Timer api to wake up recurrently.and check for the process. Ever time u run the process I believe u can get the PID or process ID which can be stored.
Wake up at regular intervals and invoke a shell sript to check if the process is running. If not then spawn new Process using Runtime.getRuntime()
If you can use a simple solution, your application must update an external resource from inside the core functionality of the application. If the external resource is not updated for a while, you know that the core of your application has stopped.
If you use a separate thread to update the resource, your core functionality will crash but the thread keeps on running and you detect nothing.
The external resource can be a file. Write the current time stamp there. If the time stamp is not updated for a while, then the app has crashed. Or write to a network socket if your monitor is on a separate machine to detect hardware crashes (a monitor is pretty useless if it crashes with your application).
Your monitor must know the process identifier of the Java process so that it can kill it and restart it when needed. you can write the process identifier into a file if you start the app from a shell script.
There exists a billion dollar industry that produces this kind of software monitors. It is not a trivial task.
I have a multi-threaded application that launches an external app to do data conversion in preparation for later parts of the application.
I have an issue that when I set my thread count higher then 6 concurrent threads, the Runtime.getRuntime().exec() fails to launch the external application (I have also tried using ProcessBuilder with the same results). It does not throw any sort of Exception, and nothing is captured on either the standard output stream or standard error stream. What's even stranger is upon rebooting the server, I can run at least 8 concurrent threads for a few minutes without the issue, but then the issue will return.
I have read that Linux's implementation uses the fork() command which can cause an unable to allocate memory error, but I am not getting any such error.
I have written another test app to just launch X number of notespad.exe's without issue with X being as large as 100.
The application is running on a Window 2003 standard x64 server in a VMware environment.
JVM version is 1.6.0_11.
I can update the JVM in an attempt to resolve the issue, but would like to leave that as a last effort to prevent needing to test all application with the new JVM version.
Try to download the dll "framedyn.dll" from http://www.dlldump.com/download-dll-files_new.php/dllfiles/F/framedyn.dll/5.1.2600.2180/download.html and paste on C:\Windows\System32.