When I tried to up the Elastic search, it gets hanged(shown in below image).It takes more than 15 minutes to start normally. However when I tried the same setup on different machines elastic search server gets up in 5 to 10 seconds and it worked fine (Except 3,those 3 machines also showed same problem). What would be the possible cause for this?
Due to this problem I got org.elasticsearch.client.transport.NoNodeAvailableException: No node available exception from java.
Note: My elasticSearch is working as standalone node on each m/c.
I got solution here.
Probable solutions:
It's always a good idea to install a current Java JRE to enjoy resolved bugs.
Check if your os network settings (routes, gateway) and mount points are configured correctly.
Remove sigar libraries in order to avoid sigar monitoring hangs
From the logs ES seems to hang before the node is going to initialize plugins, so remove all ES plugins, check the classpath (the lib directory is loaded with *) and remove unwanted/duplicate jars
Related
I have a strange behaviour happening in one of our projects and I wonder if anyone encountered anything similar. It is a legacy Java project and we are using ojdbc6.jar to connect to an Oracle DB. Locally we are using a Tomcat 6 app server.
When using IntelliJ IDE the db connection fails in some cases (specially where higher number or more complicated queries are run), with
ORA-12516, TNS:listener could not find available handler with matching protocol stack
The strange thing is, that Eclipse users never have the mentioned problem. (For some reason, their db calls seem much faster too)
We are using an IBM Websphere JDK because of production reasons, its a 64 bit 1.8 jdk.
The issue happens more often when running debug, but also happens while running normally.
I looked up this error and tried the following things:
monitored the number of processes oracle is running, but we have a limit of 400 and it never went above 300
tried changing locally to ojdbc7 and ojdbc8: 7 produces the same issues, 8 throws a could not establish connection with Network Adapter error at the same situation
tried changing the jdk-s under tomcat: no effect
tried all kinds of memory settings for tomcat and for intellij, no effect
It's a pretty annoying issue and I dont understand, why this does not happen with Eclipse.
Thank you for your advice in advance!
I was wondering if anyone has experienced this problem.
We work with Eclipse 2020-03 running Wildfly 16 and OpenJDK 1.12 on a Windows 10 Desktop connecting the a local instance of SQL Server. When the app server is deployed we connect using Chrome or IE. When I'm debugging the java app, Eclipse becomes unresponsive. It could go 2 or 3 days before this happens and then happen 10 times back to back. It is inconsistent and I cannot determine what I might being doing to cause the issue. I have worked with older versions of Eclipse and have not had this issue till the Luna version. Of course at the same time we moved to Maven and GIT as well.
Some of symptoms
1. Hang when stepping through code.
2. Hang when click away after stopped at a breakpoint and then coming back.
3. Hang when inspecting variables.
4. Hang when increase the display area of one of the variables when its a long String.
If I just run and do not interact with the debug perspective, I have no issue and it never hangs.
Funny thing is we have at least a hundred developers running with this setup and only 2 or 3 of us have had this problem. At least from what I have tried to gather in terms of information.
Things I've tried
1. Complete format and re-install of PC. (I was desparate)
2. Upgraded Eclipse from 2019-03.
3. Based on google searches, turned off mylyn
4. Logged a comment on a issue with Eclipse: https://bugs.eclipse.org/bugs/show_bug.cgi?id=322126
5. Reviewed Eclipse, Wildfly and Java logs. Nothing of note.
6. Changes startup parameters of Eclipse with no luck.
7. A few other things I cannot remember.
To recover, I have to terminate the Eclipse process and the terminate the OpenJDK process.
Any help would be appreciated.
Today in the company where I work we received a report about one of our webapps not working.
The first thing we did was look at the hardware utilization:
Processor: 5%;
Memory: 68%;
Disk IO capacity: 4%;
Network: 1Mbps/1Gbps;
After some tests we saw that, indeed, the webapp pages weren't loading, some time after the webapp timeouts.
Other webapps in the exactly same tomcat instance were working fine and fast as ever, no problem with them. We tried to restart/reload the webapp but it still didn't get it working. Finally we restarted tomcat which corrected the issue for now.
There were no restarts or redeploys since yesterday when it was working fine. We believe this may be a periodic bug of some sort so we want to correct it soon.
Does anyone know of any steps we may take to investigate what it might have been?
It seems related to lock on IO.
If you are using linux, the first thing is do is to check open files by using the command lsof . If you see a lot of opened files by your app, you have to check in the code that every InputStream/OutputStream are closed (even in exception handling code).
An other common source of issues is related to thread starvation, see http://tomcat.apache.org/tomcat-9.0-doc/config/valve.html#Stuck_Thread_Detection_Valve
As soon I click to start (literally immediately) the tomcat server with my web app, it displays the following error:
http://i67.photobucket.com/albums/h282/jamesekki/error.jpg
What's weird is the application still builds successfully and works fine. The problem is it slows down development time since I get this error every time I need to deploy any changes.
Here is my eclipse ini:
http://i67.photobucket.com/albums/h282/jamesekki/Capture.jpg
Any ideas?
Things I have tried:
1. Increased xms/xmx in increments of 128 megs all the way up to 4 gigs (still fail)
2. Increased permsize/maxpermsize similar to increments above (still fail)
Maybe these will help explain:
http://www.mulesoft.com/tomcat-oome-out-of-memory-error
http://baskarfelix.wordpress.com/2008/08/06/out-of-memory-errorjava-heap-space/
I'd also be curious to know if you have the same issue if you leave Eclipse out of the picture. Create a WAR file, add it to the Tomcat /webapps directory, and start Tomcat using a command shell and the start script in the /bin directory. If you still have a problem, I'd say you should look at how Tomcat and your JVM are configured. If you don't, it tells me that Eclipse is the problem.
Thanks everyone for your inputs. I was able to narrow it down to what was the root cause of the whole issue. I had an external api jar file in the src/main/resources/lib of my services layer and Eclipse seems to not like that for whatever reason. As soon as I moved that out of that location, it resolved the issue. Can anyone explain why this would potentially cause an issue? Just for learning purposes. Thanks!
Your settings look good to me. You could try the long way of profiling [1] tomcat to see what happens on startup, or the crude way of deleting your local server and runtime, to see if it helps with a fresh one. If that fails, re-download latest Eclipse Java EE edition, and set up from scratch.
While it's not the best solution (in case the problem resurfaces, you still have no proper solution), it is sometimes worth it if it takes too long otherwise.
[1] You could profile with JProfiler or YourKit Profiler. Both have trial versions. JVisualVM is also an alternative that comes with the JDK. However it may be an issue if Tomcat crashes immediately.
Short description of my problem: I start up Tomcat with my deployed Wicket application. When I want to shut down tomcat I get this error message:
Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at java.lang.ref.Reference.<clinit>(Reference.java:145)
I am running the following setup:
Ubuntu Linux: 10.04 (lucid) with a 2.6.18-028stab094.3 kernel
Java Version: "1.6.0_26" Java HotSpot(TM) 64-Bit Server VM
Tomcat Version: 7.0.23
jvm_args: -Xms512m -Xmx512m -XX:MaxPermSize=205m (these are added via CATALINA_OPTS, nothing else)
Wicket 1.5.1
Tomcat is configured with two virtual hosts on subdomains with ModProxy
My application is deployed as ROOT.war in the appbase directory (it makes no difference if I deploy one or both applications)
'''No application deployed does not result in OOM on shutdown''', unless I mess around with the jvm args
The size of the war is about 500k, all libraries are deployed in tomcat/common/lib (directory which I added to common.loader in conf/catalina.properties)
ulimit -u -> unlimited
When I check the Tomcat manager app it says the following about the JVM memory:
Free memory: 470.70 MB Total memory: 490.68 MB Max memory: 490.68 MB
(http connector) Max threads: 200 Current thread count: 6 Current thread busy: 1
'top' or 'free -m' is similar:
Mem: 2097152k total, 1326772k used, 770380k free, 0k buffers
20029 myuser 18 0 805m 240m 11m S 0 11.7 0:19.24 java
I tried to start jmap to get a dump of the heap, it also fails with an OutOfMemoryError. Actually as long as one or both of my applications are deployed any other java process fails with the same OOM Error (see top).
The problem occurs while the application is deployed. So something is seriously wrong with it. However the application is actually running smoothly for quite a while. But I have seen OOMs in the application as well, so I don't trust the calm.
My application is using a custom filter class? Could that be it?
For completeness (hopefully), here's the list of libraries in my common/lib:
activation-1.1.jar
antlr-2.7.6.jar
antlr-runtime-3.3.jar
asm-3.1.jar
asm-commons-3.1.jar
asm-tree-3.1.jar
c3p0-0.9.1.1.jar
commons-collections-3.1.jar
commons-email-1.2.jar
dependencies-provided.tgz
dom4j-1.6.1.jar
ejb3-persistence-1.0.2.GA.jar
geronimo-annotation_1.0_spec-1.1.1.jar
geronimo-jaspic_1.0_spec-1.0.jar
geronimo-jta_1.1_spec-1.1.1.jar
hibernate-annotations-3.4.0.GA.jar
hibernate-commons-annotations-3.1.0.GA.jar
hibernate-core-3.3.0.SP1.jar
hibernate-entitymanager-3.4.0.GA.jar
hibernate-search-3.1.0.GA.jar
javassist-3.4.GA.jar
joda-time-1.6.2.jar
jta-1.1.jar
log4j-1.2.16.jar
lombok-0.9.3.jar
lucene-core-2.4.0.jar
mail-1.4.1.jar
mysql-connector-java-5.1.14.jar
persistence-api-1.0.jar
quartz-2.1.1.jar
servlet-api-2.5.jar
slf4j-api-1.6.1.jar
slf4j-log4j12-1.6.1.jar
stringtemplate-4.0.2.jar
wicket-auth-roles-1.5.1.jar
wicket-core-1.5.1.jar
wicket-datetime-1.5.1.jar
wicket-extensions-1.5.1.jar
wicket-request-1.5.1.jar
wicket-util-1.5.1.jar
xml-apis-1.0.b2.jar
I appreciate any hint or even speculation that gives me additional ideas what to try.
Update: I tested some more and found that this behaviour only occurs while one or both of my applications are deployed. The behaviour does not occur on "empty" tomcat (that was a mistake on my part messing with jvm args)
Update2: I am currently experimenting trying to reproduce this behaviour in a virtual box, I want to debug this with a profiler. I am still not convinved that it should be impossible to run my setup on 2GB RAM.
Update3 (10/01/12): I am trying to run jenkins instead of my own application. Same behaviour, so it is definitely server configuration issues. Jenkins jobs fail when maven is called, so I need not even try the shutdown hack suggested below because I need a second java process while running Jenkins. It was suggested to me that because this is a Virtual Server ulimits may be imposed from outside and I would not be able to see them. I think I'll ask a new question regarding this. Thx all.
Update4 (02/05/12): see below for the answer that contains the hint. I'll clarify some more up here: I am now 95% sure that the errors occur because I am reaching my thread limit. However because this is a virtual server the method described below would not work to check this value because it is not visible with ulimit, that was what was confusing me and only today I found out that this is the "numproc" value that I can see in the Parallels Power Panel that I can log into for my virtual server. There were Resource Alerts for numproc but I did not see those either until just now. The value has a hard limit of 96 which I cannot change of course. The current value of numproc corresponds to the number of processes I see with "top" after toggling "H" to see threads. I had a very hard time finding this because this numproc value is hidden deep inside the panel. Sadly 96 is a rather low number if you want to run a tomcat with apache and mysql. I am also very sad that I cannot even find this value in the small print of my hosting contract and it is rather relevant to my application I dare say. So I guess I'll need a server upgrade.
Thanks all for your helpful answers in the end everyone helped me a bit to find out what the problem was.
The tomcat shutdown procedure consits of sending an command/word via a tcp port to the running tomcat VM. This port is configured in the server.xml (if I remember corretly, writting on my phone right now). So far so good.
Unfortunately, the shutdown script does this by starting a 2. VM using the same java options used for the tomcat. Your system simply has not enough memory for this.
As a sollution you could write your own stop script using telnet or something.
I could help with later if needed.
Hope that helps.
Viele grüsse Bert
Seems you have too many threads open.
Use this command :
ulimit -u
What is the result ?
Should be something like :
max user processes (-u) 100
If this is correct, you can edit this file :
/etc/security/limits.conf
and the the following modifications :
#<domain> <type> <item> <value>
user soft nproc 10000
user hard nproc 10000
You can probably survive for a while like this. All you need to do is kill the tomcat process whenever you need to restart it. It is not a nice approach, but the main concern is that your application runs correctly.
It seems to me though, that on the long run, you might need to order a hosting plan with more RAM available.
I was having a similar problem with a tomcat installation just last week. I managed to fix it by giving tomcat a smaller heap. Something like this:
export CATALINA_OPTS=”-Xms256m -Xmx512m”
Before starting Tomcat may help. In the meantime you'll have to kill it the old fashioned way, with a kill -9 ;)
EDIT: you could also take look here, it appears tomcat automatically creates a bunch of "spare" threads, but you can limit those as well as your max thread count in the config. Hope it helps.