We have a Spring 2.0.8 application in production, running on Tomcat 5.5.x and JRE 1.5.x (yeah, I know, we should upgrade, that's not the point now), with Oracle 11g as our choice of DB.
We have upgraded the application some months ago (I'd say July) and have switched from Oracle 10g to Oracle 11g in the past month or so (also changing the Oracle JDBC driver to match the database version).
We've been having serious and unexpected problems in production. As of a day ago, there have been heap space OutOfMemory errors several hours apart. This in turn either slows down response time by about a 100 times, or the users can't connect.
Our setup is:
Windows machine to run the server
Apache 2.2 and Tomcat 5.5 with SSO enabled, total memory: 128MB, max memory: 512MB
Spring 2.0.8 Webapp
Oracle 11g
Since noticing this error, this is what we tried:
checking out the logs - there doesn't seem to be a pattern.
Obviously, logs only tell you when the server is out of memory, so
they show the point of not working anymore, instead of the point
where the problem started
restarting server
reinstalling Tomcat
increasing amount of memory Tomcat can use - this just prolonged the issue, of course Tomcat ate just as much as we gave it
fresh installation of both the server and Apache+Tomcat
generating heap dumps - nothing spectacular seems out of the ordinary, most memory is used for starting up the application
checking the DB - it's fine, quick and responsive, no locks
I'm looking out for ideas on what else to do. We have this same setup in 5 different productions in total, this problematic one being with the smallest number of users and data.
Now that you have figured it out, I recommend that you add the following to the list of things to do REAL SOON:
Upgrade your JVM to Java 7. Java 5 has been "end-of-lifed" which means that you won't be getting any more security patches ... unless you are on an Oracle Java support contract.
If you can't upgrade to Java 7 ... or Java 6, then at least upgrade to the most recent patch release for Java 5 that you can get hold of.
Upgrade to Tomcat 6 or 7, or at least to the most recent Tomcat 5.5.
To head off problems where OutOfMemoryError causes severe slowdowns, make sure that you have the -XX:+UseGCOverheadLimit option on your JVM command line.
And, if you plan to do any significant development work on that system, consider upgrading it to Spring 3.x.
Ok, we figured this one out. It turns out it was a badly written SQL query which was scarcely used. Analyzing the heapdump helped find the objects that were taking up a lot of memory, and we went from there.
Related
I have a strange behaviour happening in one of our projects and I wonder if anyone encountered anything similar. It is a legacy Java project and we are using ojdbc6.jar to connect to an Oracle DB. Locally we are using a Tomcat 6 app server.
When using IntelliJ IDE the db connection fails in some cases (specially where higher number or more complicated queries are run), with
ORA-12516, TNS:listener could not find available handler with matching protocol stack
The strange thing is, that Eclipse users never have the mentioned problem. (For some reason, their db calls seem much faster too)
We are using an IBM Websphere JDK because of production reasons, its a 64 bit 1.8 jdk.
The issue happens more often when running debug, but also happens while running normally.
I looked up this error and tried the following things:
monitored the number of processes oracle is running, but we have a limit of 400 and it never went above 300
tried changing locally to ojdbc7 and ojdbc8: 7 produces the same issues, 8 throws a could not establish connection with Network Adapter error at the same situation
tried changing the jdk-s under tomcat: no effect
tried all kinds of memory settings for tomcat and for intellij, no effect
It's a pretty annoying issue and I dont understand, why this does not happen with Eclipse.
Thank you for your advice in advance!
My weblogic application server crashing at startup. Today I installed Oracle DB12c database, WEBLOGIC 10.3.6, RCU 11.1.1.6 and SOA generic 11. I created an empty weblogic soa domain, runned setSOADomainEnv.cmd than startWeblogic.cmd but weblogic throws me tons of warning and exceptions like :
java.lang.OutOfMemoryError: GC overhead limit exceeded
Here is the part of the log.
Some warnings here.
I reinstalled it all like 10 times with different JDKs 1.7, 1.6 etc, but still I got the same results.
I am using Windows 7 Professional.
Thank you for your answers.
You might want to try setting up the memory parameters for your JVM.
See: https://community.oracle.com/thread/2600343
Assuming your machine have enough RAM. Do the errors occur when you start the server or when you try to access a UI console (such as /em)?
I have a strange behaviour and maybe you can help me with it.
The environment is
jdk_7u40 (Tried with jdk_7u51 with same behaviour)
debian 6.0 (on windows I have never had this problem)
jboss 7.1.1
Geoserver 2.4.x (tried .3 and .4 with same result) which is based on spring framework
other war modules (not spring-based, but geoserver has some dependencies on them)
The problem is that after a couple of hours that jboss is running, when I try to login to the web interface of geoserver (a POST to the j_spring_security servlet) it took A LOT (4-5 minutes) to land to the welcome page of the application.
Using jstack, I found that there is a thread that consumes 100% of a core for all the time, with and the process keep working here
at sun.security.provider.SHA2.lf_S(SHA2.java:162)
at sun.security.provider.SHA2.lf_sigma0(SHA2.java:171)
at sun.security.provider.SHA2.implCompress(SHA2.java:225)
at sun.security.provider.SHA2.implDigest(SHA2.java:118)
at sun.security.provider.DigestBase.engineDigest(DigestBase.java:186)
at sun.security.provider.DigestBase.engineDigest(DigestBase.java:165)
at java.security.MessageDigest$Delegate.engineDigest(MessageDigest.java:576)
at java.security.MessageDigest.digest(MessageDigest.java:353)
at java.security.MessageDigest.digest(MessageDigest.java:399)
at org.jasypt.digest.StandardByteDigester.digest(StandardByteDigester.java:979)
- locked <0x00000006f8c30bb0> (a java.security.MessageDigest$Delegate)
at org.jasypt.digest.StandardByteDigester.matches(StandardByteDigester.java:1099)
at org.jasypt.digest.StandardStringDigester.matches(StandardStringDigester.java:1052)
at org.jasypt.util.password.StrongPasswordEncryptor.checkPassword(StrongPasswordEncryptor.java:99)
at org.jasypt.spring.security3.PasswordEncoder.isPasswordValid(PasswordEncoder.java:204)
at org.geoserver.security.password.AbstractGeoserverPasswordEncoder.isPasswordValid(AbstractGeoserverPasswordEncoder.java:138)
at org.geoserver.security.password.GeoServerMultiplexingPasswordEncoder.isPasswordValid(GeoServerMultiplexingPasswordEncoder.java:75)
at org.springframework.security.authentication.dao.DaoAuthenticationProvider.additionalAuthenticationChecks(DaoAuthenticationProvider.java:64)
Some of you had a similar issue?
EDIT (with workaround)
I find out that the problem is related to the CMS garbage collector and to the increase of the permgen space.
Environment
The application server is JBoss 7.1.1 with 5 war deployed in it (Geoserver and others). There are shared dependencies among all the wars (with Geoserver too); Java is running with -XX:+UseParallelOldGC -XX:SoftRefLRUPolicyMSPerMB=36000
What happens
When a full gc is performed, the permgen space is increased a lot above the used. After that, the computation of methods in sun.security.provider.SHA2.* became very slow.
How did I solve
Moving to G1GC garbage collector solved the problem for me (currently I'm using the following options -XX:+UseG1GC -XX:-UseAdaptiveSizePolicy -XX:SurvivorRatio=1 -XX:NewRatio=1 -XX:MaxTenuringThreshold=15 -XX:G1HeapRegionSize=32m )
I haven not. Can you report the issue on GeoServer own bug tracker? http://jira.codehaus.org/browse/GEOS
I know this has probably been asked many times before, but I still haven't seen an actual fix for it.
My day-to-day development environment is as follows:
1. NetBeans (latest), 2. Glassfish (latest as bundled with NB), 3. JPA, JSF, JAXB, Jersey for JAX-RS
I have about 600 classes in my project, spread across two EJB projects and one WAR project, all inside an EAR.
I am on latest JDK 7 (on OS X) and I am on an hourly basis getting the infamous "PermGen space" bug. Let's say if I am doing 3 incremental re-deploys a minute, I can only work for a short while before either:
Glassfish run out of PermGen space, so I just have to kill the process.
Deployment becomes extremely slow, due to me having increase max permgen space (as one is advised to do from dozens of answers on S.O.)
Often the only solution is to kill glassfish every 30 minute or so. It's definitely due to a bug somewhere that simply loads new classes for every new incremental re-deploy instead of getting rid of the old ones. I thought this was supposed to be fixed in JDK 7?
This has been a long standing bug in the kind of development environment, and I am rather shocked that it's still going on after my 5+ years of Java development. It's just so frustrating and incredibly unproductive.
(Just before anyone suggests increasing permgen space, believe me I've tried that, and the only thing it "solves" is to prolong the inevitable. I've seen redeployments take up to 400 seconds at its worst. Redeployment is supposed to take 5-6 seconds for a project this size, no more.)
EDIT: I ran jmap and jhat on the Glassfish process after the following steps:
Start glassfish
Deploy my EA
Undeploy my EA
Then did a heap dump with jmap
It turns out that all my classes (which should have been unloaded) are still loaded! Hopefully this is useful information to someone reading this...
Surely, that is a bug, and I don't think that there is an easy solution for that. (If there were, probably you have had it already).
What you can try: Use some hot code replacement tool for example JRebel, This way you don't have to deploy all the time, instead this tool watches the changes of the .class files (and even other web resources, if you configure so), and replaces the class definition within the running JVM. Sounds cool, right?
It works as a Java agent, it starts when your JVM starts.
There are 3 drawbacks of this solution: The deployment is a bit slower, it's harder to debug, and it's a proprietary software (but does not cost much)
When developing with Netbeans + Glassfish and using "Deploy on Save" we've found that libraries packaged within an application are not unloaded when the project is re-deployed; this causes GF to slow down and quickly run out of memory.
Try de-selecting "Package" for all compile-time libraries and place those not already in the Glassfish classpath in the domainX/lib directory.
Not sure but this may be related to GLASSFISH-17449 or GLASSFISH-16283.
I have a standalone application that is running in IBM Websphere 7.0.0.19. It is running in Java 6 and we pack an Axis2 JAR in our EAR. We have 'parent last' style class loading and we have disabled the Axis service that is packed with WAS7 by default.
Recently, after 6+ weeks of continuous functioning, the application experienced an OOM. Perplexing point is, the application is deployed seperately on 2 different machines. But only one machine went down. Second machine is still up.
We checked OS, server configuration like classloader policy using WAS console and they are similar in both machines.
When the application crashed, it generated a .phd file which we analysed using Eclipse Memory Analyser Tool (MAT). The analysis is shown in the screenshot.
If I'm correct the bootstrap class loader is repeatedly loading and holding on to references of AxisConfiguraiton and so GC is unable to collect them when it runs. But, if that is the case, then both servers must have come down. But only one server experienced an OOM. Memory allocated to JVM is same in both machines.
We are not sure whether the issue is with WAS 7 or with axis2-kernel-1.4.1.jar or with something else.
http://www.slideshare.net/leefs/axis2-client-memory-leak
https://issues.apache.org/jira/browse/AXIS2-3870
http://java.dzone.com/articles/12-year-old-bug-jdk-still-out
(Links may not refer to the current issue. But they are just pointers)
Has anyone experienced something similar ?
We saw memory growth and sockets left open on WebSphere 6.1 with Axis 2 1.4 in the past. It's been a long time, but my notes suggest it might be worth considering an upgrade to at least Axis 2 1.5.1 to fix this bug with the open sockets and also to ensure you're not creating new objects repeatedly where a singleton exists (e.g. the Service object).