Develop Reference

ORA-12516 Issue with ojdbc in Intellij

I have a strange behaviour happening in one of our projects and I wonder if anyone encountered anything similar. It is a legacy Java project and we are using ojdbc6.jar to connect to an Oracle DB. Locally we are using a Tomcat 6 app server.
When using IntelliJ IDE the db connection fails in some cases (specially where higher number or more complicated queries are run), with
ORA-12516, TNS:listener could not find available handler with matching protocol stack
The strange thing is, that Eclipse users never have the mentioned problem. (For some reason, their db calls seem much faster too)
We are using an IBM Websphere JDK because of production reasons, its a 64 bit 1.8 jdk.
The issue happens more often when running debug, but also happens while running normally.
I looked up this error and tried the following things:
monitored the number of processes oracle is running, but we have a limit of 400 and it never went above 300
tried changing locally to ojdbc7 and ojdbc8: 7 produces the same issues, 8 throws a could not establish connection with Network Adapter error at the same situation
tried changing the jdk-s under tomcat: no effect
tried all kinds of memory settings for tomcat and for intellij, no effect
It's a pretty annoying issue and I dont understand, why this does not happen with Eclipse.
Thank you for your advice in advance!

sun.security.provider.SHA2 use 100% cpu and hangs for 5 minutes after a while

I have a strange behaviour and maybe you can help me with it.
The environment is
jdk_7u40 (Tried with jdk_7u51 with same behaviour)
debian 6.0 (on windows I have never had this problem)
jboss 7.1.1
Geoserver 2.4.x (tried .3 and .4 with same result) which is based on spring framework
other war modules (not spring-based, but geoserver has some dependencies on them)
The problem is that after a couple of hours that jboss is running, when I try to login to the web interface of geoserver (a POST to the j_spring_security servlet) it took A LOT (4-5 minutes) to land to the welcome page of the application.
Using jstack, I found that there is a thread that consumes 100% of a core for all the time, with and the process keep working here
at sun.security.provider.SHA2.lf_S(SHA2.java:162)
at sun.security.provider.SHA2.lf_sigma0(SHA2.java:171)
at sun.security.provider.SHA2.implCompress(SHA2.java:225)
at sun.security.provider.SHA2.implDigest(SHA2.java:118)
at sun.security.provider.DigestBase.engineDigest(DigestBase.java:186)
at sun.security.provider.DigestBase.engineDigest(DigestBase.java:165)
at java.security.MessageDigest$Delegate.engineDigest(MessageDigest.java:576)
at java.security.MessageDigest.digest(MessageDigest.java:353)
at java.security.MessageDigest.digest(MessageDigest.java:399)
at org.jasypt.digest.StandardByteDigester.digest(StandardByteDigester.java:979)
- locked <0x00000006f8c30bb0> (a java.security.MessageDigest$Delegate)
at org.jasypt.digest.StandardByteDigester.matches(StandardByteDigester.java:1099)
at org.jasypt.digest.StandardStringDigester.matches(StandardStringDigester.java:1052)
at org.jasypt.util.password.StrongPasswordEncryptor.checkPassword(StrongPasswordEncryptor.java:99)
at org.jasypt.spring.security3.PasswordEncoder.isPasswordValid(PasswordEncoder.java:204)
at org.geoserver.security.password.AbstractGeoserverPasswordEncoder.isPasswordValid(AbstractGeoserverPasswordEncoder.java:138)
at org.geoserver.security.password.GeoServerMultiplexingPasswordEncoder.isPasswordValid(GeoServerMultiplexingPasswordEncoder.java:75)
at org.springframework.security.authentication.dao.DaoAuthenticationProvider.additionalAuthenticationChecks(DaoAuthenticationProvider.java:64)
Some of you had a similar issue?
EDIT (with workaround)
I find out that the problem is related to the CMS garbage collector and to the increase of the permgen space.
Environment
The application server is JBoss 7.1.1 with 5 war deployed in it (Geoserver and others). There are shared dependencies among all the wars (with Geoserver too); Java is running with -XX:+UseParallelOldGC -XX:SoftRefLRUPolicyMSPerMB=36000
What happens
When a full gc is performed, the permgen space is increased a lot above the used. After that, the computation of methods in sun.security.provider.SHA2.* became very slow.
How did I solve
Moving to G1GC garbage collector solved the problem for me (currently I'm using the following options -XX:+UseG1GC -XX:-UseAdaptiveSizePolicy -XX:SurvivorRatio=1 -XX:NewRatio=1 -XX:MaxTenuringThreshold=15 -XX:G1HeapRegionSize=32m )

I haven not. Can you report the issue on GeoServer own bug tracker? http://jira.codehaus.org/browse/GEOS

Java memory leak while using Axis2 and WAS 7

I have a standalone application that is running in IBM Websphere 7.0.0.19. It is running in Java 6 and we pack an Axis2 JAR in our EAR. We have 'parent last' style class loading and we have disabled the Axis service that is packed with WAS7 by default.
Recently, after 6+ weeks of continuous functioning, the application experienced an OOM. Perplexing point is, the application is deployed seperately on 2 different machines. But only one machine went down. Second machine is still up.
We checked OS, server configuration like classloader policy using WAS console and they are similar in both machines.
When the application crashed, it generated a .phd file which we analysed using Eclipse Memory Analyser Tool (MAT). The analysis is shown in the screenshot.
If I'm correct the bootstrap class loader is repeatedly loading and holding on to references of AxisConfiguraiton and so GC is unable to collect them when it runs. But, if that is the case, then both servers must have come down. But only one server experienced an OOM. Memory allocated to JVM is same in both machines.
We are not sure whether the issue is with WAS 7 or with axis2-kernel-1.4.1.jar or with something else.
http://www.slideshare.net/leefs/axis2-client-memory-leak
https://issues.apache.org/jira/browse/AXIS2-3870
http://java.dzone.com/articles/12-year-old-bug-jdk-still-out
(Links may not refer to the current issue. But they are just pointers)
Has anyone experienced something similar ?

We saw memory growth and sockets left open on WebSphere 6.1 with Axis 2 1.4 in the past. It's been a long time, but my notes suggest it might be worth considering an upgrade to at least Axis 2 1.5.1 to fix this bug with the open sockets and also to ensure you're not creating new objects repeatedly where a singleton exists (e.g. the Service object).

Sudden Java Heap Space errors on Tomcat 5.5

We have a Spring 2.0.8 application in production, running on Tomcat 5.5.x and JRE 1.5.x (yeah, I know, we should upgrade, that's not the point now), with Oracle 11g as our choice of DB.
We have upgraded the application some months ago (I'd say July) and have switched from Oracle 10g to Oracle 11g in the past month or so (also changing the Oracle JDBC driver to match the database version).
We've been having serious and unexpected problems in production. As of a day ago, there have been heap space OutOfMemory errors several hours apart. This in turn either slows down response time by about a 100 times, or the users can't connect.
Our setup is:
Windows machine to run the server
Apache 2.2 and Tomcat 5.5 with SSO enabled, total memory: 128MB, max memory: 512MB
Spring 2.0.8 Webapp
Oracle 11g
Since noticing this error, this is what we tried:
checking out the logs - there doesn't seem to be a pattern.
Obviously, logs only tell you when the server is out of memory, so
they show the point of not working anymore, instead of the point
where the problem started
restarting server
reinstalling Tomcat
increasing amount of memory Tomcat can use - this just prolonged the issue, of course Tomcat ate just as much as we gave it
fresh installation of both the server and Apache+Tomcat
generating heap dumps - nothing spectacular seems out of the ordinary, most memory is used for starting up the application
checking the DB - it's fine, quick and responsive, no locks
I'm looking out for ideas on what else to do. We have this same setup in 5 different productions in total, this problematic one being with the smallest number of users and data.

Now that you have figured it out, I recommend that you add the following to the list of things to do REAL SOON:
Upgrade your JVM to Java 7. Java 5 has been "end-of-lifed" which means that you won't be getting any more security patches ... unless you are on an Oracle Java support contract.
If you can't upgrade to Java 7 ... or Java 6, then at least upgrade to the most recent patch release for Java 5 that you can get hold of.
Upgrade to Tomcat 6 or 7, or at least to the most recent Tomcat 5.5.
To head off problems where OutOfMemoryError causes severe slowdowns, make sure that you have the -XX:+UseGCOverheadLimit option on your JVM command line.
And, if you plan to do any significant development work on that system, consider upgrading it to Spring 3.x.

Ok, we figured this one out. It turns out it was a badly written SQL query which was scarcely used. Analyzing the heapdump helped find the objects that were taking up a lot of memory, and we went from there.

Lightweight WebLogic development environment

We just started a project in ADF, Oracle's Java EE framework. Usually we develop in Tomcat, then deploy into the production WebLogic servers.
But we realized ADF requires a WebLogic server installed locally to develop in Eclipse/JDeveloper. This is really heavyweight, even configured as dev, it is killing our machines, and taking a long time to deploy.
So, are there any configuration parameters we could tweak so WebLogic takes as few resources as possible? Are there any alternatives we could use?
We are mostly concerned about memory (it is taking a wonderful 800MB,) and startup time (~2 minutes)

On the memory consumption issue, you might want to try setting the memory parameters of the JVM used by your WebLogic server. Log in to your WL Web Admin Console and go to Environment/Servers/[your server]/Configuration/Server Start and, on the "Arguments", setting something like -Xms256m -Xmx256m will set your JVM's initial (Xms) and maximum (Xmx) heap size to 256 megabytes. You will want to play around with these numbers and find the best values for your environment. But please be aware that your Eclipse instance might be consuming a lot of memory as well.
Regarding the startup time, although a bit larger than I would expect, they seem OK. This problem is very frequent, and I don't think you will be able to definitely solve it. WebLogic has much more features than Tomcat, and this reflects in other characteristics of the environment (like startup time). You might find some useful tips here, though: Speed up Weblogic Server startup times

There is no restriction to use a local server for development. You can always go for a shared server and deploy your projects for testing.

I agree with the previous post. If memory is a concern then you may install a stand alone WLS and deploy to it from an EAR. Make sure for the stand alone WLS you install the Application Development Runtime Libraries as by default WLS does not have domains enabled for ADF - http://www.oracle.com/technetwork/developer-tools/adf/downloads/index.html
When you install WLS, make sure you install it in development mode (which is setting the embedded WLS has too)
Beside of this, there is not much you can do to start WLS withlimited functionality.
Frank