Develop Reference

Server Stop responding because of [Pool-Cleaner]:Tomcat Connection Pool but has failed to stop it. This is very likely to create a memory leak

Problem:
org.apache.catalina.loader.WebappClassLoader - The web application [/…] appears to have started a thread named [[Pool-Cleaner]:Tomcat Connection Pool[...] but has failed to stop it. This is very likely to create a memory leak.
I have few application deployed in a tomcat server(tomcat-7.0.23) with jdk1.6.0_45 and the server i register with apache webserver. After staring the server it is working fine until i get the above error for all the application separately all of a sudden and after that my tomcat giving 404 for all request but the tomcat process is still up and in browser it shows "FILE NOT FOUND".
we use Tomcat connection pool with following configuration:
<Resource
name="Project1"
auth="Container"
type="javax.sql.DataSource"
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
url="DatabaseName"
username="_username"
password="_Password"
driverClassName="com.microsoft.sqlserver.jdbc.SQLServerDriver"
initialSize="40"
maxActive="300"
maxIdle="250"
minIdle="90"
maxWait="10000"
minEvictableIdleTimeMillis="300000"
timeBetweenEvictionRunsMillis="120000"
validationInterval="30000"
validationQuery="SELECT 1"
testOnBorrow="true"
logAbandoned="true"
removeAbandoned="true"
removeAbandonedTimeout="60"
abandonWhenPercentageFull="50"
jdbcInterceptors="ResetAbandonedTimer"
/>
And the configuration is in context.xml file.

That memory leak warning is correct. It refers to a known issue in Tomcat 7.0.x that was fixed a few weeks ago (r1744702) and will be included in 7.0.70 onwards.

Tomcat validates connection on every borrow

I am using Tomcat 7 (jdk 1.6) in Eclipse 4.3.2.
I configured my Connection Pool as below :
<Resource name="jdbc/myDS" auth="Container" type="javax.sql.DataSource"
driverClassName="com.p6spy.engine.spy.P6SpyDriver"
url="jdbc:p6spy:oracle:thin:#server:1521:XXX"
username="XXX" password="XXX" maxActive="2" maxIdle="2" maxWait="-1"
validationInterval="30000" validationQuery="SELECT 1 FROM DUAL"
/>
I am using Spring 3.2.14, Hibernate 3.2.6-GA, CXF 2.7.
Every time I receive a SOAP request, I saw in P6SPY logs that the validation query is run independently of validationInterval and its description https://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html.
I was expecting the connections to be validated at most once every 30 seconds.
Is there anything wrong with my configuration, or is this a known bug ?

The explanation is pretty simple, I did not read correctly the documentation, I need to set the factory to org.apache.tomcat.jdbc.pool.DataSourceFactory in order to use the "Tomcat High-concurrency connection pool".
After that all parameters work as expected :
<Resource
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
name="jdbc/myDS" auth="Container" type="javax.sql.DataSource"
driverClassName="com.p6spy.engine.spy.P6SpyDriver"
url="jdbc:p6spy:oracle:thin:#server:1521:XXX"
username="XXX" password="XXX" maxActive="2" maxIdle="2" maxWait="-1"
testOnBorrow="true"
testWhileIdle="true"
timeBetweenEvictionRunsMillis="10000"
validationInterval="30000"
validationQuery="SELECT 1 FROM DUAL"
/>
The connections are validated at most every validationInterval. An evictionThread runs every timeBetweenEvictionRunsMillis and validates idle connection (I choose to do this in order to spare time on connection borrow).

well, that might be because you set the testOnBorrow parameter to true. this is taken from the documentation link you give.
The indication of whether objects will be validated before being borrowed from the pool
so, I think you might want to set it to false

Too few connections from Tomcat Application

I have a mysql database configured with a max_connections value of 150. I also have a Java 6 web application running in Tomcat 5.5 configured with the following setup:
<Resource name="jdbc/myDB"
type="javax.sql.DataSource"
driver="com.mysql.jdbc.Driver"
username="username"
password="password"
maxActive="100"
maxIdle="100"
maxWait="-1"
removeAbandoned="true"
removeAbandonedTimeout="300"
logAbandoned="true"
url="jdbc:mysql://localhost:3306/myDB?autoreconnect=true"
validationQuery="SELECT 1" />
This application is not using any 3rd party framework just basic java servlets. I have a bug in some code in the java app that is not properly releasing opened mysql connections from the pool. I am working on identifying and fixing these. But in the meantime I need to figure out why at most there is only 25 connections being allowed to mysql. After these 25 connections are used up, the application becomes unresponsive.
Can someone please help me figure out why both mysql and tomcat are configured for 100+ connections but it is only allowing 25 at a time?

Tomcat JDBC Connection Pool
What connection pool do you use?
Do you use the Tomcat JDBC Connection Pool, rather than the Apache Commons pool? It has properties to detect connection leaks or abandon connection that are open for a long time than the configured timeout.

MySQL's max_connections was set to 150 but the max_user_connections was set to 25 which was the limiting factor here. I removed this setting from my.cnf to restore it to the default value of unlimited.

Connection pool in tomcat 7

Here is my current config
<Resource
name="jdbc/data"
auth="Container"
type="javax.sql.DataSource"
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/TABLE_NAME"
username="USER_NAME"
password="PASSWORD"
initialSize="10"
maxActive="50"
suspectTimeout="120"
minIdle="10"
maxIdle="20"
maxWait="1000"
testOnBorrow="true"
timeBetweenEvictionRunsMillis="30000"
minEvictableIdleTimeMillis="60000"
validationQuery="SELECT 1 FROM DUAL"
validationInterval="40"
removeAbandoned="true"
removeAbandonedTimeout="100"
/>
This is in global context so multiple apps can use it.
I am little confused about parameters.need some details.
What I understand is
initalSize a number of connection created when a pool started.
maxActive maximum 50 connections can active at a time.
minIdle 10 connections remain Idle when connection is not used else are closed after maxwait
maxIdle 20 connections can be store as idle.
But When I start tomcat server I can see a 30 IDLE connections which remains forever.Why this happens? Am I missing something ? According to my understanding about connection pool there should only 10 connections should created and can stay in IDLE mode. Is there any specific changes that I have to do with mysql my.cnf

When you say...
This is in global context so multiple apps can use it.
What specifically do you mean? Is it in $CATALINA_BASE/conf/server.xml in the GlobalNamingResources block or in $CATALINA_BASE/conf/context.xml?
Defining a Resource tag in the GlobalNamingResources block of $CATALINA_BASE/conf/server.xml will cause only one resource to be created across the entire server. This can then be shared to applications deployed on your system by adding a ResourceLink tag to the Context configuration.
Defining a Resource in $CATALINA_BASE/conf/context.xml will define the resource once for each application deployed to your Tomcat instance. Thus if you have three applications deployed, you'll end up with three separate resources. This is a guess, but probably why you are seeing 30 connections to your database server.

attempt to reconnect jdbc pool datasource after database restarts

I have a web-app with a Java back-end that uses Tomcat jdbc-pool for database connections. This works fine.
However I am trying to foolproof it before exporting it to other locations, and recently a scenario occurred where someone restarted the SQL Server database service but did not restart the Tomcat service. This caused a SQLException: java.sql.SQLException: I/O Error: Connection reset by peer: socket write error until I restarted Tomcat, forcing the jdbc-pool datasource to reconnect.
I looked for some kind of a configuration in the Tomcat jdbc-pool docs to tell the datasource to attempt to reconnect but I couldn't find anything.
Does anyone know if there is some kind of configuration for this or should I check this condition before each request?

Not 100% sure if this is your problem but on http://www.tomcatexpert.com/blog/2010/04/01/configuring-jdbc-pool-high-concurrency it says you can use testOnBorrow with a validationQuery.
<Resource type="javax.sql.DataSource"
...
testOnBorrow="true"
validationQuery="SELECT 1"
removeAbandoned="true"
/>

While checking for the same issue I came across this post which has the auto connect configurations for all app servers.
Below are the configuration which I used for auto connect in tomcat for reference.
<Resource auth="Container"
driverClassName="oracle.jdbc.OracleDriver"
initialSize="5"
maxActive="120"
maxIdle="5"
maxWait="5000"
name="jdbc/oracle/myds"
password="secret"
poolPreparedStatements="true"
type="javax.sql.DataSource"
url="jdbc:oracle:thin:#DBHOSTNAME:1521/ServiceName"
username="testuser"
validationQuery="select 1 from tab"
testOnBorrow="true"/>
Complete auto connect configurations for all app servers can be found here in Datasource autoreconnect in Java Application Servers.

Just to add on to Natan Cox's answer
Reference - http://tomcat.apache.org/tomcat-8.0-doc/jdbc-pool.html#Common_Attributes
<Resource type="javax.sql.DataSource"
...
testOnBorrow="true"
validationQuery="SELECT 1"
removeAbandoned="true"
/>
As against Geronimo, I would still like to use validationQuery
Database validationQuery notes
hsqldb - select 1 from INFORMATION_SCHEMA.SYSTEM_USERS
Oracle - select 1 from dual
DB2 - select 1 from sysibm.sysdummy1
mysql - select 1
microsoft SQL Server - select 1
postgresql - select 1
ingres - select 1
derby - values 1
H2 - select 1
Firebird - select 1 from rdb$database
Reference - DBCP - validationQuery for different Databases