Develop Reference

Camel creating too many tcp connections (over 15000) in TIME_WAIT status to connect ActiveMQ

I am using Apache Camel(with Spring) and ActiveMQ in project. Here are the settings related to JMS/ActiveMQ:
Camel version: activemq-camel-5.15.3.jar (all ActiveMQ related jars)
ActiveMQ version : 5.15.0
<!-- language: lang-xml -->
<bean id="defaultActiveMQRedeliveryPolicy" class="org.apache.activemq.RedeliveryPolicy">
</bean>
<util:list id="redeliveryPolicyEntries">
<bean id="activeMQRedeliveryPolicy1" class="org.apache.activemq.RedeliveryPolicy">
<property name="queue" value="inbox"></property>
</bean>
</util:list>
<bean id="amqRedeliveryPolicyMap"
class="org.apache.activemq.broker.region.policy.RedeliveryPolicyMap">
<property name="defaultEntry" ref="defaultActiveMQRedeliveryPolicy"></property>
<property name="redeliveryPolicyEntries" ref="redeliveryPolicyEntries"></property>
</bean>
<bean id="amqPrefetchPolicy" class="org.apache.activemq.ActiveMQPrefetchPolicy">
</bean>
<bean id="pooledConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory" init-method="start" destroy-method="stop">
<property name="maxConnections" value="20" />
<property name="maximumActiveSessionPerConnection" value="40" />
<property name="connectionFactory" ref="jmsConnectionFactory">
</property>
</bean>
<bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="${jmsConnectionFactory.brokerURL}" />
<property name="userName" value="admin" />
<property name="password" value="admin" />
<property name="prefetchPolicy" ref="amqPrefetchPolicy" />
<property name="redeliveryPolicyMap" ref="amqRedeliveryPolicyMap" />
</bean>
<bean id="jmsConfig" class="org.apache.camel.component.jms.JmsConfiguration">
<property name="connectionFactory" ref="pooledConnectionFactory" />
<property name="concurrentConsumers" value="15" />
<property name="maxConcurrentConsumers" value="30" />
<property name="asyncConsumer" value="false" />
<property name="cacheLevelName" value="CACHE_CONSUMER" />
</bean>
<!-- this bean actually represents a jms component to be used in our camel-integration
setup.make endpoints by using name(id) of this bean. -->
<bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="configuration" ref="jmsConfig" />
<property name="transacted" value="false" />
<property name="transactionManager">
<bean class="org.springframework.jms.connection.JmsTransactionManager">
<property name="connectionFactory" ref="jmsConnectionFactory" />
</bean>
</property>
</bean>
As you see I am using PooledConnectionFactory so I am expecting a fixed no of connections to connect with ActiveMQ. But unexpectedly I see a large no of TCP connections being opened in TIME_WAIT even when my application is idle and no messages are being produced/consumed at time. I confirmed this situation with infra team that confirmed all the Operating System level configuration are fine.
Here I tried debugging the doReceiveAndExecute method in AbstractPollingMessageListenerContainer- sessionToUse is not null, consumerToUse is also not null and code flows through receiveMessage(line number 304).I could not find the problem in debug trace as attached in debug screenshots:
and
and my actual problem
Is it a problem with MessageListenerContainer or with ConnectionFactory?? Am I missing some configuration which would prevent this from happening or is this an existing issue? If so is there a workaround?

Just spotted in your configuration that you configured the jmsConnectionFactory (not the pooled factory) in your transaction manager. Not sure if this could raise the issue because the pooled factory is simply not used.
<property name="transactionManager">
<bean class="org.springframework.jms.connection.JmsTransactionManager">
<property name="connectionFactory" ref="jmsConnectionFactory" />
</bean>
</property>

What are the steps to identify connection leak when using DBCP, JDBCTemplate and ScheduledExecutor?

We are creating a Spring boot web app.
DB : JDBC Template and DBCP connection pool.
Java code: A runnable is called in Executors.newSingleThreadScheduledExecutor();
Time interval: 2 min
The code in runnable hits DB using JDBCTemplate.query().
Issue: The heap usage increases to several GBs in few min.
Any Pointers would be helpful to identify the memory leak.
Note: If we comment the JDBCTemplate.query() , memory usage is constant.
Settings of DBCP:
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="oracle.jdbc.OracleDriver" />
<property name="url" value="${batch.jdbc.url}" />
<property name="username" value="******" />
<property name="password" value="******" />
<property name="connectionProperties" value="defaultRowPrefetch=10000;defaultBatchValue=200;" />
<property name="minIdle" value="10" />
<property name="maxIdle" value="12" />
<property name="maxActive" value="100" />
<property name="accessToUnderlyingConnectionAllowed" value="true" />
<property name="initialSize" value="${batch.jdbc.pool.size}"/>
<property name="validationQuery" value="select 1 from dual" />
<property name="validationQueryTimeout" value="5" />
<property name="timeBetweenEvictionRunsMillis" value="120000" />
<property name="minEvictableIdleTimeMillis" value="60000" />
<property name="testOnBorrow" value="true" />
</bean>
Suspect from Eclipse MAT report
One instance of "org.apache.commons.pool.impl.GenericObjectPool" loaded by "org.springframework.boot.loader.LaunchedURLClassLoader # 0x7fc1d90124c8" occupies 1,421,543,264 (94.69%) bytes. The memory is accumulated in one instance of "org.apache.commons.pool.impl.GenericObjectPool" loaded by "org.springframework.boot.loader.LaunchedURLClassLoader # 0x7fc1d90124c8".

Spring+Hibernate+java :Connecting to multiple database after running application

If application is running and connected to Database-1. Through application I want to copy certain data from one Database (it can be Database-1 on any other) to another database. There can be 3-4 database. And schema is exactly same for all the database. Is it possible to do so?
I have read about "AbstractRoutingDataSource" here. But I don't wan't to connect to database during runtime. It should be after runtime.
my spring-config.xml contains bean for JdbcTemplate
<bean id="EnvJdbcTemplate" class="org.springframework.jdbc.core.JdbcTemplate">
<property name="dataSource">
<ref bean="envDataSource" />
</property>
</bean>
I have created bean for envDataSource (database 1)
<bean id="envDataSource" class="org.apache.commons.dbcp.BasicDataSource">
<property name="driverClassName"/>
<property name="url"/>
<property name="username"/>
<property name="password"/>
</bean>

Seems like you just need to create more beans, and inject them where needed, and call them when you need. This isn't that smart, but you don't need over engineer this.
<bean id="envDataSource1" class="org.apache.commons.dbcp.BasicDataSource">
<property name="driverClassName"/>
<property name="url"/>
<property name="username"/>
<property name="password"/>
</bean>
<bean id="envDataSource2" class="org.apache.commons.dbcp.BasicDataSource">
<property name="driverClassName"/>
<property name="url"/>
<property name="username"/>
<property name="password"/>
</bean>
<bean id="EnvJdbcTemplate1" class="org.springframework.jdbc.core.JdbcTemplate">
<property name="dataSource">
<ref bean="envDataSource1" />
</property>
</bean>
<bean id="EnvJdbcTemplate2" class="org.springframework.jdbc.core.JdbcTemplate">
<property name="dataSource">
<ref bean="envDataSource2" />
</property>
</bean>

There is a built in feature in Hibernate to support multiple schemas/databases. find it here

Hibernate Too Many Connections With Cache

I'm using hibernate to try and retrieve a cached query.
#Transactional
public interface ProductDAO extends JpaRepository<Product, Long> {
#QueryHints({ #QueryHint(name = "org.hibernate.cacheable", value = "true") })
Product findByCode(String code);
}
I'm load testing and I'm doing this in a large loop of 1000 iterations.
for (int i = 0; i < 500; i++) {
URL myURL = new URL("http://localhost:8080/test");
URLConnection myURLConnection = myURL.openConnection();
myURLConnection.connect();
myURLConnection.getContent();
}
I've checked with showsql and I can see only 1 SQL statement is generated for my first hit to the DB after which it is cached.
Yet I still get the following error even though no SQL is being shown:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Too many connections
My Hibernate properties:
#hibernate properties
hibernate.dialect = ${hibernate.dialect}
hibernate.show_sql = false
hibernate.hbm2ddl.auto = ${hibernate.hbm2ddl}
hibernate.c3p0.min_size = 10
hibernate.c3p0.max_size = 100
hibernate.c3p0.timeout = 300
hibernate.c3p0.max_statements = 50
hibernate.c3p0.acquire_increment = 5
hibernate.c3p0.idle_test_period = 3000
hibernate.cache.use_second_level_cache=true
hibernate.cache.region.factory_class=org.hibernate.cache.ehcache.SingletonEhCacheRegionFactory
hibernate.cache.use_query_cache=true
Database config:
<bean name="dataSource" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="url" value="${db.url}" />
<property name="driverClassName" value="${db.driverClassName}" />
<property name="username" value="${db.username}" />
<property name="password" value="${db.password}" />
</bean>
<bean class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean" id="entityManagerFactory">
<property name="dataSource" ref="dataSource" />
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
<property name="showSql" value="false" />
<property name="generateDdl" value="true" />
<property name="databasePlatform" value="${hibernate.dialect}" />
</bean>
</property>
<property name="jpaProperties" ref="hibernateProperties" />
<property name="packagesToScan">
<array>
<value>com.exammple.model</value>
</array>
</property>
</bean>
<bean id="hibernateProperties" class="org.springframework.beans.factory.config.PropertiesFactoryBean">
<property name="location" value="classpath:/spring/hibernate.properties" />
</bean>
<bean id="sessionFactory" factory-bean="entityManagerFactory" factory-method="getSessionFactory" />
<bean id="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
<property name="dataSource" ref="dataSource" />
<property name="entityManagerFactory" ref="entityManagerFactory" />
</bean>
<tx:annotation-driven transaction-manager="transactionManager" />
<jpa:repositories base-package="com.example.dal" entity-manager-factory-ref="entityManagerFactory"
transaction-manager-ref="transactionManager" repository-impl-postfix="CustomImpl" />

The problem is your configuration you aren't using a connection pool.
You are configuring a DriverManagerDataSource which isn't a proper connection pool. You are injecting this bean into the LocalContainerEntityManagerFactoryBean which renders your hibernate.connection and hibernate.c3p0 properties useless, they aren't used.
Solution is quite easy drop the hibernate.c3p0 and hibernate.connection properties and replace the DriverManagerDataSource with a proper pool implementation. I would recommend HikariCP over C3P0 but that is personal preference.
<bean id="hikariConfig" class="com.zaxxer.hikari.HikariConfig">
<property name="poolName" value="springHikariCP" />
<property name="connectionTestQuery" value="SELECT 1" />
<property name="dataSourceClassName" value="${db.driverClassName}" />
<property name="dataSourceProperties">
<props>
<prop key="url">${db.url}</prop>
<prop key="user">${db.username}</prop>
<prop key="password">${jdb.password}</prop>
</props>
</property>
</bean>
<bean id="dataSource" class="com.zaxxer.hikari.HikariDataSource" destroy-method="close">
<constructor-arg ref="hikariConfig" />
</bean>

This problem may be have multiple issues:
You don't close your database connections.
You set a maximum connection pool pool size that exceeds the maximum connections allowed of your database server. When the number of clients exceeds the number of maximum allowed connections, you'd get an exception like this.
I could use FlexyPool to monitor the connection pool usage and find out if connections are leaking or if they are leased for long times.
Update
As M. Deinum already said, you are not using Connection Pooling.
You can still use the Hibernate C3P0 properties, but you have to remove the dataSource from the LocalContainerEntityManagerFactoryBean:
<property name="dataSource" ref="dataSource" />
This way Hibernate can use the hibernate.c3p0 properties.

Configuring apache dbcp PoolingDataSource with Spring

Tried to follow the pattern on apache dbcp examples, I understand everything except how and where the database properties come from and in which bean they have to be placed in application context.
I used Spring Data Source instead, but as I recall I configured it in hurry and I remember having difficulties with configuring the original dataSource provided by apache dbcp itself. So I happen to have time to face the problem and fulfill the original intent of using PoolingDataSource.
The reason I used Spring implementation is because it provides means of setting up the parameters to connect to database.
http://static.springsource.org/spring/docs/2.0.x/api/org/springframework/jdbc/datasource/DriverManagerDataSource.html
According to http://commons.apache.org/dbcp/apidocs/org/apache/commons/dbcp/PoolingDataSource.html
There are no methods to populate configuration like url or load driver.
I tried to track it through the object pools etc. , but got really lost.
Replying upfront: Yes, I don't want to use apache basicDataSource.
So now I am returning to the problem and can't really understand where to fetch the parameters? Driver? Url? It seems that url, pw and username are set on connection factory. But where to fetch postgresql driver to be loaded?
Please help to complete the configuration.
(using spring for configuration)
<!-- the one I want to use now -->
<bean id="dataSource" class="org.apache.commons.dbcp.PoolingDataSource">
<constructor-arg><ref bean="pool"/></constructor-arg>
</bean>
<!-- the one I used before as a workaround
<bean id="ds" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="org.postgresql.Driver"/>
<property name="url" value="jdbc:postgresql:postgres"/>
<property name="username" value="magicUserName"/>
<property name="password" value="magicPassword"/>
</bean> -->
<bean id="pool" class="org.apache.commons.pool.impl.GenericObjectPool">
<property name="minEvictableIdleTimeMillis"><value>300000</value </property>
<property name="timeBetweenEvictionRunsMillis"><value>60000</value </property>
</bean>
<bean id="dsConnectionFactory" class="org.apache.commons.dbcp.DataSourceConnectionFactory">
<constructor-arg><ref bean="dataSource"/></constructor-arg>
</bean>
<bean id="poolableConnectionFactory" class="org.apache.commons.dbcp.PoolableConnectionFactory">
<constructor-arg index="0"><ref bean="dsConnectionFactory"/ </constructor-arg>
<constructor-arg index="1"><ref bean="pool"/></constructor-arg>
<constructor-arg index="2"><null/></constructor-arg>
<constructor-arg index="3"><null/></constructor-arg>
<constructor-arg index="4"><value>false</value></constructor-arg>
<constructor-arg index="5"><value>true</value></constructor-arg>
</bean>
<bean id="txManager"
class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSource" />
</bean>
<tx:annotation-driven transaction-manager="txManager" />
</beans>
I believe we are interested just in the first two, but I included everything just in case.
Seems to be there are many people using workarounds:
http://forum.springsource.org/showthread.php?10772-How-do-I-create-a-org-apache-commons-dbcp-PoolableConnection

You can config as below:
<bean id="dataSource"
class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="url" value="<put database connection url here>" />
<property name="username" value="XXXXXX" />
<property name="password" value="XXXXXXXX" />
<property name="driverClassName" value="<database driver here>" />
</bean>
<bean id="pool" class="org.apache.commons.pool.impl.GenericObjectPool">
<property name="minEvictableIdleTimeMillis"><value>300000</value></property>
<property name="timeBetweenEvictionRunsMillis"><value>60000</value></property>
</bean>
<bean id="dsConnectionFactory" class="org.apache.commons.dbcp.DataSourceConnectionFactory">
<constructor-arg><ref bean="dataSource"/></constructor-arg>
</bean>
<bean id="poolableConnectionFactory" class="org.apache.commons.dbcp.PoolableConnectionFactory">
<constructor-arg index="0"><ref bean="dsConnectionFactory"/></constructor-arg>
<constructor-arg index="1"><ref bean="pool"/></constructor-arg>
<constructor-arg index="2"><null/></constructor-arg>
<constructor-arg index="3"><null/></constructor-arg>
<constructor-arg index="4"><value>false</value></constructor-arg>
<constructor-arg index="5"><value>true</value></constructor-arg>
</bean>
<bean id="pooledDS" class="org.apache.commons.dbcp.PoolingDataSource"
depends-on="poolableConnectionFactory">
<constructor-arg><ref bean="pool"/></constructor-arg>
</bean>
And you can use "pooledDS" (PoolingDataSource) the same any orther DataSource.
Ortherwise, i think you should simply use BacsicDataSource, you still can config number of connections in pool by "initialSize" and "maxActive":
<bean id="basicDataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="${jdbc.driverClassName}"/>
<property name="url" value="${jdbc.url}"/>
<property name="username" value="${jdbc.username}"/>
<property name="password" value="${jdbc.password}"/>
<property name="removeAbandoned" value="true"/>
<property name="initialSize" value="10" />
<property name="maxActive" value="50" />
</bean>