I am using a library where I need to get a data source and feed it into it. is there anyway I can get a connection from a connection pool? I am using Hibernate 4 with C3p0 connection pool.
here is my hibernate.cfg.xml
<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE hibernate-configuration PUBLIC
"-//Hibernate/Hibernate Configuration DTD 3.0//EN"
"http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<!-- Database connection settings -->
<property name="connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="connection.url">jdbc:mysql://localhost:3306/sampleDB</property>
<property name="connection.username">root</property>
<property name="connection.password">mypass</property>
<property name="connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
<property name="c3p0.max_size">100</property>
<property name="c3p0.min_size">1</property>
<property name="c3p0.idle_test_period">30</property>
<!-- SQL dialect -->
<property name="dialect">
org.hibernate.dialect.MySQLDialect
</property>
<!-- Shows Generated SQL Queries By Hibernate -->
<property name="show_sql">false</property>
<!-- Drop and re-create The Database Schema on Start up -->
<property name="hbm2ddl.auto">update</property>
<property name="cache.provider.class">org.hibernate.cache.NoCacheProvider</property>
</session-factory>
</hibernate-configuration>
If you are using hibernate and you want access to the c3p0 Connection pool it is already using , one easy approach would be to use the C3P0Registry class to find the DataSource, see here and here.
Probably you will find that getPooledDataSources() returns a Set containing single element, and that will be the DataSource hibernate has constructed. If you want to, you can also set the config param c3p0.dataSourceName (hibernate.c3p0.dataSourceName in hibernate config), and use C3P0Registry.pooledDataSourcesByName( dataSourceName ).
[If you will set your own name, it would probably be worth verifying that hibernate is not using the dataSourceName property itself. I don't think that it does, but I haven't checked. The easiest way to check would be to look at your logs for the pools config dump on init, and make sure that there's something like "dataSourceName -> z8kflt8uqkl8iymaxxkw|729f44" in it. If the name is a long random-ish String with a pipe in it, it's an instance-specific autogenerated identity token and you should feel free to set your own name. If you see a more sensible name, then hibernate has already set this property and may be expecting the name you see, so you should look up that name.]
Note that if you plan to use Connections from the DataSource directly, take care to ensure that all Connections get properly close()ed in finally blocks. If you "leak" Connections, i.e. if you check them out and fail to check them back in reliably, you'll eventually exhaust the pool and freeze your hibernate app.
Good luck!
Update: Example...
import java.util.Set;
import javax.sql.DataSource;
import com.mchange.v2.c3p0.C3P0Registry;
// you probably want better Exception handling than this...
private DataSource findUniqueDataSource()
{
Set set = C3P0Registry.getPooledDataSources();
int sz = set.size();
if ( sz == 1 ) // yay, just one DataSource
return (DataSource) set.iterator().next();
else
throw new RuntimeException("No unique c3p0 DataSource, found:" + sz);
}
// be sure you have configured a dataSourceName in your c3p0 or hibernate config
private DataSource findDataSourceByName( String dataSourceName )
{ return C3P0Registry.pooledDataSourceByName(dataSourceName); }
And no, you should not "leak" Connections and expect the pool to clean up after you. You can forget to close Statements and ResultSets, and the pool will take care of them when you close() the Connection, but the pool doesn't know when it's safe to snatch a Connection back from a client that has failed to close it. Some applications hold Connections open for a long time (although that's bad practice, if you are using a Connection pool).
You can force c3p0 to clean up leaked Connections after a period of time, see the config parameter unreturnedConnectionTimeout. But this is an icky strategy; I recommend if you have a leak to use this in concert with debugUnreturnedConnectionStackTraces only temporarily, to understand where you are leaking Connections and then fix the problem.
c3p0 provides connection pool for Hibernate, as The built-in Hibernate connection pool is in no way intended for production use. It lacks several features found on any decent connection pool - According to Hibernate Community Documentation, for configuration of c3p0 with Hibernate you can refer to this or MKYong tutorial on Hibernate Community.
Related
I am opening transaction using TransactionInterceptor and using HibernateTransactionManager and com.zaxxer.hikari.HikariDataSource as datasource.
I have set autocommit value for datasource and hibernate property hibernate.connection.provider_disables_autocommit values as follows
<property name="autoCommit" value="false"/>
<prop key="hibernate.connection.provider_disables_autocommit">true</prop>
Even then the transaction is getting opened much before it is required.
5.3.1.Final version of hibernate-core is used for the above.
I have used the following references.
https://github.com/spring-projects/spring-boot/issues/9261
https://vladmihalcea.com/why-you-should-always-use-hibernate-connection-provider_disables_autocommit-for-resource-local-jpa-transactions/
Please let me know in case anything else was required to be done or what is the exact meaning of delay.
Thanks in advance.
Apart from configuring autocommit as false in hibernate as you did, you also need to set autocommit to false at the connection pool level. Otherwise, it won't be effective as in your case. Say, for Hikari pool you need to configure as below:
HikariConfig hikariConfig = super.hikariConfig( dataSource );
hikariConfig.setAutoCommit( false );
I've been using c3p0 for connection pooling in my Spring Boot Application for a few months now. Everything was fine until about 2 weeks ago when I started experiencing connection issues especially in the morning. Every morning when I try to log-in to my application, it would throw a Could not open connection error. I would then restart my application in order to remove the problem. I am unable to figure out the root cause of the problem.
Here's my hibernate.cfg.xml:
hibernate-configuration>
<session-factory>
<property name="hibernate.connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="hibernate.connection.url">jdbc:mysql://localhost:3306/mydb?autoReconnect=true</property>
<property name="hibernate.connection.username">root</property>
<property name="hibernate.connection.password">abc123</property>
<property name="hibernate.dialect">config.CustomDialect</property>
<property name="connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
<property name="hibernate.c3p0.initialPoolSize">5</property>
<property name="hibernate.c3p0.minPoolSize">5</property>
<property name="hibernate.c3p0.maxPoolSize">100</property>
<property name="hibernate.c3p0.checkoutTimeout">3000</property>
<property name="hibernate.c3p0.maxStatementsPerConnection">30</property>
<property name="hibernate.c3p0.unreturnedConnectionTimeout">3000</property>
<property name="hibernate.c3p0.debugUnreturnedConnectionStackTraces">true</property>
<property name="show_sql">true</property>
<property name="format_sql">true</property>
<property name="hbm2ddl.auto">update</property>
...
POJO mappings
</session-factory>
</hibernate-configuration>
Here's my HibernateUtil Class:
public class HibernateUtil {
private static final SessionFactory sessionFactory;
static {
try {
Configuration configuration = new Configuration().configure();
StandardServiceRegistryBuilder builder = new StandardServiceRegistryBuilder()
.applySettings(configuration.getProperties());
sessionFactory = configuration.buildSessionFactory(builder.build());
} catch (Exception ex) {
throw new ExceptionInInitializerError(ex);
}
}
public Session openSession() {
return sessionFactory.openSession();
}
}
I've added c3p0 debug configuration to my application to cull unreturned connections(in-case of a memory leak) and generate a stack-trace for it but nothing shows up in the logs.
Here's some of the logs from this morning :
https://pastebin.com/MGb4Miau
Can anyone here help me figure where the problem lies?
EDIT: CustomDialect Class:
public class CustomDialect extends MySQL5InnoDBDialect {
public String getTableTypeString() {
return " ENGINE=InnoDB DEFAULT CHARSET=utf8";
}
}
This problem appear because you ran out of database connections.
too slow queries that hold the connection.
you have a lot of connection demands that can't fit the max pool size 100, it is the expected cause since the problem appear after a day of running.
or you have source leak, because you have not close the connection after ether the transaction success or failed.
From C3P0 logs on debug try to see how many connections are requested.
Also
Ideally, YOU must not use unreturnedConnectionTimeout in production, so you have to debug the connection leaks and when you have no more leaks remove
both unreturnedConnectionTimeout and debugUnreturnedConnectionStackTraces config.
Edit
try these config:
<property key="hibernate.connection.characterEncoding">UTF-8</property>
<property key="hibernate.connection.useUnicode">true</property>
Because sometimes the encoding in Hibernate is different from the encoding in MySQL db.
I finally figured out a solution to the problem. I made the following changes to my hibernate.cfg.xml:
<property name="hibernate.c3p0.maxIdleTime">10800</property>
<property name="hibernate.c3p0.maxIdleTimeExcessConnections">600</property>
And it works! The maxIdleTime property removes connections that have been idle from the pool for more than the specified time period(in seconds) which ensures that my connections are refreshed from time-to-time and maxIdleTimeExcessConnections lets me cull connections from the pool in excess of minPoolSize that have been idle for more than the specified time period(in seconds). This way I ensure that I don't have too many connections in the pool and that they all are fresh.
I am using Postgres 9.2, hibernate 4.3.0 final.
I have testClass:
#Entity
#Table(name="testClass")
public class testClass implements Serializable {
#Id
#Column(name = "id")
private Integer id;
#Column(name="name")
private String name;
public Integer getId() {
return id;
}
}
Created from another class's method:
try {
new Configuration().configure("/hibernate.cfg.xml");
new testClass();
} catch(Exception e) {
System.out.println(e);
}
Here is my hibernate.xml.cfg:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE hibernate-configuration PUBLIC "-//Hibernate/Hibernate Configuration DTD 3.0//EN"
"http://www.hibernate.org/dtd/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory name="postgres">
<property name="hibernate.connection.driver_class">org.postgresql.Driver</property>
<property name="hibernate.connection.password">123</property>
<property name="hibernate.connection.url">jdbc:postgresql://localhost:5432/postgres</property>
<property name="hibernate.connection.username">postgres</property>
<property name="hibernate.dialect">org.hibernate.dialect.PostgreSQLDialect</property>
<property name="hbm2ddl.auto">create</property>
<mapping class="testClass"/>
</session-factory>
</hibernate-configuration>
It executes on jboss server side:
[2014-01-06 05:59:01,592] Artifact server:ejb: Artifact is deployed successfully
17:59:22,880 INFO [org.hibernate.cfg.Configuration] configuring from resource: /hibernate.cfg.xml
17:59:22,881 INFO [org.hibernate.cfg.Configuration] Configuration resource: /hibernate.cfg.xml
17:59:22,889 INFO [org.hibernate.cfg.Configuration] Configured SessionFactory: postgres
But nothing happens :(
I'm checking for new tables in my PostgresDB but there's nothing.
What did I miss?
What did you expect to happen?
You create a new empty entity, then you exit.
You don't persist() the entity with an entity manager (or in Hibernate terms, save() to a Session). So it never exists as far as the database is concerned. It is just a plain Java object like any other, and gets garbage collected when the last reference to it is removed.
You need to:
Use the Configuation to produce a SessionFactory and store the SessionFactory somewhere accessible. You don't want to create this all the time, it should be created on startup. Container-managed persistence and injection can be handy here.
Obtain a Session from the SessionFactory
Pass the new object to Session.save(...), so it gets INSERTed in the DB after proper key generation, etc.
It might be a good idea to re-read a Hibernate and/or JPA tutorial to cover the basics of the object life cycle. The Getting Started Guide may be a good starting point, particularly the section on native Hibernate APIs.
Personally, if I was doing basic stuff I'd use the JPA APIs instead, though. PersistenceUnit, EntityManager, etc. See Getting started with Hibernate and JPA.
I'm trying to use a JDBC Job Store in Quartz with the following code:
DateTime dt = new DateTime().plusHours(2);
JobDetail jobDetail = new JobDetail(identifier, "group", TestJob.class);
SimpleTrigger trigger = new SimpleTrigger(identifier, dt.toDate());
trigger.setJobName(identifier);
trigger.setJobGroup("group");
quartzScheduler.addJob(jobDetail, true);
quartzScheduler.scheduleJob(trigger);
And am configuring the scheduler as follows:
<bean id="scheduler" class="org.springframework.scheduling.quartz.SchedulerFactoryBean" lazy-init="false">
<property name="autoStartup" value="true" />
<property name="waitForJobsToCompleteOnShutdown" value="false" />
<property name="dataSource" ref="schedulerDataSource" />
<property name="nonTransactionalDataSource" ref="nonTXdataSource" />
<property name="quartzProperties">
<props>
<!--Job Store -->
<prop key="org.quartz.jobStore.driverDelegateClass">
org.quartz.impl.jdbcjobstore.StdJDBCDelegate
</prop>
<prop key="org.quartz.jobStore.class">
org.quartz.impl.jdbcjobstore.JobStoreCMT
</prop>
<prop key="org.quartz.jobStore.tablePrefix">QRTZ_</prop>
</props>
</property>
</bean>
The schedulerDataSource is a standard JNDI data source, the nonTXdataSource is configured via a simple org.springframework.jdbc.datasource.DriverManagerDataSource I have specified the job store class to be: org.quartz.impl.jdbcjobstore.JobStoreCMT and was hoping that the code:
quartzScheduler.addJob(jobDetail, true);
quartzScheduler.scheduleJob(trigger);
would not commit the job to the database when the each method is called. Basically when I call addJob the job is immediately saved to the database, the scheduleJob method causes the trigger information to be immediately saved in the database as well, but this tends to happen over two separate transactions already.
There is a fair bit of subsequent logic in the code that needs to be committed to the database together with the scheduled jobs in one transactions, however no matter what I try the jobs are committed by the scheduler to the database as soon as they methods are called. I tried in various environments Testing/Tomcat/Glassfish and various configurations of data sources but to no avail.
Can somebody point me into the direction of where I am going wrong?
Thank you.
Having thought this over a bit, now i believe you can achieve this providing your own wrapping DataSource but you should not do this. I think Quartz maintains some internal state in memory that must be in sync with the database (or at least it can do so). If you rollback a transaction or otherwise modify database state not notifying Quartz about this fact, it may not work as expected.
On the other hand you can use Quartz's pausing of the jobs to achieve similar effect: you simply create new job and pause it before adding any triggers. Then, you resume it only after you commit your transaction.
---------------------- my original answer ----------------------
I think, but I'm not sure, not tried this, that you can try the following:
You need a transaction around a code that uses DataSource.getConnection internally. To achieve that you have to use data source that'd be aware of global transaction state. I suppose that JBoss application server gives you just that (even with plain data source).
JBoss comes with a transaction manager (Arjuna) and data sources wrappers (JBoss app server internal) that are at least aware of global transaction state.
Other options include Atomikos and a XA data source, but i have less experience here.
Edit: if Quartz uses explicit COMMIT or setAutocommit(true) internally, both my suggestions would not work.
When you set datasource on SchedulerFactoryBean, spring uses below class as JobStore ( extension to Quartz's JobStoreCMT )
LocalDataSourceJobStore
This supports both transactional and non-transactional DataSource access.
Please try following
Remove property org.quartz.jobStore.class [Edit : Its ignored ,anyways]
Make sure the method which does addJob / ScheduleJob is in spring managed transaction.
I'm using the Spring MVC to build a thin layer on top of a SQL Server database. When I began testing, it seems that it doesn't handle stress very well :). I'm using Apache Commons DBCP to handle connection pooling and the data source.
When I first attempted ~10-15 simultaneous connections, it used to hang and I'd have to restart the server (for dev I'm using Tomcat, but I'm gonna have to deploy on Weblogic eventually).
These are my Spring bean definitions:
<bean id="dataSource" destroy-method="close"
class="org.apache.commons.dbcp.BasicDataSource">
<property name="driverClassName" value="com.microsoft.sqlserver.jdbc.SQLServerDriver"/>
<property name="url" value="[...]"/>
<property name="username" value="[...]" />
<property name="password" value="[...]" />
</bean>
<bean id="partnerDAO" class="com.hp.gpl.JdbcPartnerDAO">
<constructor-arg ref="dataSource"/>
</bean>
<!-- + other beans -->
And this is how I use them:
// in the DAO
public JdbcPartnerDAO(DataSource dataSource) {
jdbcTemplate = new JdbcTemplate(dataSource);
}
// in the controller
#Autowired
private PartnerDAO partnerDAO;
// in the controller method
Collection<Partner> partners = partnerDAO.getPartners(...);
After reading around a little bit, I found the maxWait, maxActive and maxIdle properties for the BasicDataSource (from GenericObjectPool). Here comes the problem. I'm not sure how I should set them, performance-wise. From what I know, Spring should be managing my connections so I shouldn't have to worry about releasing them.
<bean id="dataSource" destroy-method="close"
class="org.apache.commons.dbcp.BasicDataSource">
<property name="driverClassName" value="com.microsoft.sqlserver.jdbc.SQLServerDriver"/>
<property name="url" value="[...]"/>
<property name="username" value="[...]" />
<property name="password" value="[...]" />
<property name="maxWait" value="30" />
<property name="maxIdle" value="-1" />
<property name="maxActive" value="-1" />
</bean>
First, I set maxWait, so that it wouldn't hang and instead throw an exception when no connection was available from the pool. The exception message was:
Could not get JDBC Connection; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
There are some long-running queries, but the exception was thrown regardless of the query complexity.
Then, I set maxActive and maxIdle so that it wouldn't throw the exceptions in the first place. The default values are 8 for maxActive and maxIdle (I don't understand why); if I set them to -1 there are no more exceptions thrown and everything seems to work fine.
Considering that this app should support a large number of concurrent requests is it ok to leave these settings to infinite? Will Spring actually manage my connections, considering the errors I was receiving? Should I switch to C3P0 considering it's kinda dead?
DBCP maxWait parameter should be defined in milliseconds. 30 ms is very low value, consider increasing it to 30000 ms and try again.
As you already found out, the default dbcp connection pool is 8 connections, so if you want to run 9 simultaneous queries one of them will be blocked. I suggest you connect to your database and run exec sp_who2 which will show you what is connected, and active, and whether any queries are being blocked. You can then confirm whether the issue is on the db or in your code.
As long as you are using Spring's JdbcTemplate family of objects your connections will be managed as you expect, and if you want to use a raw DataSource make sure you use DataSourceUtils to obtain a Connection.
One other suggestion - prior to Spring 3, don't ever using JdbcTemplate, stick to SimpleJdbcTemplate, you can still access the same methods using SimpleJdbcTemplate.getJdbcOperations(), but you should find yourself writing much nicer code using generics, and remove the need to ever create JdbcTemplate/NamedParameterJdbcTemplate instances.
Let's change the perspective.
but the exception was thrown
regardless of the query complexity
It could be because the table or the records in the table, which you are querying against has been locked (by some other active transaction) and hence it times out.
Try running the same query from SQLServer Client and if it takes a long time, then you can be sure that it is the table or record lock that is causing this.