I am doing some big queries on my database with Hibernate and I sometimes hit timeouts. I would like to avoid setting the timeout manually on every Query or Criteria.
Is there any property I can give to my Hibernate configuration that would set an acceptable default for all queries I run?
If not, how can I set a default timeout value on Hibernate queries?
JPA 2 defines the javax.persistence.query.timeout hint to specify default timeout in milliseconds. Hibernate 3.5 (currently still in beta) will support this hint.
See also https://hibernate.atlassian.net/browse/HHH-4662
JDBC has this mechanism named Query Timeout, you can invoke setQueryTime method of java.sql.Statement object to enable this setting.
Hibernate cannot do this in unified way.
If your application retrive JDBC connection vi java.sql.DataSource, the question can be resolved easily.
we can create a DateSourceWrapper to proxy Connnection which do setQueryTimeout for every Statement it created.
The example code is easy to read, I use some spring util classes to help this.
public class QueryTimeoutConfiguredDataSource extends DelegatingDataSource {
private int queryTimeout;
public QueryTimeoutConfiguredDataSource(DataSource dataSource) {
super(dataSource);
}
// override this method to proxy created connection
#Override
public Connection getConnection() throws SQLException {
return proxyWithQueryTimeout(super.getConnection());
}
// override this method to proxy created connection
#Override
public Connection getConnection(String username, String password) throws SQLException {
return proxyWithQueryTimeout(super.getConnection(username, password));
}
private Connection proxyWithQueryTimeout(final Connection connection) {
return proxy(connection, new InvocationHandler() {
//All the Statement instances are created here, we can do something
//If the return is instance of Statement object, we set query timeout to it
#Override
public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
Object object = method.invoke(connection, args);
if (object instanceof Statement) {
((Statement) object).setQueryTimeout(queryTimeout);
}
return object;
});
}
private Connection proxy(Connection connection, InvocationHandler invocationHandler) {
return (Connection) Proxy.newProxyInstance(
connection.getClass().getClassLoader(),
ClassUtils.getAllInterfaces(connection),
invocationHandler);
}
public void setQueryTimeout(int queryTimeout) {
this.queryTimeout = queryTimeout;
}
}
Now we can use this QueryTimeoutConfiguredDataSource to wrapper your exists DataSource to set Query Timeout for every Statement transparently!
Spring config file:
<bean id="sessionFactory" class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean">
<property name="dataSource">
<bean class="com.stackoverflow.QueryTimeoutConfiguredDataSource">
<constructor-arg ref="dataSource"/>
<property name="queryTimeout" value="1" />
</bean>
</property>
</bean>
Here are a few ways:
Use a factory or base class method to create all queries and set the timeout before returning the Query object
Create your own version of org.hibernate.loader.Loader and set the timeout in doQuery
Use AOP, e.g. Spring, to return a proxy for Session; add advice to it that wraps the createQuery method and sets the timeout on the Query object before returning it
Yes, you can do that.
As I explained in this article, all you need to do is to pass the JPA query hint as a global property:
<property
name="javax.persistence.query.timeout"
value="1000"
/>
Now, when executing a JPQL query that will timeout after 1 second:
List<Post> posts = entityManager
.createQuery(
"select p " +
"from Post p " +
"where function('1 >= ALL ( SELECT 1 FROM pg_locks, pg_sleep(2) ) --',) is ''", Post.class)
.getResultList();
Hibernate will throw a query timeout exception:
SELECT p.id AS id1_0_,
p.title AS title2_0_
FROM post p
WHERE 1 >= ALL (
SELECT 1
FROM pg_locks, pg_sleep(2)
) --()=''
-- SQL Error: 0, SQLState: 57014
-- ERROR: canceling statement due to user request
For more details about setting a timeout interval for Hibernate queries, check out this article.
For setting global timeout values at query level - Add the below to config file.
<bean id="jdbcTemplate" class="org.springframework.jdbc.core.JdbcTemplate">
<property name="dataSource" ref="dataSource"></property>
<property name="queryTimeout" value="60"></property>
</bean>
For setting global timeout values at transaction(INSERT/UPDATE) level - Add the below to config file.
<bean id="txManager" class="org.springframework.orm.jpa.JpaTransactionManager">
<property name="entityManagerFactory" ref="myEmf" />
<property name="dataSource" ref="dataSource" />
<property name="defaultTimeout" value="60" />
<property name="jpaDialect">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaDialect" />
</property>
</bean>
Related
I am new to spring and working on a sample program using Spring jdbc. this is to check how spring #Trsactional working and rolling back the changes to the Db if there is an exception.
But I am not able to achieve this. Through I am raising an exception in one of the DB update, still it's inserting the data to DB and not rolling back.
I know somewhere I am making mistake but not able to figure it out. Not sure if this is a correct approach.
What I am doing :-
in main methis I am calling load methos of Global class (which has jdbcTemplate as satic member as I will this jdbcTemplate to all other classes)
Global class load methos will initiate the bean using ApplicationContext.
Creating Dbclass instance in main method and sending the jdbcTemplate as parameter.
4.creating some sample data and calling executeDb method.
5.execute DB method will create the instance of other Dbclasss and setting the jdbcTemplate which earlier I initialized using bean in main method (I have separate class for each operation - like createuser, UpdataBalance etc)
then it will call the db opration method to insert data (I am using batchupdate)
EDIT - Removed all try-catch
DB opration code:-
#Transactional(rollbackFor={Exception.class})
public void executeDB(int count) throws Exception
{
CreateAccount newacc = new CreateAccount(jdbcTemplate);
CreateUser newusr = new CreateUser(jdbcTemplate);
//BalanceUpdate newbal = new BalanceUpdate(jdbcTemplate);
newacc.addList(acclist);
newusr.addToList(usrlist);
//newbal.addList(ballist);
newusr.execute(); // insert data to db
newacc.addAccount(); // insert data to db
//newbal.addBalance(); // insert data to db
newacc.getAccList().clear();
newusr.getUserList().clear();
//newbal.getBalanceList().clear();
if(count == 5000)
{
Thread.sleep(1000);
throw new Exception("Rollback");
}
count += 1000;
//throw new Exception();
}
<!-- begin snippet: js hide: false console: true babel: false -->
<!-- language: lang-xml -->
<context:component-scan base-package="com.example"></context:component-scan>
<tx:annotation-driven transaction-manager="transactionManager" proxy-target-class="true"/>
<bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSource" />
</bean>
<bean id="dataSource" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="oracle.jdbc.driver.OracleDriver"/>
<property name="url" value="jdbc:oracle:thin:#localhost:1521:xe"/>
<property name="username" value="system"/>
<property name="password" value="root"/>
</bean>
<bean id="jdbcTemplate" class="org.springframework.jdbc.core.JdbcTemplate">
<property name="dataSource" ref="dataSource"></property>
</bean>
<bean id="startit" class="com.example.springtransaction.GlobalClass">
<property name="jdbcTemplate" ref="jdbcTemplate"></property>
</bean>
<bean id="dbupdate" class="com.example.springtransaction.DbUpdate">
<property name="jdbcTemplate" ref="jdbcTemplate"></property>
</bean>
You need to throw exception from your method not silently log it in catch block.
And for checked exceptions you need to use #Transactional(rollbackFor = {Exception.class}).
http://www.logicbig.com/tutorials/spring-framework/spring-data-access-with-jdbc/transactional-roll-back/
https://www.catalysts.cc/en/wissenswertes/spring-transactional-rollback-on-checked-exceptions/
You should remove the try - catch and define that the method throws an Exception. Something like that
#Transactional(rollbackFor={Exception.class})
public void executeDB() throws Exception
{
if(usrlist.size() >= 5)
{
CreateAccount newacc = new CreateAccount(jdbcTemplate);
CreateUser newusr = new CreateUser(jdbcTemplate);
BalanceUpdate newbal = new BalanceUpdate(jdbcTemplate);
newacc.addList(acclist);
newusr.addToList(usrlist);
newbal.addList(ballist);
newusr.execute(); // insert data to db
newacc.addAccount(); // insert data to db
newbal.addBalance(); // insert data to db - raise exception here
}
}
Update
The class that contain the executeDB() method should be a #Component and inject that component in the main class.
Not create a new Dbclass() instance by your own.
In high-level the reason is that the Spring creates proxy classes upon injection for classes that declare #Transactional.
You could read more about Aspect-Oriented Programming here.
Current Situation
I am doing stress load tests using Jmeter on web and rest api server, but some of transations' response time delays a lot so I am using Spring Aspect to get method processing time. What I couldn't set up is that some procedure calls take too much time, so trying to check DB process time(get con, release con, pure db process time) by write logs with the certain transaction. JMX is not an option as I can't track transactions using it. I just want to leave DB pool status with ThreadContext marked on so that I can check the slow transaction and DB pool status at the same time.
Using DB datasource from Tomcat is not considered here as not want to go DB settings within project files.
Using datasource within Spring project is not an option I am considering at the moment.
Current Setup
Spring project's transaction manager uses Tomcat DBCP pool with Oracle datasource(oracle.jdbc.OracleDriver with javax.sql.DataSource)
applicationContext.xml - DB Setting
<bean id="dataSource" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" value="java:/comp/env/jdbc/svc"/>
<property name="resourceRef" value="true"/>
</bean>
<bean id="sqlSessionFactory" class="org.mybatis.spring.SqlSessionFactoryBean">
<property name="mapperLocations" value="classpath*:../sql/**.xml"/>
<property name="dataSource"><ref bean="dataSource"/></property>
</bean>
<bean id="sqlSession" class="org.mybatis.spring.SqlSessionTemplate">
<constructor-arg ref="sqlSessionFactory"/>
</bean>
<bean id="oracleTransactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager" p:dataSource-ref="dataSource" />
<bean id="transactionManager" class="com.xxx.xxx.api.transaction.TransactionManager">
<property name="transactionManagers">
<list>
<ref bean="oracleTransactionManager"/>
</list>
</property>
</bean>
Trying to do... logging DB pool status
I am trying to use Spring Aspect to write a log whenever functions in certain dao class is called.
The log I want to write is like DB Pool status such as
active connection counts
idle connection counts
max active connection setting
max idle connection setting
and so on.
Question
Is it possible to access Tomcat's db pool from spring project?
Which would have methods something like this in below.
getNumIdle()
getWaitCount()
getNumActive()
You can simply create a proxy to the tomcatJdbcPoolDataSource and use it as the spring bean. I have created a proxy for C3P0 pooled data source. I later create a spring bean of my class with the required config and use it as a datasource. I believe you can do something similar.
public class C3PODataSourceProxy extends AbstractComboPooledDataSource {
public C3PODataSourceProxy() {
super();
}
public C3PODataSourceProxy(boolean autoregister) {
super(autoregister);
}
public C3PODataSourceProxy(String configName) {
super(configName);
}
#Override
public Connection getConnection() throws SQLException {
try {
Connection connection = super.getConnection();
//You can call the below methods and log it, send it to some other class etc
getNumIdleConnections();
getNumBusyConnections();
return connection;
} catch (Exception exception) {
//log the exception
throw exception;
}
}
public Connection getConnection(String username, String password) throws SQLException {
try {
Connection connection = super.getConnection(username, password);
//You can call the below methods and log it, send it to some other class etc
getNumIdleConnections(username, password);
getNumBusyConnections(username, password);
return connection;
} catch (Exception exception) {
//log the exception
throw exception;
}
}
}
In one of my question asked earlier I got to know that DriverManagerDataSource is NOT intended for production use. So I changed my configuration. I know I am using DBCP which is also outdated and a lot of other connection pools are available like HIkariCP and BOneCP but
I wish to understand the way how to verify that a pool has been setup
or not?
On searching a lot I got some answer at the following link
How would you test a Connection Pool
but I didn't get a way to verify programmatically. Also I cannot debug my jar files used for connection pooling because no source code is available. I dont know why but I can't change my jars for offical reasons.
The following are my configuration (OLD and NEW)
OLD
<bean id="webLogicXADataSource"
class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="#[csa.db.driver]" />
<property name="url" value="#[csa.db.url]" />
<property name="username" value="#[csa.db.username]" />
<property name="password" value="#[csa.db.password]" />
</bean>
NEW
Using DBCP connection pool
<bean id="webLogicXADataSource"
class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="#[csa.db.driver]" />
<property name="url" value="#[csa.db.url]" />
<property name="username" value="#[csa.db.username]" />
<property name="password" value="#[csa.db.password]" />
</bean>
OTHER ELEMENTS:(Thus far I have kept them same like they were earlier)
Place holder
<bean id="placeholderConfig"
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<list>
<value>file:${DB_PROPERTIES}</value>
</list>
</property>
<property name="placeholderPrefix" value="#[" />
<property name="placeholderSuffix" value="]" />
</bean>
Transaction Manager
<bean id="transactionManager"
class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="webLogicXADataSource" />
<qualifier value="inventoryTxManager"/>
</bean>
DAOIMPL SAMPLE BEAN
<bean id="inventoryDao"
class="com.lxnx.fab.ce.icce.inventoryRoutingInvoice.dao.InventoryDaoImpl">
<property name="dataSource" ref="webLogicXADataSource" />
<property name="transactionManager" ref="transactionManager" />
Right now all the DAO classes in my project are singleton(no prototype property set for any of the beans)
The following is the sample java code of the DAOImpl.java class where I need to do all the transactions:
DAOImpl.java
public class InventoryDaoImpl implements InventoryDao {
private final static ISmLog iSmLog = Instrumentation
.getSmLog(LNConstants.SYSTEM_LOG_NAME);
private JdbcTemplate jdbcTemplate;
private DataSource dataSource;
private PlatformTransactionManager transactionManager;
public void setDataSource(DataSource dataSource) {
this.jdbcTemplate = new JdbcTemplate(dataSource);
this.dataSource = dataSource;
}
public void setTransactionManager(
PlatformTransactionManager transactionManager) {
this.transactionManager = transactionManager;
}
#Transactional private void insertRelatedInfoData(
InventoryModel inventoryModel) {
final List<String> relatedLniList = inventoryModel.getArrRelatedLni();
final String documentLni = inventoryModel.getDocumentLNI();
String sql = "INSERT INTO SCSMD_REPO.INV_RELATED_INFO(LNI, RELATED_LNI) VALUES(?,?)";
jdbcTemplate.batchUpdate(sql, new BatchPreparedStatementSetter() {
#Override
public void setValues(PreparedStatement ps, int i) throws SQLException {
String relatedLni = relatedLniList.get(i);
ps.setString(1, documentLni);
ps.setString(2, relatedLni);
}
#Override
public int getBatchSize() {
return relatedLniList.size();
}
});
}
}
I am not getting any errors. Just wanted to verify If a pool has been setup U wish to verify the same
Are all configurations fine or did I miss something?? Please help me out with you valuable answers. thanks
If you don't have logs enable then you can't verify it. however there is one more donkey logic.
every database server will have timeout functionality. if db not hit by application after some time connection will break. for example mysql server will break it's connection from application after 8 hour (if there is no hit from application). you check modify timeout to minimum time (say 30 min)in mysql config file and check after 30 minutes you get connection close exception in you appication, when you hit db
The easiest way, as explained, would be to examine the logs. It's quite likely that a connection pool will log something, at least if your logging level is low enough.
Another way would be to examine the class of the Connection that the datasource returns. If you're dealing with a connection pool, the class will be a wrapper or a proxy class for that pool. The wrapper/proxy class makes sure that when you call close() the connection isn't really closed, it's just returned to the pool for further use. For example if you were to use HikariCP as your pool, you could check if(connection instanceof IHikariConnectionProxy) to see if the pool is being used.
Adding that kind of code in your software would be a bad idea in practically all cases. If you don't know whether a connection pool is being used or not, it's not something you solve with code. It's something you solve by reading and studying more.
You've also named your bean webLogicXADataSource even though nothing seems to support it being an XA datasource. Are you perhaps working on things a bit too advanced for you?
I am using Spring + Hibernate on my JavaEE project.
In this project the user can upload an XLS file which I should import to my database. Before importing I have to validate this file checking its integrity with the other entities on my database. So I have more or less the following:
// The importer
#Component("importer")
public class Importer {
#Autowired
FirstDAO firstDao;
#Autowired
SecondDAO secondDao;
// Read the file and open it (65.000 lines for example)
public void validate() {
foreach line in the file {
firstDAO.has(line[col1]);
secondDao.has(line[col2]);
}
// It stores the valid objects in a List and persist them at the end
}
}
// The DAO
#Repository
public class FirstDao {
#PersistenceContext
protected EntityManager entityManager;
#Transactional(propagation = Propagation.NOT_SUPPORTED)
public boolean has(String name) {
List<Object> result = entityManager.createQuery( from FIRST_TABLE where name = :name)
.setParameter("name", name)
.getResultList();
if (result.size > 0) return true;
else return false;
}
}
// The PersistenceContext/Hibernate configuration
<!-- Data Source -->
<jee:jndi-lookup id="myDS" jndi-name="jdbc/my-DS" cache="true" proxy-interface="javax.sql.DataSource" />
<!-- Entity Manager -->
<bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property value="classpath:META-INF/my_persistence.xml" name="persistenceXmlLocation"/>
<property name="dataSource" ref="myDS"/>
<property name="persistenceUnitName" value="myPersistenceUnit" />
<!--
<property name="loadTimeWeaver">
<bean class="org.springframework.instrument.classloading.InstrumentationLoadTimeWeaver"/>
</property>
-->
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
<property name="database" value="ORACLE" />
<property name="showSql" value="false" />
</bean>
</property>
</bean>
After logging the application I have noticed:
For each query (has method on my DAO) a connection is opened and closed with my Database.
The memory on the server is being flooded (probably memory leak).
After a lot of opening and closing connections I have a connection reset from the Database. Don't know why. And if I still keep requesting coonections, the Datasource is suspended.
I have read somethings about entityManager but I still don't know if I am doing it right, so:
Is it right to execute the validation in a for loop that way? (One connection for each item, meaning 130.000 connections open and closed in a 65000 lines file)
I have read about Stateless Persistence Context for the entityManager. I suspect the memory leak may be there. Maybe Hibernate is kepting a lot of objects in the PersistenceContext. How do I tell Entity Manager to not cache those guys when validating?
Thanks in advance.
First of all, you really shouldn't do that line by line unless you have a very very good reason. Even if the data size is bigger than your memory you should do that 1000 lines at a time or something like that but definitely not one by one.
Because one of the most important optimization for database usage is reducing number of database hit.
Secondly you should not retrieve the data just to check if it is exist.
You should use a basic "select count" query. By that way you will get rid of all stuff like consuming IO to read data and retrieving that data through network to your server and spending memory to just get the number of object in that list.
If you will use my first advice and check the existing of records not one at a time but 1000s at a time you can select just the names instead of all rows.
Btw as far as I can see you are using a datasource if that is properly configured like number of max connection etc. you shouldn't worry about number of database connection.
I just have a new project to maintain, and using Hibernate+Spring. I wrote a DeliveryInfoServiceImpl which have a method to query for some entity but not have any update or save operation, but I have an error:
Hibernate Session bound to thread, and configuration does not allow creation of non-transactional one here
Unless I add #Transactional on the method or the class.
My questions are :
Why I have to add #Transactional though I am only executing select query?
Does adding #Transactional means enable transaction support ,which may have more unnecessary overhead when I am only using "select" query.
Below is my code snippet:
#Override
public List<UnavailableRestaurantBean> getUnavailableRestaurantBean(String custAddress, List<Long> dishesId) {
List<Dish> dishes = getDishByIds(dishesId);//exception here
....
}
private List<Dish> getDishByIds(List<Long> ids){
return deliveryInfoDao.findByIds(Dish.class,ids);
}
And I have below transaction manager config:
<tx:annotation-driven transaction-manager="myTxManager" />
<bean id="myTxManager" name="transactionManager"
class="org.springframework.orm.hibernate3.HibernateTransactionManager">
<property name="sessionFactory" ref="mySessionFactory">
</property>
</bean>