I have a strange situation with my java play framework (2.3) application. All works fine If I have deployed my applications close (geographically) my database mysql. The request, with connections to database works fine and fast. But, last day, I moved the database remotely, in another country. The application go on fine, but, each time I create a JPA Entity Manager (and I think the application create a new connections to remote database) the connections is very very slowly. And the result is that all request are extremely slow.
According your experience there is a way to optimize this situation via application?
Below my controller java code:
#Transactional
public Result testperson() {
Person person= JPAEntityManager.find(Person .class, "XXXXXX");
person.setAddress("XXXXXXX");
JPA.em().persist(person);
return ok("");
}
The #Transactional annotation intercept a play framework jpa implementation for the connections:
public static <T> F.Promise<T> withTransactionAsync(String name, boolean readOnly, play.libs.F.Function0<F.Promise<T>> block) throws Throwable {
EntityManager em = null;
EntityTransaction tx = null;
try {
em = JPA.em(name);
JPA.bindForCurrentThread(em);
if(!readOnly) {
tx = em.getTransaction();
tx.begin();
}
F.Promise<T> result = block.apply();
final EntityManager fem = em;
final EntityTransaction ftx = tx;
F.Promise<T> committedResult = result.map(new F.Function<T, T>() {
#Override
public T apply(T t) throws Throwable {
try {
if(ftx != null) {
if(ftx.getRollbackOnly()) {
ftx.rollback();
} else {
ftx.commit();
}
}
} finally {
fem.close();
}
return t;
}
});
committedResult.onFailure(new F.Callback<Throwable>() {
#Override
public void invoke(Throwable t) {
if (ftx != null) {
try { if (ftx.isActive()) ftx.rollback(); } catch(Throwable e) {}
}
fem.close();
}
});
return committedResult;
} catch(Throwable t) {
if(tx != null) {
try { tx.rollback(); } catch(Throwable e) {}
}
if(em != null) {
em.close();
}
throw t;
} finally {
JPA.bindForCurrentThread(null);
}
}
The JPA.em() create a new EntityManager...
All connections details are default for the play framework: https://www.playframework.com/documentation/2.3.x/SettingsJDBC
Maybe Is there a problem with MySQl database during remote connections?
Can there be some settings to set on the database side to improve a remote connection?
Thanks in advance!
How long does SELECT 1 take? That gives you a good clue of the new overhead for every SQL statement because of reaching into "another country".
If it turns out that there are "too many" queries, consider wrapping a set of them in a Stored Procedure. Then have your app CALL the SP -- this will be one roundtrip, not many.
Related
I'm using Hibernate on my server, (tomcat8, hibernate, postgresql).
Every end of the day my code runs (using Quartz) some code, which calls inside the stored procedure (hibernate):
public void execute(JobExecutionContext context)
throws JobExecutionException {
log.info("=========Start daily update==========");
long startTime = System.currentTimeMillis();
boolean transactionCompleted = false;
int retryCount = HibernateUtil.RETRY_COUNT;
HibernateUtil.closeCurrentEntityManager();
EntityManager em = HibernateUtil.currentEntityManager();
while (!transactionCompleted)
{
try {
em.getTransaction().begin();
dailyUpdateDao.dailyUpdate();
em.getTransaction().commit();
transactionCompleted = true;
} catch (PersistenceException ex) {
if (!HibernateUtil.isDeadlockException(ex) || retryCount == 0) {
log.error("non deadlock error", ex);
throw ex;
}
log.error("deadlock detected. Retrying {}", HibernateUtil.RETRY_COUNT - retryCount);
retryCount--;
if (em.getTransaction().isActive()) {
em.getTransaction().rollback();
}
try {
Thread.sleep(HibernateUtil.sleepIntervalWhenDeadlockDetected(retryCount));
} catch(Exception sleepex) {
log.error("non deadlock sleep ex", ex);
throw ex;
}
}
}
log.info("===========Daily Update Job Completed ============== It took {} ms", System.currentTimeMillis() - startTime);
}
The dailyUpdate function in the code above is doing the following:
public void dailyUpdate() {
String sql = "select count(*) FROM daily_update()";
EntityManager em = HibernateUtil.currentEntityManager();
em.createNativeQuery(sql).getSingleResult();
}
(calling stored procedure using native sql through hibernate).
When I run the server, it does first 2 or 3 calls normally. Next calls never finishes. I reproduced the issue locally, instead of each day I put the schedule to start the task each 1 minute. It showed me logs like this:
Daily Update Job Completed ============== It took 7338 ms
Daily Update Job Completed ============== It took 6473 ms
...
Daily Update Job Completed ============== It took 183381 ms
so the delay increased and I decided to see whats going on inside.
In the code above, when it tries to execute the
em.getTransaction().begin();
it never finishes and the stack trace is shown below as an image:
What is the reason and how to resolve the problem?
EDIT 1: currentEntityManager and closeCurrentEntityManager codes:
public class HibernateUtil {
...
private static EntityManagerFactory entityManagerFactory = Persistence.createEntityManagerFactory(PERSISTENT_UNIT);
public static EntityManager currentEntityManager(EntityManagerFactory emf)
throws HibernateException {
EntityManager em = (EntityManager) entityManager.get();
if (em == null||!em.isOpen()) {
em = emf.createEntityManager();
entityManager.set(em);
}
return em;
}
public static void closeCurrentEntityManager() {
EntityManager s = (EntityManager) entityManager.get();
try {
if (s != null) {
if (s.getTransaction().isActive()) {
if (s.getTransaction().getRollbackOnly()) {
s.getTransaction().rollback();
} else {
s.getTransaction().commit();
}
}
s.close();
}
} finally {
entityManager.remove();
}
}
Ok, perfect, after posting details about what is HibernateUtil, how do closeCurrentEntityManager() and currentEntityManager() work, and what is entityManager fiels inside this class, everything becomes clear.
Look at your code. You firstly closes "current" entity manager:
HibernateUtil.closeCurrentEntityManager();
But you should take into consideration that fact that quartz scheduller starts its tasks in different threads. So
public static void closeCurrentEntityManager() {
EntityManager s = (EntityManager) entityManager.get();
will return null (if it is new thread).
Next step you invoke
EntityManager em = HibernateUtil.currentEntityManager();
which also will create new EntityManager as it is new thread:
em = emf.createEntityManager();
Now look at you screenshot: your beginTransaction() methods awaits for new connection. What's happening here is that you create new entitmanager which opens new connection, but do not close it. So basically your are out of free connections in you pool.
Simply try move HibernateUtil.closeCurrentEntityManager(); into final { ... }block and test again.
I have a db design issue that I am facing with one of my projects. I am trying to implement a service and part of that service is a db layer. It is setup such that I have helper classes that perform get/update methods to the database and a layer on top of them that is a janitor. For ex:
public class GetStudentDBHelper {
public List<Student> get(List<Integer> ids) {
Conn getConnection...
// run sql query and construct returning Student objects
}
public List<Student> get(List<Classroom> byClassroom) {
// get all students in passed in classrooms
// run sql query and construct returning Student objects
}
}
public class StudentJanitor {
public GetStudentDBHelper getStudentDBHelper;
public UpdateStudentDBHelper updateStudentDBHelper;
public UpdateClassroomDBHelper updateClassroomDBHelper;
public List<Student> getStudents(List<Integer> ids) {
return getStudentDBHelper.get(ids);
}
public void saveStudents(List<Students> students, int classRoomid) {
Connection conn = Pool.getConnection(); // assume this gives a jdbc
conn.autocommit(false);
try {
try
{
updateStudentDBHelper.saveForClassroom(students, classRoomid, conn);
updateClassroomDBHelper.markUpdated(classRoomid, conn);
conn.commit();
}
catch
{
throw new MyCustomException(ErrorCode.Student);
}
}
catch (SQLException c)
{
conn.rollback();
}
finally {
conn.close();
}
}
public class ClassroomJanitor{
public void saveClassRoon(List<Classrooms> classrooms) {
Connection conn = Pool.getConnection()// assume this gives a jdbc
conn.autocommit(false);
try {
try {
updateClassroomDBHelper.save(classrooms, conn);
updateStudentDBHelper.save(classrooms.stream().map(Classroom::getStudents).collect(Collections.toList()), conn);
conn.commit();
}
catch {
throw new MyCustomException(ErrorCode.ClassRoom);
}
}
catch (SQLException c)
{
conn.rollback();
}
finally {
conn.close();
}
}...
public class GetClassroomDBHelper{}...
public class UpdateClassroomDBHelper{}...
The update db classes all compose multiple other updators in case they need to update values in other tables (ie. saving a student means I have to touch a classroom table in which a student belongs to update its last updated time for instance).
The issue I am having is for the update db classes, I have to pass in a connection from my Janitor class if i am touching multiple tables in order to have transactions and their rollback capabilities. See above for what I mean. Is there a better way to do this? This type of try, catch, pass in conn to db helpers, will have to be done for any multi transaction operation in my janitors.
In short, you can see that the code is generally like this duplicated across multiple methods:
Connection conn = Pool.getConnection()// assume this gives a jdbc
conn.autocommit(false);
try {
try {
//do some business logic requiring Connection conn
}
catch {
throw new MyCustomException(ErrorCode);
}
}
catch (SQLException c)
{
conn.rollback();
}
finally {
conn.close();
}
Whenever you have a code sequence that is duplicated but it only differs in some parts you can use a template method.
In your case I would introduce a TransactionTemplate class and use a callback interface for the parts that are different. E.g.
public class TransactionTemplate {
private DataSource dataSource;
public TransactionTemplate(DataSource dataSource) {
this.dataSource = Objects.requireNonNull(dataSource);
}
public <T> T execute(TransactionCallback<T> transactionCallback) throws Exception {
Connection conn = dataSource.getConnection();// assume this gives a jdbc
try {
conn.setAutoCommit(false);
T result = transactionCallback.doInTransaction(conn);
conn.commit();
return result;
} catch (Exception e) {
conn.rollback();
throw e;
} finally {
conn.close();
}
}
}
The callback interface would look like this
public interface TransactionCallback<T> {
public T doInTransaction(Connection conn) throws Exception;
}
As you can see the TransactionTemplate manages the transaction while the TransactionCallback implements the logic that must be done in one transaction.
Your client code will then look like this
public class StudentJanitor {
private TransactionTemplate transactionTemplate;
StudentJanitor(DataSource dataSource) {
transactionTemplate = new TransactionTemplate(dataSource);
}
public void saveStudents(List<Students> students, int classRoomid) {
SaveStudentsTransaction saveStudentsTransaction = new SaveStudentsTransaction(students, classRoomid);
transactionTemplate.execute(saveStudentsTransaction);
}
}
and the logic is placed in the TransactionCallback
public class SaveStudentsTransaction implements TransactionCallback<Void> {
public GetStudentDBHelper getStudentDBHelper;
public UpdateStudentDBHelper updateStudentDBHelper;
public UpdateClassroomDBHelper updateClassroomDBHelper;
private List<Students> students;
private int classRoomid;
public SaveStudentsTransaction(List<Students> students, int classRoomid) {
this.students = students;
this.classRoomid = classRoomid;
}
#Override
public Void doInTransaction(Connection conn) throws Exception {
try
{
updateStudentDBHelper.saveForClassroom(students, classRoomid, conn);
updateClassroomDBHelper.markUpdated(classRoomid, conn);
conn.commit();
}
catch
{
throw new MyCustomException(ErrorCode.Student);
}
return null;
}
}
Two main concerns you are currently facing are the boiler plate code for repetitive tasks related to connection (get/execute/close etc)
and infrastructure for getting the same connection across method boundaries. The first is typically solved using Template pattern and the latter
using Threadlocal variables to pass around appropriate connection across methods. These type of concerns have been solved in Java world long ago but
will require you to rely on framework like Spring (JDBC template) etc which have this feature from last decade or so or you would need to roll out stripped
down version of this infrastructure. If you are interested in latter then you can take hint from similar attmepts shared on Github like this.
In my web application I'm using Stateless sessions with Hibernate to have better performances on my inserts and updates.
It was working fine with H2 database (the one used in play framework in dev mode).
But when I test it with MySQL I get the following exception :
ERROR ~ Lock wait timeout exceeded; try restarting transaction
ERROR ~ HHH000315: Exception executing batch [Lock wait timeout exceeded; try restarting transaction]
Here is the code :
public static void update() {
Session session = (Session) JPA.em().getDelegate();
StatelessSession stateless = this.session.getSessionFactory().openStatelessSession();
try {
stateless.beginTransaction();
// Fetch all products
{
List<ProductType> list = ProductType.retrieveAllWithHistory();
for (ProductType pt : list) {
updatePrice(pt, stateless);
}
}
// Fetch all raw materials
{
List<RawMaterialType> list = RawMaterialType.retrieveAllWithHistory();
for (RawMaterialType rm : list) {
updatePrice(rm, stateless);
}
}
} catch (Exception ex) {
play.Logger.error(ex.getMessage());
ExceptionLog.log(ex, Thread.currentThread());
} finally {
stateless.getTransaction().commit();
stateless.close();
}
}
private static void updatePrice(ProductType pt, StatelessSession stateless) {
pt.priceDelta = computeDelta();
pt.unitPrice = computePrice();
stateless.update(pt);
PriceHistory ph = new PriceHistory(pt, price);
stateless.insert(ph);
}
private static void updatePrice(RawMaterialType rm, StatelessSession stateless) {
rm.priceDelta = computeDelta();
rm.unitPrice = computePrice();
stateless.update(rm);
PriceHistory ph = new GoodPriceHistory(rm, price);
stateless.insert(ph);
}
In this example I have 3 simple Entities (ProductType, RawMaterialType and PriceHistory).
computeDelta and computePrice are just algorithm functions with no DB stuff.
retrieveAllWithHistory functions are functions that fetch some data from the database using Play framework model functions.
So, this code retrieves some data, edit some, create new one and finally save everything.
Why have I a lock exception with MySQL and no exception with H2 ?
I'm not sure why you have a commit in a finally block. Give this structure a try:
try {
factory.getCurrentSession().beginTransaction();
factory.getCurrentSession().getTransaction().commit();
} catch (RuntimeException e) {
factory.getCurrentSession().getTransaction().rollback();
throw e; // or display error message
}
Also, it might be helpful for you to check this documentation.
I am using the MySQL JDBC Replication Driver com.mysql.jdbc.ReplicationDriver to shift load between Master and Slave.
I am using that connection URL
jdbc.de.url=jdbc:mysql:replication://master:3306,slave1:3306,slave2:3306/myDatabase?zeroDateTimeBehavior=convertToNull&characterEncoding=UTF-8&roundRobinLoadBalance=true
As soon as I am starting my application I am getting only that data from where it has been started, like I am working on a locked snapshot of the database. If I am doing any CRUD operation the data is not callable or updates are not shown. Replication of mysql is working just fine and I can query the correct data from the database.
There is no level2 cache active and I am using hibernate with pooled connections
If I am using the normal JDBC Driver com.mysql.jdbc.Driver everything is working just fine. So why am I getting always the same resultsets, no matter what I do change in the database...
Update 1
It seems like it is related to my aspect
#Aspect
public class ReadOnlyConnectionInterceptor implements Ordered {
private class ReadOnly implements ReturningWork<Object> {
ProceedingJoinPoint pjp;
public ReadOnly(ProceedingJoinPoint pjp) {
this.pjp = pjp;
}
#Override
public Object execute(Connection connection) throws SQLException {
boolean autoCommit = connection.getAutoCommit();
boolean readOnly = connection.isReadOnly();
try {
connection.setAutoCommit(false);
connection.setReadOnly(true);
return pjp.proceed();
} catch (Throwable e) {
//if an exception was raised, return it
return e;
} finally {
// restore state
connection.setReadOnly(readOnly);
connection.setAutoCommit(autoCommit);
}
}
}
private int order;
private EntityManager entityManager;
public void setOrder(int order) {
this.order = order;
}
#Override
public int getOrder() {
return order;
}
#PersistenceContext
public void setEntityManager(EntityManager entityManager) {
this.entityManager = entityManager;
}
#Around("#annotation(readOnlyConnection)")
public Object proceed(ProceedingJoinPoint pjp,
ReadOnlyConnection readOnlyConnection) throws Throwable {
Session hibernateSession = entityManager.unwrap(Session.class);
Object result = hibernateSession.doReturningWork(new ReadOnly(pjp));
if (result == null) {
return result;
}
//If the returned object extends Throwable, throw it
if (Throwable.class.isAssignableFrom(result.getClass())) {
throw (Throwable) result;
}
return result;
}
}
I annotate all my readOnly request with #ReadOnlyConnection. Before I had all my service layer methods annotated with that even though they might be calling each other. Now I am only annotating the request method and I am to the state, where I am getting the database updates on the second call.
1) Doing initial call => getting data as expected
2) Changing data in the database
3) Doing same call again => getting the exact same data from the first call
4) Doing same call again => getting the changed data
The thing with connection.setAutoCommit(false) is that it seems to not do a commit after set back to connection.setAutoCommit(true). So after adding the following line to the aspect, everything worked as expected again
try {
connection.setAutoCommit(false);
connection.setReadOnly(true);
return pjp.proceed();
} catch (Throwable e) {
return e;
} finally {
// restore state
connection.commit(); // THIS LINE
connection.setReadOnly(readOnly);
connection.setAutoCommit(autoCommit);
}
In Hibernate when i save() an object in a transaction, and then i rollback it, the saved object still remains in the DB. It's strange because this issue doesn't happen with the update() or delete() method, just with save().
Here is the code i'm using:
DbEntity dbEntity = getDbEntity();
HibernateUtil.beginTransaction();
Session session = HibernateUtil.getCurrentSession();
session.save(dbEntity);
HibernateUtil.rollbackTransaction();
And here is the HibernateUtil class (just the involved functions, i guarantee the getSessionFactory() method works well - there is an Interceptor handler, but it doesn't matter now):
private static final ThreadLocal<Session> threadSession = new ThreadLocal<Session>();
private static final ThreadLocal<Transaction> threadTransaction = new ThreadLocal<Transaction>();
/**
* Retrieves the current Session local to the thread.
* <p/>
* If no Session is open, opens a new Session for the running thread.
*
* #return Session
*/
public static Session getCurrentSession()
throws HibernateException {
Session s = (Session) threadSession.get();
try {
if (s == null) {
log.debug("Opening new Session for this thread.");
if (getInterceptor() != null) {
log.debug("Using interceptor: " + getInterceptor().getClass());
s = getSessionFactory().openSession(getInterceptor());
} else {
s = getSessionFactory().openSession();
}
threadSession.set(s);
}
} catch (HibernateException ex) {
throw new HibernateException(ex);
}
return s;
}
/**
* Start a new database transaction.
*/
public static void beginTransaction()
throws HibernateException {
Transaction tx = (Transaction) threadTransaction.get();
try {
if (tx == null) {
log.debug("Starting new database transaction in this thread.");
tx = getCurrentSession().beginTransaction();
threadTransaction.set(tx);
}
} catch (HibernateException ex) {
throw new HibernateException(ex);
}
}
/**
* Rollback the database transaction.
*/
public static void rollbackTransaction()
throws HibernateException {
Transaction tx = (Transaction) threadTransaction.get();
try {
threadTransaction.set(null);
if ( tx != null && !tx.wasCommitted() && !tx.wasRolledBack() ) {
log.debug("Tyring to rollback database transaction of this thread.");
tx.rollback();
}
} catch (HibernateException ex) {
throw new HibernateException(ex);
} finally {
closeSession();
}
}
Thanks
Check if your database supports a roll back i.e. if you're using InnoDB tables and not MyISAM (you can mix transactional and non-transactional tables but in most cases, you want all your tables to be InnoDB).
MySQL by default uses the MyIsam storage engine. As the MyISAM does not support transactions, insert, update and delete statements are directly written to the database. The commit and rollback statements are ignored.
In order to use transaction you need to change the storage engine of you tables. Use this command:
ALTER TABLE table_name ENGINE = InnoDB;
(note how ever, that the two storage engines are different and you need to test you're application if it still behaves as expected)