Randomly my Spring MVC/Oracle 11G/Tomcat 7 application is throwing the following error: “ORA-02067: transaction or savepoint rollback required”
Once that error appears ALL subsequent operations to the database that uses that same connection through the Tomcat will fail throwing the ORA-02067 error.
Example:
"detailErrorInfo": "CallableStatementCallback;
uncategorized SQLException for SQL
[{call SCHEMAX.[PACKAGEX].PROCEDUREX()}]; SQL state [60000];
error code [604];
ORA-00604: error occurred at recursive SQL level 1
ORA-02067: transaction or savepoint rollback required;
nested exception is java.sql.SQLException:
ORA-00604: error occurred at recursive SQL level 1
ORA-02067: transaction or savepoint rollback required",
Curiously, if I call that same Oracle package/procedure from other connection or application, it works.
The error stops only on the Spring application the moment the administrator restart the Tomcat.
My theory is there is a transaction that is failing and is not doing a rollback. Because the Tomcat is using a pool of JDBC connections, the error persists on the application.
I have not being able to reproduce that on the development environment.
Does Oracle provides tools to pinpoint the culprit transaction? Is there any view that display transaction that are pending a rollback?
Thanks
Related
I have a method annotated with #Transactional which obtain messages and perform some operations on the database to persist things correctly.
Now, some times this method throws a DeadLock with the following stacktrace:
Caused by: org.hibernate.TransactionException: Unable to commit against JDBC Connection
at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.commit(AbstractLogicalConnectionImplementor.java:87)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.commit(JdbcResourceLocalTransactionCoordinatorImpl.java:272)
at org.hibernate.engine.transaction.internal.TransactionImpl.commit(TransactionImpl.java:104)
at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:532)
... 34 common frames omitted
Caused by: com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:123)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
at com.mysql.cj.jdbc.ConnectionImpl.commit(ConnectionImpl.java:813)
at com.zaxxer.hikari.pool.ProxyConnection.commit(ProxyConnection.java:361)
at com.zaxxer.hikari.pool.HikariProxyConnection.commit(HikariProxyConnection.java)
at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.commit(AbstractLogicalConnectionImplementor.java:81)
... 37 common frames omitted
The problem I am facing is that on the database side, MySQL is not registering any deadlock to happen in that moment.
I've enabled the innodb_print_all_deadlocks variable and I verified that it is working as expected by manually raising deadlocks which are immediately logged by the db.
I've also verified that when a real deadlock happens within the application (with real I mean that MySQL logs the deadlock on its side), the stacktrace is different and usually the deadlock is found by the database server way before the commit, for example when the application tries to ClientPreparedStatement.executeUpdateInternal.
It looks weird to me that the deadlock is raised on ConnectionImpl.commit. I've checked the ConnectionImpl code and I can't understand why a deadlock could be thrown there (at the line in the stacktrace, there's just return; )
Is it possible that JPA is somehow raising a deadlock on the app side?
Leaving this as it might help other people stuck with the same weird problem.
The reason for my problem was the fact that the application was using MariaDB driver instead of MySQL one. For some reasons, the MariaDB driver was throwing fake deadlock exceptions, so the MySQL server was correct in not logging any deadlock happening.
Switching the driver with the more correct one, all problems stopped.
The following similar exceptions I'm getting in my log files frequently. Afterwards server itself wont accept any data and sync services are not working, even for valid Userlogin credentials.
Can anyone suggest me what will be the cause for it and how can I resolve it.?
---- exception report ----------------------------------------------------------
Failure in findOne operation for entity [UserLogin]: org.ofbiz.entity.transaction.GenericTransactionException: The current transaction is marked for rollback, not beginning a new transaction and aborting current operation; the rollbackOnly was caused by: Failure in create operation for entity [InvoiceAttribute]: org.ofbiz.entity.GenericEntityException: Error while inserting: [GenericEntity:InvoiceAttribute]
Environment : Ubuntu 14.04
Database : MySql v5.6
I have a pretty large transaction, annotated with #Transactional. There are a few long-running queries in it, but usually runs fine. About 20% of the time the Connection appears to be getting forcibly closed outside of the transaction, and when the transaction tries to continue doing work, it fails with the following stack trace. Even worse, the transaction is not rolling back.
JDBC commits by default on connection close. However, Spring should be setting the Connection's auto-commit to false using something like the following before opening the transaction (Spring #Transactional and JDBC autoCommit). I've confirmed that the version we are using still does this. We use SimpleJdbcTemplate to execute the queries and the connections are obtained from an Apache Commons DBCP pool.
Is this an issue where DBCP thinks the connection is stale (as the transaction has a few long-running queries in it) and so the pool tries to reclaim the connection, committing it along the way? Shouldn't autocommit=false prevent this? Anyone have any other suggestions?
Thanks
Using (All pretty old):
SpringJDBC 2.5
Oracle JDBC Drivers 11.1.0.7 (ojdbc6)
Apache Commons DBCP 1.2.1
Stack trace:
Activity threw exception:
org.springframework.transaction.TransactionSystemException: Could not roll back JDBC transaction; nested exception is java.sql.SQLException: Connection oracle.jdbc.driver.T4CConnection#56a2191a is closed.
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doRollback(DataSourceTransactionManager.java:279)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:800)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:777)
at org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:339)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
[...]
Caused by: java.sql.SQLException: Connection oracle.jdbc.driver.T4CConnection#56a2191a is closed.
at org.apache.commons.dbcp.DelegatingConnection.checkOpen(DelegatingConnection.java:398)
at org.apache.commons.dbcp.DelegatingConnection.rollback(DelegatingConnection.java:368)
at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.rollback(PoolingDataSource.java:323)
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doRollback(DataSourceTransactionManager.java:276)
... 21 more
I keep getting this error randomly when I test an application with mssql and don't have an idea of what I can do to correct the issue.
Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.0.v20130507-3faac2b):
org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: com.microsoft.sqlserver.jdbc.SQLServerException:
Connection reset ClientConnectionId:a2fb508d-0e8e-48b0-92ed-2d1f262b4db1 Error Code: 0
My setup consists of
Eclipselink as the ORM
sqljdbc v4.1
sql server 2014
Java 8
I believe the problem is probably a setting within sql-server though I haven't seen anything that I could correct. The application works perfect on postgres and mysql but the customer only wants to use mssql.
MSFT JDBC drivers do not call the sp_reset_connection procedure used to cleanup from the previous session.
I am thinking that this error is coming from something left hanging that is being cleaned up with the other databases.
You will need to run a SQL Profile trace on all the connections until the error occurs to see the actual cause.
More can be found at :
What does "exec sp_reset_connection" mean in Sql Server Profiler?
We have a stand-alone single user desktop application that has a Swing front-end and uses an embedded Derby database (10.8.2.2 1181258). Spring (3.0.4) is used as the go between the application and the database. A customer (in Hebrew/Israel locale iw_IL) running the application using JRE 1.7.0_21 ran into an issue. The following stack trace was logged:
ERROR [index-stat-thread] stderr - Exception in thread "index-stat-thread"
ERROR [index-stat-thread] stderr - java.lang.ClassCastException: org.apache.derby.impl.store.access.btree.BranchControlRow cannot be cast to org.apache.derby.impl.store.access.btree.LeafControlRow
ERROR [index-stat-thread] stderr - at org.apache.derby.impl.store.access.btree.BTreeScan.positionAtNextPage(Unknown Source)
ERROR [index-stat-thread] stderr - at org.apache.derby.impl.store.access.btree.BTreeForwardScan.fetchRows(Unknown Source)
ERROR [index-stat-thread] stderr - at org.apache.derby.impl.store.access.btree.BTreeScan.fetchNextGroup(Unknown Source)
ERROR [index-stat-thread] stderr - at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl$KeyComparator.fetchRows(Unknown Source)
ERROR [index-stat-thread] stderr - at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl.updateIndexStatsMinion(Unknown Source)
ERROR [index-stat-thread] stderr - at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl.generateStatistics(Unknown Source)
ERROR [index-stat-thread] stderr - at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl.processingLoop(Unknown Source)
ERROR [index-stat-thread] stderr - at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl.run(Unknown Source)
ERROR [index-stat-thread] stderr - at java.lang.Thread.run(Thread.java:722)
Any idea what might cause such an error?
Subsequently, Spring logged the following:
org.springframework.jdbc.support.SQLErrorCodesFactory - Error while extracting database product name - falling back to empty error codes
org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLNonTransientConnectionException: No current connection.
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:293)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:320)
at org.springframework.jdbc.support.SQLErrorCodesFactory.getErrorCodes(SQLErrorCodesFactory.java:212)
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.setDataSource(SQLErrorCodeSQLExceptionTranslator.java:141)
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.<init>(SQLErrorCodeSQLExceptionTranslator.java:104)
at org.springframework.jdbc.support.JdbcAccessor.getExceptionTranslator(JdbcAccessor.java:99)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:602)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:636)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:665)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:673)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:713)
at org.springframework.jdbc.core.JdbcTemplate.queryForList(JdbcTemplate.java:796)
at org.springframework.jdbc.core.simple.SimpleJdbcTemplate.queryForList(SimpleJdbcTemplate.java:235)
Given that the application uses an embedded Derby database on the same machine and file system as the application, the JDBC connection error is baffling. Could the first Derby error be the reason for this?
Following this is another error from Spring while loading entries for one table, which essentially seems to be the same as the first error from Derby:
ERROR [main] org.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback; uncategorized SQLException for SQL [select * from XYZ where scresult_id=?]; SQL state [XJ001]; error code [0]; Java exception:
'org.apache.derby.impl.store.access.btree.BranchControlRow cannot be cast to org.apache.derby.impl.store.access.btree.LeafControlRow: java.lang.ClassCastException'.; nested exception is java.sql.SQLException: Java exception:
'org.apache.derby.impl.store.access.btree.BranchControlRow cannot be cast to org.apache.derby.impl.store.access.btree.LeafControlRow: java.lang.ClassCastException'.
The same application has been used successfully by other customers for several months. This is the first time a customer has run into these problems.
It looks like your database has become corrupt. Those are internal messages that you should never see.
Here is information about how to use your backups and logs to recover your database: http://db.apache.org/derby/docs/10.10/adminguide/cadminhubbkup98797.html
Before you restore the database, you should retain a copy of the damaged database, the derby.log file, and any other information about the circumstances under which the database became corrupt, as they may give you important clues about how to prevent this problem in the future.
In general, it should not be possible to corrupt a Derby database like this; hence arriving at such a situation might indicate a bug in Derby itself. But there could be other possible explanations, such as damaged or misconfigured hardware.
I assume from your initial question that you do not know how to reproduce this problem? In that case, preserving as much information as you can about the incident, and about what was occurring at the moment that the corruption occurred (did the power fail? Was the machine hard-rebooted? did a disk become full? was a virus-checker running on the machine? There are a million possible questions like this to ask...) may give you clues about how to reproduce the problem yourself.