Our Postgresql application is getting Hibernate error: org.hibernate.util.JDBCExceptionReporter - ERROR: deadlock detected. One of the ways recommended to deal with this problem is setting transaction timeout
(Hibernate Reference Documentation) :
sess.getTransaction().setTimeout(3)
How this value of 3 seconds is defined?
If your queries are deadlocking, look into why they are deadlocking and fix that. The error message in the PostgreSQL server error log tells you about the transactions that deadlocked. If that alone isn't enough, set log_statement = 'all', add a log_line_prefix to identify transactions or use csv logging, and analyze the log files to see what's happening.
If you're stuck you can hand-recreate the deadlock and take a look at pg_locks for additional information about what's going on; see the lock monitoring wiki article.
If the deadlock detection timeout is too long for you, lower PostgreSQL's deadlock detection timeout, don't add statement timeout hacks in the application. See the documentation on lock management.
Related
I am running a postgres query that takes more than two hours.
This query is executed using hibernate in a java program.
After about 1.5 hours the query stops showing up in the server status in pg_admin.
Since, the query disappeared from the list of active queries on the database, I am expecting a success or a timeout exception. But, I get none.(No exception) and my thread in stuck in the wait state.
I know the query has not finished because it was supposed to do some inserts in a table and I cannot find the expected rows in the table.
I am using pgbouncer for the connection pooling and the query_timeout is disabled.
Had it been a hibernate timeout I should have got an exception.
OS parameters on the DB machine and Client machine(Machine running java program)
tcp_keepalive_time is 7200 (seconds)
tcp_keepalive_intvl = 75
tcp_keepalive_probes = 9 (number of probes)
Both the machines run RHEL operating system.
I am unable to put my finger on the issue.
I found that the issue was caused due to the TCP connection getting dropped and the client still hanging waiting for the response.
I altered the following parameters at OS level:-
/proc/sys/net/ipv4/tcp_keepalive_time = 2700
Default value was 7200.
This causes a keep alive check at every 2700 seconds instead of 7200 seconds.
I am sure you would have already looked at the following resources:
PostgreSQL Timeout Docs
PgBouncer timeout (you already mention).
Hibernate timeout parameters, if any.
Once that is done, (just like triaging permission issues during a new installation, ) I recommend that you try the following SQL, from different scenarios (given below) and ascertain what is actually causing this timeout:
SELECT pg_sleep(7200);
Login to the server (via psql) and see whether this SQL times-out.
Login to the PgBouncer (again via psql) and see whether PgBouncer times out.
Execute this SQL via Hibernate (via PgBouncer), and see whether there is a timeout.
This should allow you to clearly isolate the cause for this.
I am working on IntelliJ IDEA 14.1.4, Recently we connected database to the Client and it was working okay, until some recent times when we start getting error
[2015-09-28 10:12:55] locked by transaction: #console:Oracle - <<DBName>>#localhost
Now we can't perform any transaction as we keep getting the same error. We tried googling the error but could not find anything to solve the problem. Any help would be appreciated!!!
I had this happen after a previous query I ran failed -- using PyCharm, not IntelliJ, but it is the same JetBrains system. I did not have Auto-commit turned on. Pycharm couldn't recover from the failed query for some reason, and it left an unresolved transaction active in the database. I disconnected from the database (hit the red stop button in the database window), and then I was able to resume with new queries with no problem. You might also try the rollback button at the top of the window with your SQL statements if reconnecting would cause you problems.
Putting up answer for the same comment.
I think there should be some row lock on the DB. if you have an Oracle DBA, check out for table/row lock. It can happen if the transaction exception occurred and rollback didn't happen properly or a transaction is still open for a longer time.
You can execute the following query to check for the same if transaction is pending.
SELECT COUNT(*)
FROM v$transaction t, v$session s, v$mystat m
WHERE t.ses_addr = s.saddr
AND s.sid = m.sid;
Additional resources :
Oracle: How to find out if there is a transaction pending?
How to find locked rows in Oracle
Consult with your DBA after your initial research. I'm not aware about releasing locks.
I'm using Wildfly 8.2 and fire a series of DB requests when a certain web page is opened. All queries are invoked thru JPA Criteria API, return results as expected - and - none of them delivers a warning, error or exception. It all runs in Parallel Plesk.
Now, I noticed that within 2 to 3 days the following error appears and the site becomes unresponsive. I restart and I wait approx another 3 days till it happens again (depending on the number of requests I have).
I checked the tcpsndbuf on my linux server and I noticed it is constantly at max. Unless I restart Wildfly. Apparently it fails to release the connections.
The connections are managed by JPA/Hibernate and the Wildfly container. I don't do any special or custom transaction handling e.g. open, close. etc. I leave it all to Wildfly.
The MySQL Driver I'm using is 5.1.21 (mysql-connector-java-5.1.21-bin.jar)
In the standalone.xml I have defined the following datasource datasource values (among others):
<transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation>
<pool>
<min-pool-size>3</min-pool-size>
<max-pool-size>10</max-pool-size>
</pool>
<statement>
<prepared-statement-cache-size>32</prepared-statement-cache-size>
<shared-prepared-statements>true</shared-prepared-statements>
</statement
Has anyone experience the same rise of tcpsndbuf values (or this error)? In case you require more config or log files, let me know. Thanks!
UPDATE
Despite the following additional timeout settings, it still runs into the hanger. And thus, it will then use 100% CPU time, whenever the max tcpsndbuf is reached.,
Try adding this Hibernate property:
<property name="hibernate.connection.release_mode">after_transaction</property>
By default, JTA mandates that connection should be released after each statement, which is undesirable for most use cases. Most Drivers don't allow multiplexing a connection over multiple XA transactions anyway.
Do you use openvz? I think this question should be asked on serverfault. It is related to linux configuration. You can read: tcpsndbuf. You should count opened sockets and check condition:
We are facing the following exception in weblogic server v10.3.2.0. We are using JRockit JRE 6.0.
We have around 6-7 XA datasources involved in every server request. We face this exception when processing on the last datasource just begins.
Please someone advise.
java.sql.SQLException: Unexpected exception while enlisting XAConnection
java.sql.SQLException: Transaction rolled back: setRollbackOnly called on transaction
at weblogic.jdbc.jta.DataSource.enlist(DataSource.java:1616)
at weblogic.jdbc.jta.DataSource.refreshXAConnAndEnlist(DataSource.java:1503)
at weblogic.jdbc.jta.DataSource.getConnection(DataSource.java:446)
at weblogic.jdbc.jta.DataSource.connect(DataSource.java:403)
at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:364)
at com.ibatis.sqlmap.engine.transaction.jta.JtaTransaction.init(JtaTransaction.java:68)
at com.ibatis.sqlmap.engine.transaction.jta.JtaTransaction.getConnection(JtaTransaction.java:131)
at com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryForObject(MappedStatement.java:120)
at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForObject(SqlMapExecutorDelegate.java:518)
at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForObject(SqlMapExecutorDelegate.java:493)
at com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryForObject(SqlMapSessionImpl.java:106)
at com.ibatis.sqlmap.engine.impl.SqlMapClientImpl.queryForObject(SqlMapClientImpl.java:82)
As you wrote, the cause is unkown in this sample.
We can see the transaction has been marked as "must roll back", probably by the previous datasources when something went wrong.
Maybe you can check previous logs, for the previous datasource, to find the cause ?
You say that it is the last datasource - have you read this ? : http://muness.blogspot.com/2005/09/distributed-transactions-and-timeouts.html .
If you need more info, can you replace ibatis with a version with hacked com.ibatis.sqlmap.engine.transaction.jta.JtaTransaction.init() ? Add some logging there and you'll know more, probably.
If I had to guess I would say the last datasource is not configured properly as an XA datasource, doesn't have the the XA driver installed, or doesn't support XA.
Are you doing any funny exception handling here that would truncate the stack(Catching re-throwing but only keeping the top set of stack frames) or using a custom exception handling library? If you are I would abandon it. It seems like there should be a caused by: with additonal lower level stack related to your datasource's drivers that would reveal additional information.
If that isn't the case and this is the only info you're getting. It might be time to crank your server's logging up to debug or trace and get down and dirty with how weblogic gets things done..
Alternatively, if you are supported I would verify your driver versions/ configurations with your vendor. If you're not, you need to track down the documentation and verify for yourself.
I have a long-running method which executes a large number of native SQL queries through the EntityManager (TopLink Essentials). Each query takes only milliseconds to run, but there are many thousands of them. This happens within a single EJB transaction. After 15 minutes, the database closes the connection which results in following error:
Exception [TOPLINK-4002] (Oracle TopLink Essentials - 2.1 (Build b02-p04 (04/12/2010))): oracle.toplink.essentials.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: Closed Connection
Error Code: 17008
Call: select ...
Query: DataReadQuery()
at oracle.toplink.essentials.exceptions.DatabaseException.sqlException(DatabaseException.java:319)
.
.
.
RAR5031:System Exception.
javax.resource.ResourceException: This Managed Connection is not valid as the phyiscal connection is not usable
at com.sun.gjc.spi.ManagedConnection.checkIfValid(ManagedConnection.java:612)
In the JDBC connection pool I set is-connection-validation-required="true" and connection-validation-method="table" but this did not help .
I assumed that JDBC connection validation is there to deal with precisely this kind of errors. I also looked at TopLink extensions (http://www.oracle.com/technetwork/middleware/ias/toplink-jpa-extensions-094393.html) for some kind of timeout settings but found nothing. There is also the TopLink session configuration file (http://download.oracle.com/docs/cd/B14099_19/web.1012/b15901/sessions003.htm) but I don't think there is anything useful there either.
I don't have access to the Oracle DBA tables, but I think that Oracle closes connections after 15 minutes according to the setting in CONNECT_TIME profile variable.
Is there any other way to make TopLink or the JDBC pool to reestablish a closed connection?
The database is Oracle 10g, application server is Sun Glassfish 2.1.1.
All JPA implementations (running on a Java EE container) use a datasource with an associated connection pool to manage connectivity with the database.
The persistence context itself is associated with the datasource via an appropriate entry in persistence.xml. If you wish to change the connection timeout settings on the client-side, then the associated connection pool must be re-configured.
In Glassfish, the timeout settings associated with the connection pool can be reconfigured by editing the pool settings, as listed in the following links:
Changing timeout settings in GlassFish 3.1
Changing timeout settings in GlassFish 2.1
On the server-side (whose settings if lower than the client settings, would be more important), the Oracle database can be configured to have database profiles associated with user accounts. The session idle_time and connect_time parameters of a profile would constitute the timeout settings of importance in this aspect of the client-server interaction. If no profile has been set, then by default, the timeout is unlimited.
Unless you've got some sort of RAC failover, when the connection is terminated, it will end the session and transaction.
The admins may have set into some limits to prevent runaway transactions or a single job 'hogging' a connection in a pool. You generally don't want to lock a connection in a pool for an extended period.
If these queries aren't necessarily part of the same transaction, then you could try terminating and restarting a new connection.
Are you able to restructure your code so that it completes in under 15 minutes. A stored procedure in the background may be able to do the job a lot quicker than dragging the results of thousands of operations over the network.
I see you set your connection-validation-method="table" and is-connection-validation-required="true", but you do not mention that you specified the table you were validating on; did you set validation-table-name="any_table_you_know_exists" and provide any existing table-name? validation-table-name="existing_table_name" is required.
See this article for more details on connection validation.
Related StackOverflow article with similar problem - he wants to flush the entire invalid connection pool.