Spring Batch - Connection closed in when processing is done in external process - java

I have a job that is built of several steps - one of the steps is a tasklet that activates processing Pentaho
I pass to Pentaho the parameters it needs in order to connect to the DB on its own and it works OK
The issue I have starts when the processing time in Pentaho is long
Pentaho completes successfully and the code in the tasklet that activated it completes OK, but in the job mechanism that wraps it I get an error when it tries to update the job execution table in the db because the connection it has was already closed
o.s.j.s.SQLErrorCodesFactory: Error while extracting database product name - falling back to empty error codes
org.springframework.jdbc.support.MetaDataAccessException: Error while extracting DatabaseMetaData;
nested exception is
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:296)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:320)
at org.springframework.jdbc.support.SQLErrorCodesFactory.getErrorCodes(SQLErrorCodesFactory.java:214)
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.setDataSource(SQLErrorCodeSQLExceptionTranslator.java:141)
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.<init>(SQLErrorCodeSQLExceptionTranslator.java:104)
at org.springframework.jdbc.support.JdbcAccessor.getExceptionTranslator(JdbcAccessor.java:99)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:603)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:812)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:868)
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.persistSerializedContext(JdbcExecutionContextDao.java:230)
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.updateExecutionContext(JdbcExecutionContextDao.java:159)
at org.springframework.batch.core.repository.support.SimpleJobRepository.updateExecutionContext(SimpleJobRepository.java:203)
...
14:21:37.143 UTC [ERROR] jobScheduler_Worker-2 T:b U: o.s.t.i.TransactionInterceptor: Application exception overridden by rollback exception
org.springframework.dao.RecoverableDataAccessException: PreparedStatementCallback; SQL [UPDATE BAT_STEP_EXECUTION_CONTEXT SET SHORT_CONTEXT = ?, SERIALIZED_CONTEXT = ? WHERE STEP_EXECUTION_ID = ?]; Communications link failure
It looks like the connection that the job repository received when the job started was abandoned and I'm trying to understand if there is a way to order it get a new connection or give it some keep alive command
I tried the following workarounds
change the step status in a job listener so the job will complete - fails with the same DB error
mark this exception as if it can be skipped - fails with the same DB error
<batch:no-rollback-exception-classes>
<batch:include class="com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException" />
<batch:include class="org.springframework.jdbc.support.MetaDataAccessException" />
</batch:no-rollback-exception-classes>
Any ideas how I can work around this?
Can I configure a job listener that will restart the job from the step that follows the Pentaho step?
Additional info
I think that the issue is here -
org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSource)
This
ConnectionHolder conHolder = (ConnectionHolder) TransactionSynchronizationManager.getResource(dataSource);
thinks that the connection is valid
so I guess the solution will be to call org.springframework.transaction.support.TransactionSynchronizationManager.unbindResource(Object)
and the question is how can I get the data source object to pass to this method
I will try querying the
org.springframework.transaction.support.TransactionSynchronizationManager.getResourceMap() and see where it gets me
update
no luck - the get resources map gives me just the repositories I'm using, not the data source. Still digging...
Another update
I'm debugging the process and it seems that the problem is indeed org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSource) the connection holder is holding a connection that is closed but the code here doesn't check if the connection is open; it only checks if the connection isn't null and if it was some weak reference maybe it was enough here - but in this use case it just proceedes with the closed connection instead of requesting a new one.

add this to the tasklet definition
<batch:transaction-attributes propagation="NEVER" />
since the Tasklet is doing external processing and doesn't need a spring batch transaction it need to tell spring batch not to open a transaction for this tasklet.
see
http://www.javabeat.net/transaction-management-in-spring-batch-components/
http://forum.spring.io/forum/spring-projects/batch/91158-legacy-integration-tasklet-transaction

Related

Best way to solve conflict between EclipseLink #Version Timestamp-Field and DB2

In the current project I'm working on, we are using EclipseLink as JPA-Provider. In the background is a DB2-database.
We want to introduce "Optimistic Locking" with the restriction only using Timestamp-fields. Furthermore, these timestamp fields are CURRENT TIMESTAMP fields, so when updating an entity, the CURRENT TIMESTAMP field updates the value itself. This is necessary to guarantee the reliability of the whole system.
When annotating the attribute of an entity with #Version, I get the following error message:
Internal Exception: com.ibm.websphere.ce.cm.StaleConnectionException: THE SQL STATEMENT IS NOT SUPPORTED. SQLCODE=-142, SQLSTATE=42612, DRIVER=4.22.29
Error Code: -142
Call: VALUES CURRENT TIMESTAMP
Query: ValueReadQuery(sql="VALUES CURRENT TIMESTAMP")
at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl$1.handleException(EntityManagerSetupImpl.java:767)
at org.eclipse.persistence.transaction.AbstractSynchronizationListener.handleException(AbstractSynchronizationListener.java:275)
[…]
Note: I only get this error message when using the mentioned annotation.
My research resulted in the following:
Internal Exception: com.ibm.websphere.ce.cm.StaleConnectionException:
--> A StaleConnectionException is an exception that is generated by the WebSphere Application Server database connection code when a JDBC driver returns a fatal error from a connection request or operation. In WebSphere Application Server, the StaleConnectionException is issued when the database vendor issues an exception indicating that a connection currently in the connection pool is no longer valid. ( https://developer.ibm.com/answers/questions/205910/how-to-resolve-the-staleconnectionexception-in-web/ )
THE SQL STATEMENT IS NOT SUPPORTED. SQLCODE=-142,
--> An SQL statement was detected that is not supported by the database. The statement might be valid for other IBM® relational database products or it might be valid in another context. For example, statements such as VALUES and SIGNAL or RESIGNAL SQLSTATE can be used only in certain contexts, such as in a trigger body or in an SQL Procedure. ( https://www.ibm.com/support/knowledgecenter/en/SSEPEK_10.0.0/codes/src/tpc/n142.html )
SQLSTATE=42612,
--> Seems to be internal, nothing for my relevance
After getting more into the topic, I came across the following link, which appears as the most encouraging so far:
https://www.idug.org/p/fo/et/thread=36380
In summary, the suggested way is to extend org.eclipse.persistence.platform.database.DB2MainframePlatform and manipulate the generated SQL-statement.
My expectation is, that there should be a better way, since this should be a standard-case for the framework. If not, is there an more proper/better way to solve this problem?

Azure SQL : error code for Connection is closed exception's

I am getting below exceptions at times in my java code which connects to Azure SQL server. For this, I need to implement retry logic i.e. when I face below exception I will retry 1) Create new connection 2) Re-execute the SQL query 3) Commit the transaction.
But, I am unable to get the Azure SQL server error code for below error. Please let me know the error code ? I do not see the error code when I run below query:
SELECT * FROM sys.messages WHERE language_id = 1033
Exception:
com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:227)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.checkClosed(SQLServerConnection.java:796)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.commit(SQLServerConnection.java:2681)
Also, is it a good practice to use e.getMessage() and check if return string is "The connection is closed." and retry my logic ?
I do not see the error code in Error codes from SQL
This exception seems to be related to an attempt to convert from a supported data type to an unsupported data type. This is a good example.
Try another version of the JDBC driver.

Using galera mariadb with jdbc

I've created a mariadb cluster and I'm trying to get a Java application to be able to failover to another host when one of them dies.
I've created an application that creates a connection with "jdbc:mysql:sequential://host1,host2,host3/database?socketTimeout=2000&autoReconnect=true". The application makes a query in a loop every second. If I kill the node where the application is currently executing the query (Statement.executeQuery()) I get a SQLException because of a timeout. I can catch the exception and re-execute the statement and I see that the request is being sent to another server, so failover in that case works ok. But I was expecting that executeQuery() would not throw an exception and silently retry another server automatically.
Am I wrong in assuming that I shouldn't have to handle an exception and explicitely retry the query? Is there something more I need to configure for that to happen?
It is dangerous to auto reconnect for the following reason. Let's say you have this code:
BEGIN;
SELECT ... FROM tbl WHERE ... FOR UPDATE;
(line 3)
UPDATE tbl ... WHERE ...;
COMMIT;
Now let's say the server crashes at (line 3). The transaction will be rolled back. In my fabricated example, that only involves releasing the lock on tbl.
Now let's say that some other connection succeeds in performing the same transaction on the same row while you are auto-reconnecting.
Now, with auto-reconnect, the first thread is oblivious that the first half of the transaction was rolled back and proceeds to do the UPDATE based on data that is now out of date.
You need to get an exception so that you can go back to the BEGIN so that you can be "transaction safe".
You need this anyway -- With Galera, and no crashes, a similar thing could happen. Two threads performing that transaction on two different nodes at the same time... Each succeeds until it gets to the COMMIT, at which point the Galera magic happens and one of the COMMITs is told to fail. The 'right' response is replay the entire transaction on the server that was chosen for failure.
Note that Galera, unlike non-Galera, requires checking for errors on COMMIT.
More Galera tips (aimed at devs and dbas migrating from non-Galera)
Failover doesn't mean that application doesn't have to handle exceptions.
Driver will try to reconnect to another server when connection is lost.
If driver fail to reconnect to another server a SQLNonTransientConnectionException will be thrown, pools will automatically discard those connection.
If connection is recovered, there is some marginals cases where relaunching query is safe: when query is not in a transaction, and connection is currently in read-only mode (using Spring #Transactional(readOnly = false)) for example. For thoses cases, MariaDb java connection will then relaunch query automatically. In those particular cases, no exception will be thrown, and failover is transparent.
Driver cannot re-execute current query during a transaction.
Even without without transaction, if query is an UPDATE command, driver cannot know if the last request has been received by the database server and executed.
Then driver will send an SQLException (with SQLState begining by "25" = INVALID_TRANSACTION_STATE), and it's up to the application to handle those cases.

Teradata 3130 Error in JPA/Hiberate context

In batch script I use a loop to execute a bunch of sql (hql) against
a Teradata databse. After some iterations I receive the following error:
Teradata databse: 3130 Response limit exceeded
Now the documentation suggests (as well the answer on this question) that this is due to to many open result sets for the same session.
Now the session and the ResultSet are managed by the EntityManager, and I wonder if there is a way to avoid closing and reopening the connection in this case via jpa/hiberate.
I have tried entityManager.clear or flush without any effect.
is there a way to handle this better? maybe I am missing something. My "batch" runes under spring 2.5. in a "cli" mode.
in my case it turned out to be a row with large blob data. after refining steps I could retrieve data without 3130 popping out.

InvalidStateException while trying to enter data into DB

I have a method that return the entity manager for particular DB.Now when i use the method for the first time to get entity manager everything works fine.I can save data into any tables A,B,C using entity manager.Now say i get a exception while saving in table B
Now when I try to perform any operation on DB after geting exception above, the next time i try to run same code it fails when updating in table A itself.I can see folloing eception
<openjpa-1.2.2-SNAPSHOT-r422266:778978M-OPENJPA-975 nonfatal user error> org.apache.openjpa.persistence.InvalidStateException: The factory has been closed. The stack trace at which the factory was closed is available if Runtime=TRACE logging is enabled.
at org.apache.openjpa.kernel.AbstractBrokerFactory.assertOpen(AbstractBrokerFactory.java:673)
at org.apache.openjpa.kernel.AbstractBrokerFactory.newBroker(AbstractBrokerFactory.java:182)
at org.apache.openjpa.kernel.DelegatingBrokerFactory.newBroker(DelegatingBrokerFactory.java:142)
at org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:192)
at ..
Somewhere in your code you (or a framework) close your EntityManagerFactory. Make sure you didn't call close() on it.

Categories

Resources