I'm using Spring Batch for import items from XML to database. After import I create log with invalid records. I get these items by configuring skippable exceptions:
<batch:chunk reader="reader" processor="processor" writer="writer" commit-interval="10" skip-limit="99999999">
<batch:skippable-exception-classes>
<batch:include class="java.lang.Exception"/>
</batch:skippable-exception-classes>
<batch:listeners>
<batch:listener ref="recordSkipListener"/>
<batch:listener ref="itemReadListener"/>
</batch:listeners>
</batch:chunk>
My log after step execution:
Start time: 30.12.2015 01:05:38
End time: 30.12.2015 01:20:59
Read count: 3842
Skip count: 0
Write count: 3522
Skip count calculated by this expression:
int skipCount = stepExecution.getReadSkipCount() + stepExecution.getProcessSkipCount() + stepExecution.getWriteSkipCount();
"RecordSkipListener" handles all exceptions throwed in processor and writer, but it read, process and write methods are never called during step execution. Only ChunkListener.afterChunkError method is called, but its arguments doesn't contains info about failed record.
Therefore I have two questions:
How I can log failed items?
Why Spring Batch doesn't implement behavior same as behavior on ItemWriter.write:
failed chunk are divided onto chunks with size=1 and then processed on dedicated transaction?
// Update
Even if I decrease chunk-size to 1 then about 50 records are not written. And I cannot log their.
I'll answer your questions in reverse order:
Why Spring Batch doesn't implement behavior same as behavior on ItemWriter.write: failed chunk are divided onto chunks with size=1 and then processed on dedicated transaction?
When an exception is thrown during the process or write phase of the step, we can provide the item that caused the error because we have a valid item. The issue with an exception that occurs during the read phase is that since an exception has been thrown, we don't have an item to give to anyone. The only way a reader has to communicate what the input was that caused the exception is by embedding information about that input into the exception being thrown. An example of this is when an exception is thrown during the parsing of a line in the FlatFileItemReader, we throw a FlatFileParseException. That exception has both the line number and the contents of the line that caused the exception. You can use the ItemReadListener#onReadError method to catch the exception and obtain the information provided in the exception there.
How I can log failed items?
With XML, the ability to provide insight into what went wrong is difficult since it's not really within the reader's control (it's really a function of the unmarshaller and what went wrong there). The best you can do is to impute the bad record from the record count (via the ExecutionContext). That would tell you what number record caused the error within your XML file.
Issue of this problem is that at chunk commiting was exception: length of one of entity fields was exceeded allowed size. In this case processing and writing doesn't retry
Related
How do deal with h2 database inability to deal with interrupts, I was occasionally seeing that my embedded h2 database appeared to get corrupted, in particular I had amended an ExecutorService so that if a task took too long it would cancel the task. The task would be cancelled okay but then subsequent database access failed with exceptions such as
23/07/2019 14.23.31:BST:DeleteDuplicatesController:start:SEVERE: commit failed
org.hibernate.TransactionException: commit failed
at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.commit(AbstractTransactionImpl.java:187)
at com.jthink.songkong.db.ReportCache.save(ReportCache.java:46)
at com.jthink.songkong.reports.AbstractReport.setReportDatabaseObject(AbstractReport.java:365)
at com.jthink.songkong.reports.DeleteDuplicatesReport.setReportDatabaseObject(DeleteDuplicatesReport.java:333)
at com.jthink.songkong.reports.DeleteDuplicatesReport.closeReport(DeleteDuplicatesReport.java:377)
at com.jthink.songkong.analyse.toplevelanalyzer.DeleteDuplicatesController.deleteAnyDups(DeleteDuplicatesController.java:606)
at com.jthink.songkong.analyse.toplevelanalyzer.DeleteDuplicatesController.start(DeleteDuplicatesController.java:665)
at com.jthink.songkong.ui.swingworker.DeleteDuplicates.doInBackground(DeleteDuplicates.java:43)
at com.jthink.songkong.ui.swingworker.DeleteDuplicates.doInBackground(DeleteDuplicates.java:20)
at javax.swing.SwingWorker$1.call(SwingWorker.java:295)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at javax.swing.SwingWorker.run(SwingWorker.java:334)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.hibernate.TransactionException: unable to commit against JDBC connection
at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doCommit(JdbcTransaction.java:116)
at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.commit(AbstractTransactionImpl.java:180)
... 14 more
Caused by: org.h2.jdbc.JdbcSQLNonTransientException: General error: "java.lang.IllegalStateException: Reading from nio:C:/Users/Paul/AppData/Roaming/SongKong/Database/Database.mv.db failed; file length -1 read length 4096 at 1541494 [1.4.199/1]"; SQL statement:
COMMIT [50000-199]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:502)
at org.h2.message.DbException.getJdbcSQLException(DbException.java:427)
at org.h2.message.DbException.get(DbException.java:194)
at org.h2.message.DbException.convert(DbException.java:347)
at org.h2.command.Command.executeUpdate(Command.java:280)
at org.h2.jdbc.JdbcConnection.commit(JdbcConnection.java:542)
at com.mchange.v2.c3p0.impl.NewProxyConnection.commit(NewProxyConnection.java:1284)
at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doCommit(JdbcTransaction.java:112)
... 15 more
Caused by: java.lang.IllegalStateException: Reading from nio:C:/Users/Paul/AppData/Roaming/SongKong/Database/Database.mv.db failed; file length -1 read length 4096 at 1541494 [1.4.199/1]
at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:883)
at org.h2.mvstore.DataUtils.readFully(DataUtils.java:420)
at org.h2.mvstore.FileStore.readFully(FileStore.java:98)
at org.h2.mvstore.MVStore.readBufferForPage(MVStore.java:1048)
at org.h2.mvstore.MVStore.readPage(MVStore.java:2186)
at org.h2.mvstore.MVMap.readPage(MVMap.java:554)
at org.h2.mvstore.Page$NonLeaf.getChildPage(Page.java:1086)
at org.h2.mvstore.Page.get(Page.java:221)
at org.h2.mvstore.MVMap.get(MVMap.java:402)
at org.h2.mvstore.MVMap.get(MVMap.java:389)
at org.h2.mvstore.MVStore.getMapName(MVStore.java:2737)
at org.h2.mvstore.MVStore.renameMap(MVStore.java:2650)
at org.h2.mvstore.tx.TransactionStore.commit(TransactionStore.java:453)
at org.h2.mvstore.tx.Transaction.commit(Transaction.java:389)
at org.h2.engine.Session.commit(Session.java:691)
at org.h2.command.dml.TransactionCommand.update(TransactionCommand.java:46)
at org.h2.command.CommandContainer.update(CommandContainer.java:133)
at org.h2.command.Command.executeUpdate(Command.java:267)
... 18 more
Caused by: java.nio.channels.ClosedChannelException
at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:721)
at org.h2.store.fs.FileNio.read(FilePathNio.java:74)
at org.h2.mvstore.DataUtils.readFully(DataUtils.java:406)
... 34 more
23/07/2019 14.23.31:BST:Errors:addError:SEVERE: Adding Error:commit failed
I have since found this issue
Basically if using H2 in embedded mode, and it receives an interrupt then all subsequent access fails until the thread pool is close and reopened. In the example I give of a process having to be cancelled because it appears to be stuck there is no solution except for interrupting
I also have another case whereby usually the controller thread that doesn't directly do a database work itself so I was struggling to see why when an interrupt occurred why this would cause database errors since this is handled by controller thread. I have now worked out the issue is that Im using an ExecutorService with a fixed size BlockingQueue (so that we dont have a big queue build up in memory), but if the queue gets full then new task is actually executed by the controller thread (because of CallerRunsPolicy), so the controller thread can be making calls to database after all.
Im using H2 with hibernate and in both cases calling the following immediately after the interrupt
HibernateUtil.closeFactory();
seems to solve the issue, however I guess this means that any other threads with hibernate sessions will be broken, but at least newly opened sessins will be okay. So im not particularly happy with this workaround, any other ideas ?
Using H2 as a server is not a solution since the whole point of H2 was an embedded db self contained within application.
Although not properly documented using the async protocol allows a connection to be interrupted without breaking all other connections.
My web application runs on Spring (MVC) 4.2.9.RELEASE, Hibernate 5.1.3.Final, Spring Data 1.8.2.RELEASE, and MS SQL Server (2014).
In the Spring context, I have the following exceptioin hanlder:
<bean id="simpleMappingExceptionResolver" class="myproject.CustomMappingExceptionResolver">
...
</bean>
to catch and save stack trace. I am able to see the following deep in a long stack trace printed in the logs:
......
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136)
... 113 more
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 73) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:258)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1535)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:467)
How can I find the following exception class in the above exceptioin hanlder (and given an Exception instance) :
com.microsoft.sqlserver.jdbc.SQLServerException
AND the corresponding message:
Transaction (Process ID 73) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
If I understood you correctly you need to catch a nested exception. That's a bit tricky, but doable. You need to have a try-catch block for the top level exception that you expect. In the catch clause, you can use exception.getCause() to step down one nesting level at a time, and see if that level is an instanceof your SQL exception class. You can also check the message if necessary by using getMessage(). If the exception fits your criteria, congratulations you caught it. If not, simply throw it again.
Two things to keep in mind:
this approach may lead to poor performance if many exceptions occur, and only a small fraction of those actually matches your criteria.
if an exception has no cause, then e.getCause() will return e itself. Watch out for infinite recursion here.
I encountered the following problem when start running a hama BSP job. This exception occurs when hama tries to load and partition the input data before it actually runs my own code. This is a known problem discussed in some websites but unfortunate without a known cause (eg. see here).
My BSP job works perfectly ok when I only runs part of the data set. However, when I run the full data set, the problem occurs :(
Can I know how to resolve or avoid this problem?
13/11/18 01:19:30 INFO bsp.FileInputFormat: Total input paths to process : 32
13/11/18 01:19:30 INFO bsp.FileInputFormat: Total input paths to process : 32
13/11/18 01:19:30 INFO bsp.BSPJobClient: Running job: job_201311180115_0002
13/11/18 01:19:33 INFO bsp.BSPJobClient: Current supersteps number: 0
13/11/18 01:19:33 INFO bsp.BSPJobClient: Job failed.
13/11/18 01:19:33 ERROR bsp.BSPJobClient: Error partitioning the input path.
java.io.IOException: Runtime partition failed for the job.
at org.apache.hama.bsp.BSPJobClient.partition(BSPJobClient.java:465)
at org.apache.hama.bsp.BSPJobClient.submitJobInternal(BSPJobClient.java:333)
at org.apache.hama.bsp.BSPJobClient.submitJob(BSPJobClient.java:293)
at org.apache.hama.bsp.BSPJob.submit(BSPJob.java:228)
at org.apache.hama.bsp.BSPJob.waitForCompletion(BSPJob.java:235)
at edu.wisc.cs.db.opener.hama.ConnectedEntityBspDriver.main(ConnectedEntityBspDriver.java:183)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hama.util.RunJar.main(RunJar.java:146)
After stuck at this problem for several hours, I found that once the number of input files is greater than the number of allowed bsp tasks, then this error will occur. I think it is probably a bug that Hama should fix in the future.
A quick fix to this problem is to increase the number of maximum bsp tasks, specified by the variable bsp.tasks.maximum in the hama-site.xml file. For example, the following uses 10 instead of the default setting 3:
<property>
<name>bsp.tasks.maximum</name>
<value>10</value>
<description>The maximum number of BSP tasks that will be run simultaneously
by a groom server.</description>
</property>
Following is our cypher for a find by ID kind of service:
START n=node:PATIENTS('MEMBER_PLAN_ID:(1 2)') return n
Where 1 2 are the ids passed. When we pass around 2000 ids following error occurs:
java.lang.RuntimeException: org.apache.lucene.queryParser.ParseException: Cannot parse 'MEMBER_PLAN_ID:(1 2)': too many boolean clauses
at org.neo4j.index.impl.lucene.IndexType.query(IndexType.java:304)
at org.neo4j.index.impl.lucene.LuceneIndex.query(LuceneIndex.java:227)
at org.neo4j.index.impl.lucene.LuceneIndex.query(LuceneIndex.java:238)
at org.neo4j.cypher.internal.spi.gdsimpl.GDSBackedQueryContext$$anon$1.indexQuery(GDSBackedQueryContext.scala:87)
at org.neo4j.cypher.internal.executionplan.builders.IndexQueryBuilder$$anonfun$getNodeGetter$2.apply(IndexQueryBuilder.scala:83)
at org.neo4j.cypher.internal.executionplan.builders.IndexQueryBuilder$$anonfun$getNodeGetter$2.apply(IndexQueryBuilder.scala:81)
at org.neo4j.cypher.internal.pipes.StartPipe$$anonfun$internalCreateResults$1.apply(StartPipe.scala:36)
at org.neo4j.cypher.internal.pipes.StartPipe$$anonfun$internalCreateResults$1.apply(StartPipe.scala:35)
at scala.collection.Iterator$$anon$13.__AW_hasNext(Iterator.scala:371)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala)
at org.neo4j.cypher.internal.ClosingIterator$$anonfun$hasNext$1.apply$mcZ$sp(ClosingIterator.scala:36)
at org.neo4j.cypher.internal.ClosingIterator$$anonfun$hasNext$1.apply(ClosingIterator.scala:35)
at org.neo4j.cypher.internal.ClosingIterator$$anonfun$hasNext$1.apply(ClosingIterator.scala:35)
at org.neo4j.cypher.internal.ClosingIterator.failIfThrows(ClosingIterator.scala:86)
at org.neo4j.cypher.internal.ClosingIterator.hasNext(ClosingIterator.scala:35)
at org.neo4j.cypher.PipeExecutionResult.hasNext(PipeExecutionResult.scala:157)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:29)
at org.neo4j.cypher.PipeExecutionResult$$anon$1.hasNext(PipeExecutionResult.scala:73)
at net.ahm.graph.dao.PatientDAO.__AW_findPatients(PatientDAO.java:376)
Caused by: org.apache.lucene.queryParser.ParseException: Cannot parse 'MEMBER_PLAN_ID:(1 2)': too many boolean clauses
at org.apache.lucene.queryParser.QueryParser.parse(QueryParser.java:221)
at org.neo4j.index.impl.lucene.IndexType.query(IndexType.java:300)
... 38 more
Caused by: org.apache.lucene.search.BooleanQuery$TooManyClauses: maxClauseCount is set to 1024
at org.apache.lucene.search.BooleanQuery.add(BooleanQuery.java:136)
at org.apache.lucene.queryParser.QueryParser.getBooleanQuery(QueryParser.java:958)
at org.apache.lucene.queryParser.QueryParser.getBooleanQuery(QueryParser.java:933)
at org.apache.lucene.queryParser.QueryParser.Query(QueryParser.java:1281)
at org.apache.lucene.queryParser.QueryParser.Clause(QueryParser.java:1323)
at org.apache.lucene.queryParser.QueryParser.Query(QueryParser.java:1245)
at org.apache.lucene.queryParser.QueryParser.TopLevelQuery(QueryParser.java:1234)
at org.apache.lucene.queryParser.QueryParser.parse(QueryParser.java:206)
... 39 more
We see this in the stack trace: maxClauseCount is set to 1024
Is there a way to configure this limit from Neo4j while using Cypher?
However, if i add this line to our service, things work fine:
BooleanQuery.setMaxClauseCount(20000);
We did some load tests where 1000 concurrent users hit the service with 10,000 ids each. Not seeing service crash / any unexpected performance problems afterwards.
Try this (untested):
System.setProperty("org.apache.lucene.maxClauseCount", "3000");
I have a spring batch job which has multiple steps.
Step 1 : Loads 10 records from the database. (Tasklet does that job)
Step 2: Chunk Oriented processing in configured here using ItemReader,ItemProcessor,ItemWriter implementations with commit -interval =1
Now as I understand , for every record this would happen
Begin Transaction ( Read - Process - Write ) Commit Tx
My problem is imagine it processed six records and now with the 7th Record it got an exception in the ItemProcessor Implementation, It tries to rollback but not able to rollback due to transaction in unknown state
Even if it is not able to rollback tx for 7th record it does not processes 8th,9th,10th record at all and job is STOPPED.
Note: ItemProcessor implementation is calling services (#Service annotated) which are marked as transactional using #Transactional(readOnly=false) annotation.
Please suggest the solution.
ItemProcessor Code Below
public Long process(LoanApplication loanApplication)throws Exception {
Long createLoan = null;
LoanStatus loanStatus = null;
StaffUser user = staffUserService.getStaffUserByName(Constants.AUTO_USER);
String notes = null;
try{
try{
loanValidationService.validate(loanApplication);
loanStatus=LoanStatus.U;
}catch(LoanValidationException e){
loanStatus=LoanStatus.UC;
notes=e.getMessage();
}
dataLoadLoanManagementService.setText(notes);
createLoan = dataLoadLoanManagementService.createLoan(loanApplication, user,loanStatus);
}catch(Exception le){
logger.error("Error creating the loan application ; Parent Application Ref : "
+ loanApplication
+ " with status as " +(loanStatus!=null ?loanStatus.getStatus():loanStatus)
+"\n"
+" School Ref :"
+ loanApplication.getSchoolRef()
+"\n"
+" Client Details :"
+loanApplication.getClientDetails()
+ "Error Details: " + ExceptionUtils.getStackTrace(le));
}
return createLoan;
}
Even with Skippable Exception Classes configured this does not work.
To explain bit more i get Persistence Exception in Item Processor and i catch it , therefore Spring batch executes the writer but after writer execution i get the exception below
INFO 06-21 11:38:00 Commit failed while step execution data was already updated. Reverting to old version. (TaskletStep.java:342)
ERROR 06-21 11:38:00 Rolling back with transaction in unknown state (TaskletStep.java:351)
ERROR 06-21 11:38:00 Encountered an error executing the step (AbstractStep.java:212)
org.springframework.transaction.TransactionSystemException: Could not commit JPA transaction; nested exception is javax.persistence.RollbackException: Transaction marked as rollbackOnly
For both of your questions, skipping the exception occured during the prosessor phase will solve your problems.
This may be configured by skippable-exception-classes element if you know the root cause of the exception. For example, if you get a divide by zero exception during the processor phase and do want to neglect it, a sample configuration might be:
<chunk reader="reader" processor="processor" writer="writer"
commit-interval="1" >
<skippable-exception-classes>
<include class="java.lang.ArithmeticException" />
</skippable-exception-classes>
</chunk>
Since the given exception class and its subclasses will be skipped, you may even try java.lang.Exception
check transaction propagation option at loanValidationService.validate
Guess from "Transaction marked as rollbackOnly", "Commit failed while step execution data was already updated"
Current transaction was rollbacked and parent Transaction should be commit but transaction already over.
if chunk's transaction is same at LoanValidationService.validate
change propagation option to REQUIRES_NEW
following article may help to understand metadata's transaction.
http://blog.codecentric.de/en/2012/03/transactions-in-spring-batch-part-1-the-basics/
class LoanValidationServiceImpl implements LoanValidationService {
#Trasnactional(propagation=REQUIRES_NEW, rollbackFor=Exception.class)
validate(LoanApplication loanApplication) {
// business logic
}
}