Develop Reference

Spark Error: I/O error constructing remote block reader. java.nio.channels.ClosedByInterruptException at java.nio.channels.ClosedByInterruptException

The execution was ok locally in unit test, but fails when the Spark Streaming execution is propagated to the real cluster executors, like they silently crash and no longer available for the context:
stream execution thread for kafkaDataGeneratorInactiveESP_02/Distance [id = 438f45a0-acd6-4729-953f-5a18ae208f1f, runId = a98c6d39-fe14-4ed5-b7fe-7e4009de51b2]] impl.BlockReaderFactory (BlockReaderFactory.java:getRemoteBlockReaderFromTcp(765)) - I/O error constructing remote block reader.
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:656)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:533)
at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:2940)
at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:822)
at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:747)
at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:380)
at org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:644)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:575)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:757)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:829)
at java.io.DataInputStream.read(DataInputStream.java:149)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:74)
at org.apache.spark.sql.execution.streaming.CommitLog.deserialize(CommitLog.scala:56)
at org.apache.spark.sql.execution.streaming.CommitLog.deserialize(CommitLog.scala:48)
at org.apache.spark.sql.execution.streaming.HDFSMetadataLog.get(HDFSMetadataLog.scala:153)
at org.apache.spark.sql.execution.streaming.HDFSMetadataLog.$anonfun$getLatest$2(HDFSMetadataLog.scala:190)
at scala.runtime.java8.JFunction1$mcVJ$sp.apply(JFunction1$mcVJ$sp.java:23)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofLong.foreach(ArrayOps.scala:258)
at org.apache.spark.sql.execution.streaming.HDFSMetadataLog.getLatest(HDFSMetadataLog.scala:189)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.populateStartOffsets(MicroBatchExecution.scala:300)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$2(MicroBatchExecution.scala:194)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:352)
at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:350)
at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:69)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$1(MicroBatchExecution.scala:191)
at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:57)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:185)
at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:334)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:245)
First thing I have tried was changing the query name, having slash and space: kafkaDataGeneratorInactiveESP_02/Distance
After repacing it to the correct one in
.queryName("kafkaDataGeneratorInactive" + currentIter.metadata.getString("label"))
100% proven from having / or space in the string, the error hasn't gone.

The reason of the failure was actually in using the same name for the query name and checkpoint location path (but not in the part that was attempteed to be improved for the first time). Later I have fount one more error log:
2021-12-01 15:05:46,906 WARN [main] streaming.StreamingQueryManager (Logging.scala:logWarning(69)) - Stopping existing streaming query [id=b13a69d7-5a2f-461e-91a7-a9138c4aa716, runId=9cb31852-d276-42d8-ade6-9839fa97f85c], as a new run is being started.
Why the query was stopped? That's because in Scala I was creating streaming queries in a loop, iterating the collection, while keeping all the query names and all the checkpoint names the same. After making them unique (i just used the string values from the collection), the failure problem has gone.

How do deal with h2 databases inability to deal with interrupts

How do deal with h2 database inability to deal with interrupts, I was occasionally seeing that my embedded h2 database appeared to get corrupted, in particular I had amended an ExecutorService so that if a task took too long it would cancel the task. The task would be cancelled okay but then subsequent database access failed with exceptions such as
23/07/2019 14.23.31:BST:DeleteDuplicatesController:start:SEVERE: commit failed
org.hibernate.TransactionException: commit failed
at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.commit(AbstractTransactionImpl.java:187)
at com.jthink.songkong.db.ReportCache.save(ReportCache.java:46)
at com.jthink.songkong.reports.AbstractReport.setReportDatabaseObject(AbstractReport.java:365)
at com.jthink.songkong.reports.DeleteDuplicatesReport.setReportDatabaseObject(DeleteDuplicatesReport.java:333)
at com.jthink.songkong.reports.DeleteDuplicatesReport.closeReport(DeleteDuplicatesReport.java:377)
at com.jthink.songkong.analyse.toplevelanalyzer.DeleteDuplicatesController.deleteAnyDups(DeleteDuplicatesController.java:606)
at com.jthink.songkong.analyse.toplevelanalyzer.DeleteDuplicatesController.start(DeleteDuplicatesController.java:665)
at com.jthink.songkong.ui.swingworker.DeleteDuplicates.doInBackground(DeleteDuplicates.java:43)
at com.jthink.songkong.ui.swingworker.DeleteDuplicates.doInBackground(DeleteDuplicates.java:20)
at javax.swing.SwingWorker$1.call(SwingWorker.java:295)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at javax.swing.SwingWorker.run(SwingWorker.java:334)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.hibernate.TransactionException: unable to commit against JDBC connection
at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doCommit(JdbcTransaction.java:116)
at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.commit(AbstractTransactionImpl.java:180)
... 14 more
Caused by: org.h2.jdbc.JdbcSQLNonTransientException: General error: "java.lang.IllegalStateException: Reading from nio:C:/Users/Paul/AppData/Roaming/SongKong/Database/Database.mv.db failed; file length -1 read length 4096 at 1541494 [1.4.199/1]"; SQL statement:
COMMIT [50000-199]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:502)
at org.h2.message.DbException.getJdbcSQLException(DbException.java:427)
at org.h2.message.DbException.get(DbException.java:194)
at org.h2.message.DbException.convert(DbException.java:347)
at org.h2.command.Command.executeUpdate(Command.java:280)
at org.h2.jdbc.JdbcConnection.commit(JdbcConnection.java:542)
at com.mchange.v2.c3p0.impl.NewProxyConnection.commit(NewProxyConnection.java:1284)
at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doCommit(JdbcTransaction.java:112)
... 15 more
Caused by: java.lang.IllegalStateException: Reading from nio:C:/Users/Paul/AppData/Roaming/SongKong/Database/Database.mv.db failed; file length -1 read length 4096 at 1541494 [1.4.199/1]
at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:883)
at org.h2.mvstore.DataUtils.readFully(DataUtils.java:420)
at org.h2.mvstore.FileStore.readFully(FileStore.java:98)
at org.h2.mvstore.MVStore.readBufferForPage(MVStore.java:1048)
at org.h2.mvstore.MVStore.readPage(MVStore.java:2186)
at org.h2.mvstore.MVMap.readPage(MVMap.java:554)
at org.h2.mvstore.Page$NonLeaf.getChildPage(Page.java:1086)
at org.h2.mvstore.Page.get(Page.java:221)
at org.h2.mvstore.MVMap.get(MVMap.java:402)
at org.h2.mvstore.MVMap.get(MVMap.java:389)
at org.h2.mvstore.MVStore.getMapName(MVStore.java:2737)
at org.h2.mvstore.MVStore.renameMap(MVStore.java:2650)
at org.h2.mvstore.tx.TransactionStore.commit(TransactionStore.java:453)
at org.h2.mvstore.tx.Transaction.commit(Transaction.java:389)
at org.h2.engine.Session.commit(Session.java:691)
at org.h2.command.dml.TransactionCommand.update(TransactionCommand.java:46)
at org.h2.command.CommandContainer.update(CommandContainer.java:133)
at org.h2.command.Command.executeUpdate(Command.java:267)
... 18 more
Caused by: java.nio.channels.ClosedChannelException
at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:721)
at org.h2.store.fs.FileNio.read(FilePathNio.java:74)
at org.h2.mvstore.DataUtils.readFully(DataUtils.java:406)
... 34 more
23/07/2019 14.23.31:BST:Errors:addError:SEVERE: Adding Error:commit failed
I have since found this issue
Basically if using H2 in embedded mode, and it receives an interrupt then all subsequent access fails until the thread pool is close and reopened. In the example I give of a process having to be cancelled because it appears to be stuck there is no solution except for interrupting
I also have another case whereby usually the controller thread that doesn't directly do a database work itself so I was struggling to see why when an interrupt occurred why this would cause database errors since this is handled by controller thread. I have now worked out the issue is that Im using an ExecutorService with a fixed size BlockingQueue (so that we dont have a big queue build up in memory), but if the queue gets full then new task is actually executed by the controller thread (because of CallerRunsPolicy), so the controller thread can be making calls to database after all.
Im using H2 with hibernate and in both cases calling the following immediately after the interrupt
HibernateUtil.closeFactory();
seems to solve the issue, however I guess this means that any other threads with hibernate sessions will be broken, but at least newly opened sessins will be okay. So im not particularly happy with this workaround, any other ideas ?
Using H2 as a server is not a solution since the whole point of H2 was an embedded db self contained within application.

Although not properly documented using the async protocol allows a connection to be interrupted without breaking all other connections.

JRocket : Thread Stuck at jrockit/vm/Locks.park0

Seeing very strange behaviour. My code is executing well but not sure what happen, method is calling to other method but other method doesnt get called ( i cant see logs which is there in the first line of other method )
"jaxws-engine-1-thread-2" id=447 idx=0x73c tid=4031 prio=5 alive, parked, native_blocked, daemon
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2230)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:196)
at java/util/concurrent/SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at java/util/concurrent/SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java/util/concurrent/SynchronousQueue.poll(SynchronousQueue.java:874)
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:955)
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
at java/lang/Thread.run(Thread.java:682)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
Code -
public static void startMicroSessionTimer(TimerName timerName, Data Data) {
logger.debug("Starting a micro-timer for timer name: " + timerName);
//Start a micro timer to process the soap response in worker thread
SipApplicationSession applicationSession = Util.getAppSession((String)Data.get(DataAttribute.ID));
Util. AbcTimer (applicationSession, 1L, timerName.getTimerName());
}
public static void AbcTimer(SipApplicationSession appSession,
long timeInMillies, String timerName) {
logger.debug("Inside AbcTimer”);
//Some Logic
}
Logs -
16 May 2018 09:13:07,506 [jaxws-engine-1-thread-12] DEBUG -----SOME LOGS…..
16 May 2018 09:13:07,506 [jaxws-engine-1-thread-12] DEBUG [AbcUtils] [ODhlNjQ0ZjAzMTMzN2U5MGNhMTE2MTgxOTg2MTdmYjA.] Starting a micro-timer for timer name: HAHAHA
Not able to see any log after above line for Thread jaxws-engine-1-thread-12. As per log this log Inside AbcTimer should come as it is in the starting of called method ie AbcTimer. There is no Exception occured.
I have taken ThreadDump as well which I have posted above.
Not Sure but think that it is a machine specific issue. Also google it and saw that this type of issue occurred to other people as well but i didnt get the solution.
Using below JRocket Version
java version "1.6.0_141"
Java(TM) SE Runtime Environment (build 1.6.0_141-b12)
Oracle JRockit(R) (build R28.3.13-15-173128-1.6.0_141-20161219-1845-linux-x86_64, compiled mode)

Selenium Marionette new connections will no longer be accepted

When i'm starting my java program i'm getting this error message:
1494797744010 geckodriver INFO Listening on 127.0.0.1:5968
1494797744672 geckodriver::marionette INFO Starting browser \\?\C:\Program Files\Mozilla Firefox\firefox.exe with args ["-marionette"]
1494797744951 addons.manager ERROR startup failed: [Exception... "Component returned failure code: 0x80070057 (NS_ERROR_ILLEGAL_VALUE) [nsIFile.create]" nsresult: "0x80070057 (NS_ERROR_ILLEGAL_VALUE)" location: "JS frame :: resource://gre/modules/FileUtils.jsm J::a FvialSecUrtiiplts _egrertoDri:r r:e:s oluirncee 70://gre/modul"e s /dAadtdao:n Mnaon]a gSetra.cjks mt,r alcien:e F165i7l:e UNtSi_lEsR_RgOeRt_DNiOrT(_)I#NrIeTsIoAuLrIcZeE:D/:/ gAre/moddduolneMsa/nFaigleerU tiisl sn.ojts mi:n7i0t i<a lFiizleedU
tils_getFile()#resource://gre/modules/FileUtils.jsm:42 < validateBlocklist()#resource://gre/modules/AddonManager.jsm:671 < startup()#resource://gre/modules/AddonManager.jsm:834 < startup()#resource://gre/modules/AddonManager.jsm:3129 < observe()#resource://gre/components/addonManager.js:65
1494797746182 Marionette INFO Listening on port 50911
JavaScript error: resource://gre/modules/AddonManager.jsm, line 2570: NS_ERROR_NOT_INITIALIZED: AddonManager is not initialized
maj 14, 2017 11:35:46 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
While program is running it appears some new errors:
JavaScript error: resource://gre/modules/FileUtils.jsm, line 70: NS_ERROR_ILLEGAL_VALUE: Component returned failure code: 0x80070057 (NS_ERROR_ILLEGAL_VALUE) [nsIFile.create]
JavaScript error: resource://app/modules/WindowsJumpLists.jsm, line 403: NS_ERROR_FILE_NOT_FOUND: Component returned failure code: 0x80520012 (NS_ERROR_FILE_NOT_FOUND) [nsIJumpListShortcut.app]
JavaScript error: resource://app/modules/WindowsJumpLists.jsm, line 403: NS_ERROR_FILE_NOT_FOUND: Component returned failure code: 0x80520012 (NS_ERROR_FILE_NOT_FOUND) [nsIJumpListShortcut.app]
JavaScript error: https://apis.google.com/_/scs/apps-static/_/js/k=oz.gapi.pl.LabDolPUOAM.O/m=gapi_iframes,gapi_iframes_style_common/rt=j/sv=1/d=1/ed=1/am=AQ/rs=AGLTcCNgJpGnsPOpCNGmcIvZ9rnqApDcrQ/cb=gapi.loaded_0, line 52: TypeError: a is null
JavaScript error: https://apis.google.com/_/scs/apps-static/_/js/k=oz.gapi.pl.LabDolPUOAM.O/m=gapi_iframes,gapi_iframes_style_common/rt=j/sv=1/d=1/ed=1/am=AQ/rs=AGLTcCNgJpGnsPOpCNGmcIvZ9rnqApDcrQ/cb=gapi.loaded_0, line 52: TypeError: a is null
At the end im getting this information and the program stops working:
1494798016809 Marionette INFO New connections will no longer be accepted
What could be the reason for that error and how to fix it?
I'm using Firefox 53.0.2 and geckodriver 0.16.1

Here is the Answer to your Question:
The first set of error what you are seeing as 1494797744951 addons.manager ERROR startup failed: [Exception... "Component returned failure code: 0x80070057 (NS_ERROR_ILLEGAL_VALUE) is because you havn't configured the geckodriver to suppress/ignore those errors on console. But they are harmless and if not ignored/suppressed you should be fine living with them.
The second set of error you see JavaScript error: resource://gre/modules/FileUtils.jsm are JavaScript errors which again reflects that you haven't configured the geckodriver properly either to ignore or suppress the JavaScript errors from appearing on the console. Again they are harmless and if not ignored/suppressed you should be fine living with them.
The last set of error you see New connections will no longer be accepted is something serious Marionette complaining about which essentially means you are trying a test step which can't be achieved by the geckodriver. Hence you may need to change your code for the geckodriver to procedd with the task.
Let me know if this Answers your Question.

"heros.solver.CountingThreadPoolExecutor - Worker thread execution failed: null Exceptions" when running nightly build soot-trunk

I was converting apk file to jimple files and then converting they back to the .dex file immediately. But I cannot do the second the step successfully.
Command line used:
java -Xmx4g -jar soot-trunk.jar soot.Main -p cg.spark verbose:true,on-fly-cg:true -w -allow-phantom-refs -force-android-jar /home/xia/Downloads/android-platforms-master/android-17 -src-prec apk -f jimple -process-dir com.halfbrick.fruitninjafree.apk
In the beginning, it keeps on throwing this exception:
java.lang.RuntimeException: Error parsing class com.google.android.gms.plus.model.moments.MomentBuffer [22,40] expecting: quoted name, identifier
at soot.JimpleClassSource.resolve(JimpleClassSource.java:61)
at soot.SootResolver.bringToHierarchy(SootResolver.java:239)
at soot.SootResolver.bringToSignatures(SootResolver.java:266)
at soot.SootResolver.bringToBodies(SootResolver.java:304)
at soot.SootResolver.processResolveWorklist(SootResolver.java:163)
at soot.SootResolver.resolveClass(SootResolver.java:131)
at soot.Scene.loadClass(Scene.java:725)
at soot.Scene.loadClassAndSupport(Scene.java:710)
at soot.Scene.loadNecessaryClasses(Scene.java:1448)
at soot.Main.run(Main.java:243)
at soot.Main.main(Main.java:147)
Caused by: soot.jimple.parser.parser.ParserException: [22,40] expecting: quoted name, identifier
at soot.jimple.parser.parser.Parser.parse(Parser.java:1454)
at soot.jimple.parser.JimpleAST.(JimpleAST.java:57)
at soot.JimpleClassSource.resolve(JimpleClassSource.java:42)
... 10 more
I found that there are some syntax mistakes (might be) in the converted jimple file. For example, in the jimple files, there are some classes named like this:
com.google.android.gms.internal.'if'
java.lang.'annotation'.Annotation
Then I fixed the mistakes manually (by remove single quote and replace 'if' with other variable name like iff).
After I fixed the above mistakes, it shows another exception:
Warning: android.content.OperationApplicationException is a phantom class!
Warning: android.database.MatrixCursor$RowBuilder is a phantom class!
Warning: android.content.ContentProviderResult is a phantom class!
[Call Graph] For information on where the call graph may be incomplete, use the verbose option to the cg phase.
[Thread-1] ERROR heros.solver.CountingThreadPoolExecutor - Worker thread execution failed: null
[Thread-8] ERROR heros.solver.CountingThreadPoolExecutor - Worker thread execution failed: null
[Thread-7] ERROR heros.solver.CountingThreadPoolExecutor - Worker thread execution failed: null
java.lang.NullPointerException
at soot.toolkits.graph.UnitGraph.(UnitGraph.java:76)
at soot.toolkits.graph.ExceptionalUnitGraph.(ExceptionalUnitGraph.java:158)
at soot.jimple.toolkits.scalar.UnreachableCodeEliminator.internalTransform(UnreachableCodeEliminator.java:79)
at soot.BodyTransformer.transform(BodyTransformer.java:51)
at soot.Transform.apply(Transform.java:105)
at soot.JimpleBodyPack.applyPhaseOptions(JimpleBodyPack.java:61)
at soot.JimpleBodyPack.internalApply(JimpleBodyPack.java:95)
at soot.Pack.apply(Pack.java:125)
at soot.jimple.JimpleMethodSource.getBody(JimpleMethodSource.java:49)
at soot.SootMethod.getBodyFromMethodSource(SootMethod.java:91)
at soot.SootMethod.retrieveActiveBody(SootMethod.java:322)
at soot.PackManager$3.run(PackManager.java:1223)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[Thread-6] ERROR heros.solver.CountingThreadPoolExecutor - Worker thread execution failed: null
java.lang.NullPointerException
at soot.toolkits.graph.UnitGraph.(UnitGraph.java:76)
at soot.toolkits.graph.ExceptionalUnitGraph.(ExceptionalUnitGraph.java:158)
at soot.jimple.toolkits.scalar.UnreachableCodeEliminator.internalTransform(UnreachableCodeEliminator.java:79)
at soot.BodyTransformer.transform(BodyTransformer.java:51)
at soot.Transform.apply(Transform.java:105)
at soot.JimpleBodyPack.applyPhaseOptions(JimpleBodyPack.java:61)
...
At first I think it's the bug of the tool, but the answer of other question says that the bug has been fixed. But I still get this problem. Does any one know where the problem is? Thank you very much.