Develop Reference

How do deal with h2 databases inability to deal with interrupts

How do deal with h2 database inability to deal with interrupts, I was occasionally seeing that my embedded h2 database appeared to get corrupted, in particular I had amended an ExecutorService so that if a task took too long it would cancel the task. The task would be cancelled okay but then subsequent database access failed with exceptions such as
23/07/2019 14.23.31:BST:DeleteDuplicatesController:start:SEVERE: commit failed
org.hibernate.TransactionException: commit failed
at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.commit(AbstractTransactionImpl.java:187)
at com.jthink.songkong.db.ReportCache.save(ReportCache.java:46)
at com.jthink.songkong.reports.AbstractReport.setReportDatabaseObject(AbstractReport.java:365)
at com.jthink.songkong.reports.DeleteDuplicatesReport.setReportDatabaseObject(DeleteDuplicatesReport.java:333)
at com.jthink.songkong.reports.DeleteDuplicatesReport.closeReport(DeleteDuplicatesReport.java:377)
at com.jthink.songkong.analyse.toplevelanalyzer.DeleteDuplicatesController.deleteAnyDups(DeleteDuplicatesController.java:606)
at com.jthink.songkong.analyse.toplevelanalyzer.DeleteDuplicatesController.start(DeleteDuplicatesController.java:665)
at com.jthink.songkong.ui.swingworker.DeleteDuplicates.doInBackground(DeleteDuplicates.java:43)
at com.jthink.songkong.ui.swingworker.DeleteDuplicates.doInBackground(DeleteDuplicates.java:20)
at javax.swing.SwingWorker$1.call(SwingWorker.java:295)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at javax.swing.SwingWorker.run(SwingWorker.java:334)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.hibernate.TransactionException: unable to commit against JDBC connection
at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doCommit(JdbcTransaction.java:116)
at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.commit(AbstractTransactionImpl.java:180)
... 14 more
Caused by: org.h2.jdbc.JdbcSQLNonTransientException: General error: "java.lang.IllegalStateException: Reading from nio:C:/Users/Paul/AppData/Roaming/SongKong/Database/Database.mv.db failed; file length -1 read length 4096 at 1541494 [1.4.199/1]"; SQL statement:
COMMIT [50000-199]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:502)
at org.h2.message.DbException.getJdbcSQLException(DbException.java:427)
at org.h2.message.DbException.get(DbException.java:194)
at org.h2.message.DbException.convert(DbException.java:347)
at org.h2.command.Command.executeUpdate(Command.java:280)
at org.h2.jdbc.JdbcConnection.commit(JdbcConnection.java:542)
at com.mchange.v2.c3p0.impl.NewProxyConnection.commit(NewProxyConnection.java:1284)
at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doCommit(JdbcTransaction.java:112)
... 15 more
Caused by: java.lang.IllegalStateException: Reading from nio:C:/Users/Paul/AppData/Roaming/SongKong/Database/Database.mv.db failed; file length -1 read length 4096 at 1541494 [1.4.199/1]
at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:883)
at org.h2.mvstore.DataUtils.readFully(DataUtils.java:420)
at org.h2.mvstore.FileStore.readFully(FileStore.java:98)
at org.h2.mvstore.MVStore.readBufferForPage(MVStore.java:1048)
at org.h2.mvstore.MVStore.readPage(MVStore.java:2186)
at org.h2.mvstore.MVMap.readPage(MVMap.java:554)
at org.h2.mvstore.Page$NonLeaf.getChildPage(Page.java:1086)
at org.h2.mvstore.Page.get(Page.java:221)
at org.h2.mvstore.MVMap.get(MVMap.java:402)
at org.h2.mvstore.MVMap.get(MVMap.java:389)
at org.h2.mvstore.MVStore.getMapName(MVStore.java:2737)
at org.h2.mvstore.MVStore.renameMap(MVStore.java:2650)
at org.h2.mvstore.tx.TransactionStore.commit(TransactionStore.java:453)
at org.h2.mvstore.tx.Transaction.commit(Transaction.java:389)
at org.h2.engine.Session.commit(Session.java:691)
at org.h2.command.dml.TransactionCommand.update(TransactionCommand.java:46)
at org.h2.command.CommandContainer.update(CommandContainer.java:133)
at org.h2.command.Command.executeUpdate(Command.java:267)
... 18 more
Caused by: java.nio.channels.ClosedChannelException
at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:721)
at org.h2.store.fs.FileNio.read(FilePathNio.java:74)
at org.h2.mvstore.DataUtils.readFully(DataUtils.java:406)
... 34 more
23/07/2019 14.23.31:BST:Errors:addError:SEVERE: Adding Error:commit failed
I have since found this issue
Basically if using H2 in embedded mode, and it receives an interrupt then all subsequent access fails until the thread pool is close and reopened. In the example I give of a process having to be cancelled because it appears to be stuck there is no solution except for interrupting
I also have another case whereby usually the controller thread that doesn't directly do a database work itself so I was struggling to see why when an interrupt occurred why this would cause database errors since this is handled by controller thread. I have now worked out the issue is that Im using an ExecutorService with a fixed size BlockingQueue (so that we dont have a big queue build up in memory), but if the queue gets full then new task is actually executed by the controller thread (because of CallerRunsPolicy), so the controller thread can be making calls to database after all.
Im using H2 with hibernate and in both cases calling the following immediately after the interrupt
HibernateUtil.closeFactory();
seems to solve the issue, however I guess this means that any other threads with hibernate sessions will be broken, but at least newly opened sessins will be okay. So im not particularly happy with this workaround, any other ideas ?
Using H2 as a server is not a solution since the whole point of H2 was an embedded db self contained within application.

Although not properly documented using the async protocol allows a connection to be interrupted without breaking all other connections.

JConn4 sybase jdbc driver much slower than JConn2

Recently I upgraded from JConn2 Sybase driver to JConn4. My Java jdbc code repeatedly calls a proc. Each proc call is ~1000 times slower with JConn4. JConn2 the avg is ~2ms. With JConn4 it is ~2s.
The proc is called via CallableStatement.
Does anybody know if there are some connection properties that have changed with JConn4 that perhaps need to be set to eliminate this performance issue?
This is Linux environment.
I took the thread dump and all I see is traces like this:
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at com.sybase.jdbc4.timedio.RawDbio.reallyRead(RawDbio.java:404)
at com.sybase.jdbc4.timedio.Dbio.doRead(Dbio.java:257)
at com.sybase.jdbc4.timedio.InStreamMgr.readIfOwner(InStreamMgr.java:583)
at com.sybase.jdbc4.timedio.InStreamMgr.doRead(InStreamMgr.java:319)
at com.sybase.jdbc4.tds.TdsProtocolContext.getChunk(TdsProtocolContext.java:622)
at com.sybase.jdbc4.tds.PduInputFormatter.readPacket(PduInputFormatter.java:239)
at com.sybase.jdbc4.tds.PduInputFormatter.read(PduInputFormatter.java:72)
at com.sybase.jdbc4.tds.TdsInputStream.read(TdsInputStream.java:91)
at com.sybase.jdbc4.tds.TdsInputStream.readUnsignedByte(TdsInputStream.java:124)
at com.sybase.jdbc4.tds.Tds.nextResult(Tds.java:2912)
at com.sybase.jdbc4.jdbc.ResultGetter.nextResult(ResultGetter.java:78)
at com.sybase.jdbc4.jdbc.SybStatement.nextResult(SybStatement.java:293)
at com.sybase.jdbc4.jdbc.SybStatement.nextResult(SybStatement.java:275)
at com.sybase.jdbc4.jdbc.SybStatement.executeLoop(SybStatement.java:2827)
at com.sybase.jdbc4.jdbc.SybCallableStatement.execute(SybCallableStatement.java:244)

The primary change between Jconn2 and Jconn4 is the default setting of the DYNAMIC_PREPARE parameter. Try setting it to FALSE in the connect parameters.
This page tells how to change the DYNAMIC_PREPARE parameter.

beanshell - Deadlock issue

Has anyone got any experience of having deadlocks with beanshell? This is something we have been encountering recently in our production system where script execution is blocking other threads, due to it's lock on classloading via tomcat. The following is the stacktrace for the lock owner in thread dump:
"Thread-64" : 150 : BLOCKED : cpu=37812500000 : cpuLoad= 0.0
BlockedCount:93354 BlockedTime:-1 LockName:java.lang.Object#219d66b6 LockOwnerID:151 LockOwnerName:Thread-65
WaitedCount:13 WaitedTime:-1 InNative:false IsSuspended:false at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntries(AbstractSingleArchiveResourceSet.java:66)
at org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:262)
at org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:281)
at org.apache.catalina.webresources.Cache.getResource(Cache.java:62)
at org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:216)
at org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(StandardRoot.java:225)
at org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2173)
at org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:811)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1260)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119)
at java.lang.Class.forName0(Class.java:-2)
at java.lang.Class.forName(Class.java:348)
at bsh.classpath.ClassManagerImpl.classForName(null:-1)
at bsh.NameSpace.classForName(null:-1)
at bsh.NameSpace.getImportedClassImpl(null:-1)
at bsh.NameSpace.getClassImpl(null:-1)
at bsh.NameSpace.getClass(null:-1)
at bsh.Name.consumeNextObjectField(null:-1)
at bsh.Name.toObject(null:-1)
at bsh.BSHAmbiguousName.toObject(null:-1)
at bsh.BSHAmbiguousName.toObject(null:-1)
at bsh.BSHPrimaryExpression.eval(null:-1)
at bsh.BSHPrimaryExpression.eval(null:-1)
at bsh.BSHVariableDeclarator.eval(null:-1)
at bsh.BSHTypedVariableDeclaration.eval(null:-1)
at bsh.Interpreter.eval(null:-1)
at bsh.Interpreter.eval(null:-1)
at bsh.Interpreter.eval(null:-1)
at my.package.MyClassFile(MyClassFile:2332)
I see that Groovy is a more popular choice for Java scripting, but I haven't seen many posts where it says that bsh can cause deadlocks.
It would be good to get some ideas from SO users.
Regards,

There's a fix for one dead lock in GUI does not start in Java 8 found in Beanshell (almost latest) version 2.0b5.
You can open a new issue in Beanshell project.
It may be connected to ClassManagerImpl:
Bsh has a multi-tiered class loading architecture. No class loader is
created unless/until a class is generated, the classpath is modified,
or a class is reloaded.
Note: we may need some synchronization in here

Hazelcast - client mode topology / distributed map lock issue

Below is the description of problem we faced in production. Please note that I could not reproduce the issue in test or local environment and therfore can not provide you with test code.
We have a hazelcast cluster with two members M1, M2 and three clients C1,C2,C3. Hazelcast version is 3.9.
Clients use IMap.tryLock() method with timeout of 10 seconds. After getting the lock, critical and long running operations are performed and finally the lock is released using IMap.unlock() method.
The problem occured in production is as follows:
At some time instant t, we first saw heartbeat failure to M2 at client C2. Afterwards there are errors in fetching partition table casued by com.hazelcast.spi.exception.TargetDisconnectedException:
[hz.client_0.internal-2 ] WARN [] HeartbeatManager - hz.client_0 [mygroup] [3.9] HeartbeatManager failed to connection: .....
[hz.client_0.internal-3 ] WARN [] ClientPartitionService - hz.client_0 [mygroup] [3.9] Error while fetching cluster partition table!
java.util.concurrent.ExecutionException: com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out to owner connection ClientConnection{alive=true, connectionId=1, ......
Around 250 ms after initial heartbeat failure, client gets disconnected and then reconnects in 20 ms.
[hz.client_0.cluster- ] INFO [] LifecycleService - hz.client_0 [mygroup] [3.9] HazelcastClient 3.9 (20171023 - b29f549) is CLIENT_DISCONNETED
[hz.client_0.cluster- ] INFO [] LifecycleService - hz.client_0 [mygroup] [3.9] HazelcastClient 3.9 (20171023 - b29f549) is CLIENT_CONNECTED
The problem we are having is, for some keys that are previously acquired by C2, C1 and C3 can not acquire the lock even if it seems to be released by C2. C2 can get the lock, but this puts unacceptable delays
to the application and is not acceptable.. All clients should get since lock is released...
We were notified of the problem after receiving complaints, and then restarted the client application C2.
As documented in http://docs.hazelcast.org/docs/latest-development/manual/html/Distributed_Data_Structures/Lock.html, locks acquired by restarted member (C2 in my case) seemed to be removed after restart operation.
Currently the issue seems to go away, but we are not sure if it will recur.
Do you have any suggestions about the probable cause and more importantly do you have any recommendations?
Would enabling redo-operation in client help for this problem case?
As I tried to explain client seems to recover the problem, but keys remain locked in cluster and this is fatal to my application.
Thanks

It looks like the client had lost the ownership of the lock because of its disconnection from the cluster. You can use IMap#forceUnlock API in cases such as you faced. It releases the lock regardless of the lock owner and it always successfully unlocks, never blocks, and returns immediately.

c3p0 hangs in awaitAvailable with hibernate

I have console application that hangs during execution. Here is my configuration:
cfg.setProperty("hibernate.connection.driver_class", "com.mysql.jdbc.Driver");
cfg.setProperty("hibernate.connection.url", "jdbc:mysql://localhost:3306/db?user=db&password=db");
cfg.setProperty("hibernate.connection.username", "db");
cfg.setProperty("hibernate.connection.password", "db");
cfg.setProperty("hibernate.connection.pool_size", "5");
cfg.setProperty("hibernate.connection.autocommit", "false");
cfg.setProperty("hibernate.c3p0.min_size", "5");
cfg.setProperty("hibernate.c3p0.max_size", "20");
cfg.setProperty("hibernate.c3p0.timeout", "300");
cfg.setProperty("hibernate.c3p0.max_statements", "50");
cfg.setProperty("hibernate.c3p0.idle_test_period", "3000");
Here is my stacktrace:
"main" prio=10 tid=0x000000000168f800 nid=0x1c37 in Object.wait() [0x00007fa60d0ad000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007400f4c68> (a com.mchange.v2.resourcepool.BasicResourcePool)
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1315)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
- locked <0x00000007400f4c68> (a com.mchange.v2.resourcepool.BasicResourcePool)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:84)
at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:281)
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:297)
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:169)
at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:67)
at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:160)
at org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1392)
at org.kriyak.parser.IndexArchiveRapid.indexFile(IndexArchiveRapid.java:70)
at org.kriyak.parser.IndexArchiveRapid.main(IndexArchiveRapid.java:53)
I open only one conencton and it doesn't seem that I leak them. And also I use one thread. I haven't adjusted any mysql settings except memory usage. Mysql works fine from console. Why can this happen? Is this c3p0 error?

does this happen immediately, or after a while? that is, do checkouts initially succeed, but then hang like this? if so, it looks like a Connection leak. please try setting c3p0 params unreturnedConnectionTimeout and debugUnreturnedConnectionStackTraces to see if there is a leak. See
http://www.mchange.com/projects/c3p0/#configuring_to_debug_and_workaround_broken_clients , http://www.mchange.com/projects/c3p0/#unreturnedConnectionTimeout , http://www.mchange.com/projects/c3p0/#debugUnreturnedConnectionStackTraces .
if this happens immediately, if no Connections are successfully checked out, the question is whether the pool ever succeeds at acquiring Connections. by default, if it never does succeed, after about 30 seconds your thread should break with a failure. (it doesn't look like you've done this, but if for example you'd set acquireRetryAttempts to zero, c3p0 might hang indefinitely waiting for Connections.)
to debug c3p0 issues, it's helpful to capture the version and config information that c3p0 dumps to logs at INFO level on pool initialization.
good luck!

Also, you haven't seem to initialize the checkoutTime parameter for c3p0, which specifies the amount of time a client should wait for acquiring a connection from the connection pool.
see http://www.mchange.com/projects/c3p0/#checkoutTimeout

You may need to increase c3p0.numHelperThreads. The helper threads are responsible for creating new database connections and adding them to the pool. If there aren't enough threads to keep up with demand then application threads will be stuck waiting in awaitAvailable(). To confirm this is the case look at all the HelperThreads and see if they are all in use and in the process of connecting to the database.