Develop Reference

Gremlin Java client throws a java.util.concurrent.TimeoutException

We have a AWS Neptune db.r4.large instance which with 2vCPUs can run 4 threads and hence 4 queries maximum at a time. We use the Gremlin Java client to connect to AWS Neptune. We are seeing regular persistent java.util.concurrent.TimeoutException exceptions while querying data.
Reading up on this AWS doc: https://docs.aws.amazon.com/neptune/latest/userguide/best-practices-gremlin-java-exceptions.html
It looks to be a case of client side throttling since we do not see any utilization of the Neptune queue with MainRequestQueuePendingRequests = 0. We did as recommended in the post (code below). With maxConnectionPoolSize = maxSimultaneousUsagePerConnection = maxInProcessPerConnection all set to 64, we have maxParallelQueries = 4096. Given our db instance and query latency, this seems like an overkill but we hoped we would at least shift some of these errors on to AWS Neptune to see the queue being utilized and see some throttling exceptions. However, all the requests that reach Neptune are processed correctly. The problem remains that almost half the requests never reach AWS Neptune and error out on the client side.
Other relevant metrics are the maxWaitForConnection = 16s (default) and the
avg(aws.neptune.gremlin_web_socket_open_connections) = ~10 (pretty steady). Anyone has any suggestions how we could resolve these exceptions?
#Bean("gremlinClusterRW", destroyMethod = "close")
fun gremlinClusterRW(): Cluster {
return Cluster.build()
.addContactPoint(endpointRW)
.port(port)
.channelizer(SigV4WebSocketChannelizer::class.java)
.maxConnectionPoolSize(64)
.maxInProcessPerConnection(64)
.maxSimultaneousUsagePerConnection(64)
.enableSsl(true)
.create()
}
Exception Stacktrace:
org.springframework.kafka.listener.ListenerExecutionFailedException: Listener method 'public void com.listeners.a.b(org.apache.kafka.clients.consumer.ConsumerRecord<java.lang.Long, a>)' threw exception; nested exception is java.lang.IllegalStateException: org.apache.tinkerpop.gremlin.process.remote.RemoteConnectionException: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: Timed out while waiting for an available host - check the client configuration and connectivity to the server if this message persists; nested exception is java.lang.IllegalStateException: org.apache.tinkerpop.gremlin.process.remote.RemoteConnectionException: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: Timed out while waiting for an available host - check the client configuration and connectivity to the server if this message persists
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.decorateException(KafkaMessageListenerContainer.java:2114)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeErrorHandler(KafkaMessageListenerContainer.java:2106)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:2001)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeWithRecords(KafkaMessageListenerContainer.java:1928)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:1814)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:1531)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1178)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1075)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateException: org.apache.tinkerpop.gremlin.process.remote.RemoteConnectionException: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: Timed out while waiting for an available host - check the client configuration and connectivity to the server if this message persists
at
...
(redacted)
...
at jdk.internal.reflect.GeneratedMethodAccessor251.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:171)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:120)
at org.springframework.kafka.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:48)
at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:330)
at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:86)
at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:51)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:2069)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:2051)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:1988)
... 8 common frames omitted
Caused by: org.apache.tinkerpop.gremlin.process.remote.RemoteConnectionException: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: Timed out while waiting for an available host - check the client configuration and connectivity to the server if this message persists
at org.apache.tinkerpop.gremlin.driver.remote.DriverRemoteConnection.submitAsync(DriverRemoteConnection.java:227)
at org.apache.tinkerpop.gremlin.process.remote.traversal.step.map.RemoteStep.promise(RemoteStep.java:89)
... 36 common frames omitted
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: Timed out while waiting for an available host - check the client configuration and connectivity to the server if this message persists
at org.apache.tinkerpop.gremlin.driver.Client$AliasClusteredClient.submitAsync(Client.java:573)
at org.apache.tinkerpop.gremlin.driver.remote.DriverRemoteConnection.submitAsync(DriverRemoteConnection.java:225)
... 37 common frames omitted
Caused by: java.lang.RuntimeException: java.util.concurrent.TimeoutException: Timed out while waiting for an available host - check the client configuration and connectivity to the server if this message persists
at org.apache.tinkerpop.gremlin.driver.Client.submitAsync(Client.java:371)
at org.apache.tinkerpop.gremlin.driver.Client$AliasClusteredClient.submitAsync(Client.java:591)
at org.apache.tinkerpop.gremlin.driver.Client$AliasClusteredClient.submitAsync(Client.java:571)
... 38 common frames omitted
Caused by: java.util.concurrent.TimeoutException: Timed out while waiting for an available host - check the client configuration and connectivity to the server if this message persists
at org.apache.tinkerpop.gremlin.driver.Client$ClusteredClient.chooseConnection(Client.java:495)
at org.apache.tinkerpop.gremlin.driver.Client$AliasClusteredClient.chooseConnection(Client.java:630)
at org.apache.tinkerpop.gremlin.driver.Client.submitAsync(Client.java:366)
... 40 common frames omitted

java connection from oracle database to oracle essbase server is breaking after 5 minutes i.e. 300 seconds

Facing one strange issue, we are connecting from oracle database to Oracle Essbase Server using JAPI connection (its http connection and no jdbc).
We are able to execute the program till 300 seconds then we get below error: -
invokeMethod localException11111 ::: java.net.SocketTimeoutException: recv() timed out
Resource temporarily unavailable
java.net.SocketTimeoutException: recv() timed out
Resource temporarily unavailable
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:740)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:683)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1280)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379)
at com.essbase.api.session.EssOrbPluginHttp.invokeMethod(EssOrbPluginHttp.java:139)
at com.essbase.api.session.EssOrbPluginHttp.invokeMethod(EssOrbPluginHttp.java:109)
at com.essbase.api.session.EssOrbPlugin.essMainCalc(Unknown Source)
at com.essbase.api.datasource.EssCube.calcWithRunTimeSubVars(Unknown Source)
at com.essbase.api.datasource.EssCube.calculate(Unknown Source)
at com.ing.mass.essconnect.Database.calculate(Database.java:314)
at com.ing.mass.essconnect.Database.calculate(Database.java:300)
at com.ing.mass.services.Cubes.calculateNow(Cubes.java:1704)
at com.ing.mass.services.Cubes.calculate(Cubes.java:1510)
invokeMethod finally :::
TESTING .....com.essbase.api.base.EssException: Cannot connect to Provider Server. java.net.SocketTimeoutException: recv() timed out
Resource temporarily unavailable
at com.essbase.api.session.EssOrbPl uginHttp.invokeMethod(EssOrbPluginHttp.java:244)
at com.essbase.api.session.EssOrbPluginHttp.invokeMethod(EssOrbPluginHttp.java:109)
at com.essbase.api.session.EssOrbPlugin.essMainCalc(Unknown Source)
at com.essbase.api.datasource.EssCube.calcWithRunTimeSubVars(Unknown Source)
at com.essbase.api.datasource.EssCube.calculate(Unknown Source)
at com.ing.mass.essconnect.Database.calculate(Database.java:314)
at com.ing.mass.essconnect.Database.calculate(Database.java:300)
at com.ing.mass.services.Cubes.calculateNow(Cubes.java:1704)
at com.ing.mass.services.Cubes.calculate(Cubes.java:1510)
Calculation completed
Calculation completed 111
Cubes calculateNow method finallly
It has been a week scratching our heads to resolve this issue but no luck so far.
Looking forward to hear back with your expert advise.
thanks in advance
Pankaj

this issue is now resolved -
problem was with new java code which required to add a timeout parameter like this
sun.net.client.defaultReadTimeout (set in miliseconds)

Jhipster Registry 4.0.0 exception pre-packaged WAR

When I run Gateway I get this error on Registry service.
2018-07-31 11:23:33.472 ERROR 1617 --- [get_localhost-3] c.n.e.cluster.ReplicationTaskProcessor : It seems to be
a socket read timeout exception, it will retry later. if it continues to happen and some eureka node occupied all the cpu time, you should set property 'eureka.server.peer-node-read-timeout-ms' to a bigger value
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out
at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:187)
at com.netflix.eureka.cluster.DynamicGZIPContentEncodingFilter.handle(DynamicGZIPContentEncodingFilter.java:48)
at com.netflix.discovery.EurekaIdentityHeaderFilter.handle(EurekaIdentityHeaderFilter.java:27)
at com.sun.jersey.api.client.Client.handle(Client.java:652)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:570)
at com.netflix.eureka.transport.JerseyReplicationClient.submitBatchUpdates(JerseyReplicationClient.java:116
)
at com.netflix.eureka.cluster.ReplicationTaskProcessor.process(ReplicationTaskProcessor.java:80)
at com.netflix.eureka.util.batcher.TaskExecutors$BatchWorkerRunnable.run(TaskExecutors.java:187)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:161)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:82)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:278)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.jav
a:286)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:257
)
at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java
:230)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:684)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:173)
... 10 common frames omitted
I use pre-packaged war file downloaded from Releases page.
Version 4.0.0
Interesting part is when I run microservice I don't get this error. It happens only with gateway.
Also when creating new gateway and run it exception also not appear. It happens after 1 day and after a day if you got this exception you get this every time...

I got the same error using the default configuration. The solution lies on the error message itself which is to increase the peer-node-read-timeout-ms value (default value is 200).
Go to your application.yml
server:
# see discussion about enable-self-preservation:
# https://github.com/jhipster/generator-jhipster/issues/3654
enable-self-preservation: false
peer-node-read-timeout-ms: 5000

ElasticSearch LockObtainFailedException on restoring index from s3 repository

I keep getting the following exception on trying to restore a snapshot using cloud_aws plugin from s3 repository:
[WARN ][cluster.action.shard ] [Landslide] [index_name][0] received shard failed for target shard [[index_name][0], node[U2w_femBQYO3f5TuOI5daw], [P], v[109], restoring[elasticsearch:backup_name], s[INITIALIZING], a[id=gJBMpmcVT6G132-h1ONGgw], unassigned_info[[reason=ALLOCATION_FAILED], at[2016-09-01T13:01:53.147Z], details[failed to create shard, failure ElasticsearchException[failed to create shard]; nested: LockObtainFailedException[Can't lock shard [index_name][0], timed out after 5000ms]; ]]], indexUUID [7quZdjJqRRmhzr7WBXqlgQ], message [failed to create shard], failure [ElasticsearchException[failed to create shard]; nested: LockObtainFailedException[Can't lock shard [index_name][0], timed out after 5000ms]; ]
[index_name][[index_name][0]] ElasticsearchException[failed to create shard]; nested: LockObtainFailedException[Can't lock shard [index_name][0], timed out after 5000ms];
at org.elasticsearch.index.IndexService.createShard(IndexService.java:389)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:601)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:501)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:166)
at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:610)
at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.lucene.store.LockObtainFailedException: Can't lock shard [index_name][0], timed out after 5000ms
at org.elasticsearch.env.NodeEnvironment$InternalShardLock.acquire(NodeEnvironment.java:609)
at org.elasticsearch.env.NodeEnvironment.shardLock(NodeEnvironment.java:537)
at org.elasticsearch.index.IndexService.createShard(IndexService.java:306)
... 10 more
ES version 2.4.0.
There's no other operation going on on elastic server. Also note that the restore operation does complete inspite of the above exception, but it takes very long to restore a < 1GB index on a 5-6 Mbps internet. Any suggestions?
EDIT: It seems its a problem with s3 buckets. Finding no particular solution to this, i have moved on to FS repository.
i'd still be interested in the solution to this problem

Why sleep mode connections are not reused by c3p0?

I develop an web application using Spring MVC + MySQL. To manage JDBC connection I used c3p0 for connection pooling.
If I am using c3p0, What I suppose if, 5 connections are open in pool and all 5 connections are in sleep mode and if I request getConnection() from java then one of them should be returned as Connection. Am I correct?
If Yes Then consider following load testing scenario where I am getting Exception Connections could not be acquired from the underlying database!. Using Jmeter I started load test for 250 users in 15 seconds. I m continuously observing DB connections using SHOE FULL PROCESSLIST for first some minutes all is going well but as all the 250 users thrown to web server connections count is reached to 500 which is our maxPoolSize. So after this we are getting Connections could not be acquired from the underlying database!.
At this point if I execute SHOE FULL PROCESSLIST then I can see all 500 connections are in sleep mode If I am correct in above statement that any open connection that is in sleep mode will be returned by c3p0. Then why I am getting this exception.?
Here is my c3p0 properties
MINPOOLSIZE=10
ACQUIREINCREMENT=1
MAXPOOLSIZE=500
INITIALPOOLSIZE=10
NUMBERHELPERTHREAD=100
MAXIDLETIME=10
MAXSTATEMENT=20
MAXSTATEMENTPERCONNECTION=5
IDLECONNECTIONTESTPERIOD=120
ACQUIRERETRYATTEMPT=10
ACQUIRERETRYDELAY=100
AUTOCOMMITONCLOSE=false
BREAKAFTERACQUIREFAILURE=false
TESTCONNECTIONONCHECKOUT=true
TESTCONNECTIONONCHECKIN=true
Update
I found this warning before Connections could not be acquired from the underlying database! Exception
WARN : 30 Jun 2014 10:40:12.078 com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1839) - com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask#e49561 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (10). Last acquisition attempt exception:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Too many connections
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.Util.getInstance(Util.java:386)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1015)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4187)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4119)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:927)
at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1709)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1252)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2483)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2516)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2301)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.GeneratedConstructorAccessor18.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:346)
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:183)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:172)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:188)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1074)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1061)
at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1798)
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:636)
WARN : 30 Jun 2014 10:40:12.078 com.mchange.v2.resourcepool.BasicResourcePool.forceKillAcquires(BasicResourcePool.java:882) - Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool#1035ff9 is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests.