Develop Reference

Programmatically restart HikariPool in Spring Boot application?

I have a Spring Boot application which uses Hibernate, and HikariDataSource / HikariPool to talk to the database.
A special feature in the app triggers database restart. Currently this breaks the connections in HikariPool:
Caused by: org.postgresql.util.PSQLException: ERROR: relation "relation_which_really_exists" does not exist
Position: 113
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2532)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2267)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:312)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:448)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:369)
Old version of the app does call programmatically org.hibernate.SessionFactory.close(); which causes restart of HikariDataSource / HikariCP:
2020-08-17T11:36:42.628Z [qtp1340328248-76] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Shutdown initiated...
2020-08-17T11:36:42.698Z [qtp1340328248-76] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Shutdown completed.
2020-08-17T11:36:51.266Z [qtp1340328248-12] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-2 - Starting...
2020-08-17T11:36:51.515Z [qtp1340328248-12] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-2 - Start completed.
I would like to do the same, but how can I programmatically restart the connection pool in my app? I've seen things like Spring Boot - Handle to Hibernate SessionFactory to get a handle to sessionFactory, and maybe something similar to get a handle to DataSource or CP....but is it OK to close / restart those objects violently, is the Spring Boot context designer to handle such action properly?
There's probably some HikariCP configuration parameters that I can start to experiment with to try reach the same end result, but replicating the old implementation tempts as the easiest / most figured out and most probable to reach same end result.

After some study, found out that you can get handle of HikariCP and trigger connection eviction by:
HikariDataSource hikariDs = (HikariDataSource) dataSource;
HikariPoolMXBean poolBean = hikariDs.getHikariPoolMXBean();
poolBean.softEvictConnections();

There is a parameter in HikariCP which is connectionTestQuery
This is the query that will be executed just before a connection is given to you from the pool to validate that the connection to the database is still alive. So I think you can use it to check if the connection is still alive or not and then force things. But they mention this in the documentation "If your driver supports JDBC4 we strongly recommend not setting this property. This is for "legacy" drivers that do not support the JDBC4 Connection.isValid() API"

There are following possiblties.
make sure allowPoolSuspension is available in configuration.
make sure hikari version 4 or above.
reset credential:-
HikariDataSource ds = (HikariDataSource) applicationContext.getBean("dataSource");
HikariPoolMXBean hikariPoolMXBean = ds.getHikariPoolMXBean();
if (hikariPoolMXBean != null) {
hikariPoolMXBean.softEvictConnections();
}ds.setUsername(userName);ds.setPassword(password);

failing to eagerly initialize maps in hazelcast on startup

Currently I have two maps in hazelcast, and they are configured like so:
<hz:map name="some-map"
max-idle-seconds="0"
time-to-live-seconds="0">
<hz:map-store enabled="true"
initial-mode="EAGER"
write-delay-seconds="0"
class-name="SomeMapStore">
</hz:map-store>
<hz:partition-strategy>com.hazelcast.partition.strategy.DefaultPartitioningStrategy</hz:partition-strategy>
</hz:map>
I would expect the initial-mode="EAGER" from the hazelcast-beans.xml configuration to populate the hazelcast map. Instead the application process hangs for a moment, and then I see the following error:
my-service 21:14:15.247Z [hz.my-service-name.SlowOperationDetectorThread] WARN com.hazelcast.spi.impl.operationexecutor.slowoperationdetector.SlowOperationDetector - [localhost]:8085 [my-service-name-local] [3.9.4] Slow operation detected: com.hazelcast.map.impl.operation.PutTransientOperation
Has anyone run into this? I'm on hazelcast 3.9.4

Connecting to as400 JDBC on Play Framework

I'm having issues making a connection to an AS400 database inside of Play!.
My application.conf looks like:
db.default.driver="com.ibm.as400.access.AS400JDBCDriver"
db.default.url="jdbc:as400://SERVER;libraries=A,B,C;toolbox trace=all;trace=true"
db.default.username="user"
db.default.password="password"
I've set up jt400 in the classpath, and I can see under "external libraries" that it shows up and is available. But essentially I get an error message about failing to connect (on user/password I know works) and failure to execute isValid(), which is a function that can not be found inside of AS400JDBCConnection class.
[error] c.z.h.p.PoolBase - HikariPool-1 - Failed to execute isValid() for connection, configure connection test query. (com.ibm.as400.access.AS400JDBCConnection.isValid(I)Z)
[error] application -
! #72265nf0a - Internal server error, for (GET) [/] ->
play.api.Configuration$$anon$1: Configuration error[Cannot connect to database [default]]
at play.api.Configuration$.configError(Configuration.scala:154)
at play.api.Configuration.reportError(Configuration.scala:806)
at play.api.db.DefaultDBApi$$anonfun$connect$1.apply(DefaultDBApi.scala:48)
at play.api.db.DefaultDBApi$$anonfun$connect$1.apply(DefaultDBApi.scala:42)
at scala.collection.immutable.List.foreach(List.scala:381)
at play.api.db.DefaultDBApi.connect(DefaultDBApi.scala:42)
at play.api.db.DBApiProvider.get$lzycompute(DBModule.scala:72)
at play.api.db.DBApiProvider.get(DBModule.scala:62)
at play.api.db.DBApiProvider.get(DBModule.scala:58)
at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
Caused by: play.api.Configuration$$anon$1: Configuration error[Failed to initialize pool: com.ibm.as400.access.AS400JDBCConnection.isValid(I)Z]
at play.api.Configuration$.configError(Configuration.scala:154)
at play.api.PlayConfig.reportError(Configuration.scala:996)
at play.api.db.HikariCPConnectionPool.create(HikariCPModule.scala:70)
at play.api.db.PooledDatabase.createDataSource(Databases.scala:199)
at play.api.db.DefaultDatabase.dataSource$lzycompute(Databases.scala:123)
at play.api.db.DefaultDatabase.dataSource(Databases.scala:121)
at play.api.db.DefaultDatabase.getConnection(Databases.scala:142)
at play.api.db.DefaultDatabase.getConnection(Databases.scala:138)
at play.api.db.DefaultDBApi$$anonfun$connect$1.apply(DefaultDBApi.scala:44)
at play.api.db.DefaultDBApi$$anonfun$connect$1.apply(DefaultDBApi.scala:42)
Caused by: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: com.ibm.as400.access.AS400JDBCConnection.isValid(I)Z
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:512)
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:105)
at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:71)
at play.api.db.HikariCPConnectionPool$$anonfun$1.apply(HikariCPModule.scala:58)
at play.api.db.HikariCPConnectionPool$$anonfun$1.apply(HikariCPModule.scala:54)
at scala.util.Try$.apply(Try.scala:192)
at play.api.db.HikariCPConnectionPool.create(HikariCPModule.scala:54)
at play.api.db.PooledDatabase.createDataSource(Databases.scala:199)
at play.api.db.DefaultDatabase.dataSource$lzycompute(Databases.scala:123)
at play.api.db.DefaultDatabase.dataSource(Databases.scala:121)
Caused by: java.lang.AbstractMethodError: com.ibm.as400.access.AS400JDBCConnection.isValid(I)Z
at com.zaxxer.hikari.pool.PoolBase.checkDriverSupport(PoolBase.java:400)
at com.zaxxer.hikari.pool.PoolBase.setupConnection(PoolBase.java:375)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:346)
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:506)
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:105)
at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:71)
at play.api.db.HikariCPConnectionPool$$anonfun$1.apply(HikariCPModule.scala:58)
at play.api.db.HikariCPConnectionPool$$anonfun$1.apply(HikariCPModule.scala:54)
at scala.util.Try$.apply(Try.scala:192)
at play.api.db.HikariCPConnectionPool.create(HikariCPModule.scala:54)
I'm able to connect in other java-based projects using something like:
try {
Class.forName("com.ibm.as400.access.AS400JDBCDriver");
Connection con = DriverManager.getConnection("jdbc:as400://" +
ApplicationAuthentication.server + "/" +
ApplicationAuthentication.library,
ApplicationAuthentication.user,
ApplicationAuthentication.password
);
} catch (Exception e) {
System.err.println(e);
throw new WebApplicationException(genericError, Response.Status.UNAUTHORIZED);
}

Guessing from the stacktrace, it appears that the connection returned from your driver is not playing well with the connection Hikari Connection Pool. Hikari is default connection pool in playframework.
Specifically, your exception trace shows that the Hikari CP is attempting to call isValid method on the connection object returned by your JDBC driver and then failing with java.lang.AbstractMethodError.
You can try switching to BoneCP connection pool and see if it helps. You can also check comments on this issue on hikari github issue list

Try adding the following to application.config
db.default.hikaricp.connectionTestQuery="SELECT 1"
Not tested in Play Framework but I had similar issue on spring framework and solved in that way.

Use liquibase datasource like below with connection-test-query
#Bean
#LiquibaseDataSource
public DataSource liquibaseDataSource() {
HikariDataSource dataSource = (HikariDataSource) DataSourceBuilder.create().url("url")
.username("username")
.password("password")
.type(HikariDataSource.class).build();
dataSource.setConnectionTestQuery("select 1 from sysibm.sysdummy1");
return dataSource;
}

EJB client can't find a DataSource that tests successfully in WebLogic admin console

Disclaimer: I'm completely new to Java EE/EJB and all that, so bear with me.
I have a simple EJB that I can successfully deploy using JDeveloper 11g's integrated WebLogic server and a remote database connection (JDBC). I have a DataSource named "PGY2" defined in WebLogic, and I can test it successfully from the admin console.
Here's the code of the client I'm trying to test it with (generated entirely by JDev except for the three method calls on adminManager):
public class AdminManagerClient {
public static void main(String [] args) {
try {
final Context context = getInitialContext();
AdminManager adminManager = (AdminManager)context.lookup("Uran-AdminManager#hu.elte.pgy2.BACNAAI.UranEJB.AdminManager");
adminManager.addAdmin("root", "root", "Kovács Isten");
adminManager.addStudent("BACNAAI", "matt", "B Cs", 2005);
adminManager.addTeacher("SIPKABT", "patt", "S P", "numanal", "Dr.");
} catch (Exception ex) {
ex.printStackTrace();
}
}
private static Context getInitialContext() throws NamingException {
Hashtable env = new Hashtable();
// WebLogic Server 10.x connection details
env.put( Context.INITIAL_CONTEXT_FACTORY, "weblogic.jndi.WLInitialContextFactory" );
env.put(Context.PROVIDER_URL, "t3://127.0.0.1:7101");
return new InitialContext( env );
}
}
But when I try to run this, I get the following error on the line with adminManager.addAdmin (that is, after the lookup):
javax.ejb.EJBException: EJB Exception: ; nested exception is:
Exception [EclipseLink-4002] (Eclipse Persistence Services - 1.0.2 (Build 20081024)): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: Internal error: Cannot obtain XAConnection Creation of XAConnection for pool PGY2 failed after waitSecs:30 : java.sql.SQLException: Data Source PGY2 does not exist.
Why can't the client find the data source, and how do I make it find it?
EDIT: I took a closer look at WebLogic's output during deployment, and I found this. I have no idea what it means, but it may be relevant:
<2010.05.20. 0:50:43 CEST> <Error> <Deployer> <BEA-149231> <Unable to set the activation state to true for the application 'PGY2'.
weblogic.application.ModuleException:
at weblogic.jdbc.module.JDBCModule.activate(JDBCModule.java:349)
at weblogic.application.internal.flow.ModuleListenerInvoker.activate(ModuleListenerInvoker.java:107)
at weblogic.application.internal.flow.DeploymentCallbackFlow$2.next(DeploymentCallbackFlow.java:411)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:37)
at weblogic.application.internal.flow.DeploymentCallbackFlow.activate(DeploymentCallbackFlow.java:74)
Truncated. see log file for complete stacktrace
weblogic.common.ResourceException: is already bound
at weblogic.jdbc.common.internal.RmiDataSource.start(RmiDataSource.java:387)
at weblogic.jdbc.common.internal.DataSourceManager.createAndStartDataSource(DataSourceManager.java:136)
at weblogic.jdbc.common.internal.DataSourceManager.createAndStartDataSource(DataSourceManager.java:97)
at weblogic.jdbc.module.JDBCModule.activate(JDBCModule.java:346)
at weblogic.application.internal.flow.ModuleListenerInvoker.activate(ModuleListenerInvoker.java:107)
Truncated. see log file for complete stacktrace
>

Actually, it's not the EJB client that can't find the DataSource, it's the code running inside the container and using JPA. More precisely, it's the JPA provider - EclipseLink here - that can't find it. Double check that the datasource name defined in your persistence.xml (in the <jta-data-source>) matches the JNDI name of the PGY2 DataSource.
WebLogic provides a JNDI browser (to browse the JNDI tree), use it if required to see under which name your datasource is registrered.
Update: Could you have linebreaks in the JNDI name field of the datasource (see this blog post)?

weblogic context lookup error : java.rmi.UnmarshalException: error unmarshalling arguments

We are facing an issue in our production env. We have searched the net high and low and we were not able to come up with any answers.
This error(stacktrace below) occurs when an ejb lookup is made from managed server 1 to manager server 2. Virtual ip is used for the lookup. It occurs intermittently and at random intervals. We are not able to identify any pattern and If the ejb call is attempted two or three times, it gets through successfully.
Env details :
server : weblogic 10.0 MP1 running on java 1.5
os : solaris
Pls revert if any other details are required.
Source used for lookup :
private TreControlRemote getController() throws Exception {
Context context = null;
Properties p = new Properties();
TreControlHome treHome = null;
TreControlRemote remote = null;
ConfigurationLoader lAppLoader = null;
try {
mLog.debug("Entering");
lAppLoader = PropertiesFileLoader.getInstance("context.properties");
p.put(Context.INITIAL_CONTEXT_FACTORY, lAppLoader.getValue("INITIAL_CONTEXT_FACTORY"));
p.put(Context.PROVIDER_URL, lAppLoader.getValue("PROVIDER_URL"));
context = new InitialContext(p);
mLog.debug("context : " + context.getEnvironment());
remote = null;
treHome = (TreControlHome) context.lookup("CONTROL");
mLog.debug("Object --->>>>" + treHome);
remote = (TreControlRemote) treHome.create();
mLog.debug("Leaving");
} catch (Exception ex) {
mLog.fatal("Exception while getting remote", ex);
ex.printStackTrace();
throw ex;
} finally {
lAppLoader = null;
}
return remote;
}
The url is a virtual ip pointing to managed server 2 and it contains a ejb with jndi "CONTROL". The problem is that it successful on certain occassions and fails randomly with the error:
stack trace of the error :
*javax.naming.CommunicationException [Root exception is java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
java.io.StreamCorruptedException]
at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:74)
at weblogic.jndi.internal.WLContextImpl.translateException(WLContextImpl.java:426)
at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:382)
at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:367)
at javax.naming.InitialContext.lookup(InitialContext.java:351)
```````````````````````````````````````````````````````````````````
Caused by: java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
java.io.StreamCorruptedException
at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:221)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:338)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:252)
at weblogic.jndi.internal.ServerNamingNode_1001_WLStub.lookup(Unknown Source)
at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:379)
... 33 more
Caused by: java.io.StreamCorruptedException
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1332)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
at weblogic.utils.io.ChunkedObjectInputStream.readObject(ChunkedObjectInputStream.java:195)
at weblogic.rjvm.MsgAbbrevInputStream.readObject(MsgAbbrevInputStream.java:565)
at weblogic.utils.io.ChunkedObjectInputStream.readObject(ChunkedObjectInputStream.java:191)
at weblogic.jndi.internal.RootNamingNode_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:589)
at weblogic.rmi.cluster.ClusterableServerRef.invoke(ClusterableServerRef.java:224)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:479)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:475)
at weblogic.rmi.internal.BasicServerRef.access$300(BasicServerRef.java:59)
at weblogic.rmi.internal.BasicServerRef$BasicExecuteRequest.run(BasicServerRef.java:1016)
... 2 more*
Obtained the below mentioned stacktrace from the weblogic log. Could this error be related to our problem mentioned above?
*####<Aug 25, 2009 2:11:04 AM BST> <Info> <RJVM> <pkssv049> <M1AP4> <ACTIVE ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <1251162664181> <BEA-000513> <Failure in heartbeat trigger for RJVM: 5433424963141690658S:169.93.73.0:10040,10040,-1,-1,-1,-1,-1:pkssv049.***.net:10240,pkssv049.***.net:10241,pkssv050.***.net:10240,pkssv050.***.net:10241:LIQP1_LMSDomain:M1AP3
java.io.IOException: The connection manager to ConnectionManager for: 'weblogic.rjvm.RJVMImpl#189ed0e - id: '5433424963141690658S:169.93.73.0:10040,10040,-1,-1,-1,-1,-1:pkssv049.***.net:10240,pkssv049.***.net:10241,pkssv050.***.net:10240,pkssv050.***.net:10241:LIQP1_LMSDomain:M1AP3' connect time: 'Mon Aug 24 20:24:02 BST 2009'' has already been shut down.
java.io.IOException: The connection manager to ConnectionManager for: 'weblogic.rjvm.RJVMImpl#189ed0e - id: '5433424963141690658S:169.93.73.0:10040,10040,-1,-1,-1,-1,-1:pkssv049.***.net:10240,pkssv049.***.net:10241,pkssv050.***.net:10240,pkssv050.***.net:10241:LIQP1_LMSDomain:M1AP3' connect time: 'Mon Aug 24 20:24:02 BST 2009'' has already been shut down
at weblogic.rjvm.ConnectionManager.getOutputStream(ConnectionManager.java:1686)
at weblogic.rjvm.ConnectionManager.createHeartbeatMsg(ConnectionManager.java:1629)
at weblogic.rjvm.ConnectionManager.sendHeartbeatMsg(ConnectionManager.java:607)
at weblogic.rjvm.RJVMImpl$HeartbeatChecker.timerExpired(RJVMImpl.java:1540)
at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:464)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)*
Any help would be greatly appreciated.
Here is some additional info..
Is the problem intermittent, or does reproduce every single time? If the problem is intermittent, do you know what conditions it occurs under?
It occurs intermittently and we are not able to observe any pattern.
Are there any other errors/warnings logged either on the local server or on the remote server?
We see a lot of connection refused errors in the weblogic log
Are both the managed servers in the same domain?
Yes

when you pass an instance of com.myclientcompany.server.eai.interactionspecimpl as argument to
your ejb. the weblogic needs to deserialize(unmarshal) the object under the ejb context, and its needs the required class for unmarshalling. so if you include the interactionspecimpl class in your ejb-jar file, then you do not need to include those classes in your servers classpath

This issue can occur if you have either a Duplicate entry for or due to a blank space in between.
You need to check all the configuration files including the JDBC , JMS and the config.xml file to find such and entry.
Check if you have left a blank space while entering the JNDI name from the console as well.
Removing the blank space or removing the duplicate entry resolves this issue.