I am working on a web application which runs in pcf environment and it has approximately 100 users. I am using Hikari CP library to manage databae connections and customized connectionTimedout property by setting it to 1 sec in the application code. Connection pool size is set to 100.
In one scenario, making a call to stored procedure where I am explicitly creating
Connection = DriverManager.getConnection()
object as ArrayDescriptor() is expecting connection object.
I am using ArrayDescriptor as input for stored procedure requires array of object.
However this code is throwing Socket Read Timed Out error randomly
The same code was working fine when configured with dbcp library managed connection pool.
Can anyone help? What's the problem with Hikari CP library?
As per compliance rules I can't post code on public domains.
connectionTimeout
This property controls the maximum number of milliseconds that a
client (that's you) will wait for a connection from the pool. If this
time is exceeded without a connection becoming available, a
SQLException will be thrown. Lowest acceptable connection timeout is
250 ms. Default: 30000 (30 seconds)
Related
I have a spring boot application with below HikariCP properties enabled in application.properties and in mysql i have wait_timeout = 28800
spring.datasource.hikari.minimumIdle=9
spring.datasource.hikari.maximumPoolSize=10
spring.datasource.hikari.maxLifetime=28799990
I still get the below error
13:02:46.103 [http-nio-8082-exec-2] WARN com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Failed to validate connection com.mysql.cj.jdbc.ConnectionImpl#13f6e098 (No operations allowed after connection closed.). Possibly consider using a shorter maxLifetime value.
what values i need to set in HikariCP to fix this issue
Thanks in advance
Edit
#Autowired
JdbcTemplate jdbcTemplate;
public Map<String, Object> getSlideData(String date, String sp){
SimpleJdbcCall simpleJdbcCall = new SimpleJdbcCall(jdbcTemplate).withProcedureName(sp)
.withoutProcedureColumnMetaDataAccess()
.useInParameterNames(ReportGeneratorConstants.TIMEPERIOD)
.declareParameters(
new SqlParameter(ReportGeneratorConstants.TIMEPERIOD,java.sql.Types.VARCHAR)
);
Map<String,String> map=new HashMap<>();
map.put(ReportGeneratorConstants.TIMEPERIOD, date);
return simpleJdbcCall.execute(map);
}
}
I am using simpleJdbcCall to call the stored procedure, I know that simpleJdbcCall uses multithread to run the stored procedure, What i want to know is, does simpleJdbcCall releases/close all connection to pool once execute() is completed (does spring boot take care of closing connections) if yes, where i can see the code for it.
There are many reasons why connections can be terminated. Usually there's something in the middle, tipically a firewall, that drops connections after set amount of time. Find what time that is and set the max life to at least 1 minutes before that. Hikari has a background thread that expires connections but I think it triggers every few seconds; I'm mentioning this as it's not exact to the millisecond.
A note that Hikari will try to keep the minimumIdle number of connections, so usually new connections will be open in the background and won't create a pause in the application to wait for a new connection.
Edit
I'm not familiar with Azure, so I'm not sure if the network stack drops connections after a certain amount of time. Personally, I never set the idle timeout nor max-life settings as long as the ones you have as the chances of a connection dropping is quite high in a cloud environment (in my experience). As an example, in AWS, our data team suggests to use a max lifetime of 5 minutes and idletimeout of 1 minute and this is in a platform with high traffic.
Opening a DB connection is quite fast and usually happens behind the scenes. My suggestion would be to use these settings, but you have to validate them that they work ok for the load of your application (based on the ones you added on the comment below):
# set this equal to maximum pool size if the traffic has burst
spring.datasource.hikari.minimumIdle=5
spring.datasource.hikari.maximumPoolSize=30
# It's the same as the default = 30 minutes
spring.datasource.hikari.maxLifetime=1800000
# It's the same as the default = 10 minutes
spring.datasource.hikari.keepalive-time=60000
I am using withRemote to connect my java application to gremlin server running in AWS with dynamodb storage backend. I am getting connection timeout after few seconds (~3.3 seconds):
org.apache.tinkerpop.gremlin.process.remote.RemoteConnectionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.nio.channels.ClosedChannelException]]
I need to figure out how to reconnect which means detecting if the connection is closed. I am not sure how to detect that. I get the above exception when I use the graph traversal, is there a way to discover it before and reconnect or is there an option in configuration that allows reconnecting automatically (like create new connection before this one closes) so my application is always connected?
In case you need, this is how I am doing connection - currently connection part is singleton when the application starts:
this.graph = EmptyGraph.instance();
GryoMessageSerializerV1d0 gryoMessageSerializerV1d0 = new GryoMessageSerializerV1d0(
GryoMapper.build().addRegistry(JanusGraphIoRegistry.getInstance()));
this.cluster = Cluster.build().serializer(gryoMessageSerializerV1d0)
.addContactPoint(configuration.getString("graphDb.host", "localhost"))
.port(configuration.getInt("graphDb.port", 8182)).create();
this.graphTraversalSource = this.graph.traversal().withRemote(DriverRemoteConnection.using(cluster));
I feel like this problem is already solved with connection.keepAlive configuration option. It defaults to 180 seconds so it's longer than your timeout of 60 seconds in your load balancer which is why it gives up.
That said, the driver should be reconnecting on its own. It's constantly trying to do that given the connectionPool.reconnectInterval but perhaps there is a condition where you're quickly exhausting all the connections to the point of getting that error....not sure. Either way, hopefully the
I'm upgrading our application to MQ7 (7.5.0.5) and I'm seeing some odd behavior in a small test application that I have written.
My application uses Springs CachingConnectionFactory and is configured to use only one thread.
I can see that by debugging through the code 2 tcp connections are created, one for the initial connection and one for the JMS session. Every 60 seconds, the 2 tcp connections that are used by my message sink are broken and replaced with 2 new connections.
The following error is present in the error logs on the queue manager.
05/16/2016 09:38:26 AM - Process(1609.14) User(mqm) Program(amqrmppa)
Host(xxxxxxxxx) Installation(Installation1)
VRMF(7.5.0.2) QMgr(xxxxx)
AMQ9271: Channel 'XX.XXXX.X' timed out.
EXPLANATION:
A timeout occurred while waiting to receive from the other end of channel
'XX.XXX.X'. The address of the remote end of the connection was '57.4.4.145'.
ACTION:
The return code from the (recv) [TIMEOUT] 60 seconds call was 0 (X'0').
Record these values and tell the systems administrator.
I have the following settings on my channel: DISCINT(60), SHARECNV(1), the exceptions are linked to the DISCINT time, changing that changes the frequency of the exceptions, also the
exceptions disappear with a SHARECNV value >1
Can anyone tell my why the connections are broken even when the channel is active and messages are being sent and received?
Thanks!
This sounds like APAR IV62728 which describes the symptoms you're seeing:
http://www-01.ibm.com/support/docview.wss?rs=171&uid=swg1IV62728
Fixed in 7.5.0.6. Try upgrading to that level and see if it solves the problem.
I managed to find a solution to this issue. When using the CachingConnectionFactory with an underlying IBM connection factory, an initial connection is created in a stopped state. That connection in then used to create JMSSessions.
The issue was that the initial common connection was timing out.
I managed to keep the connection active by adjusting the HBINT value to 5. It appears that a number of heatbeats are required to keep the connection open and my initial value of 20 was too high.
I'm connecting my Java application via JDBC driver and Tomcat configurations. I used this class to define my configurations. But sometimes, I got following exceptions:
com.mysql.jdbc.exceptions.MySQLTimeoutException: Statement cancelled due to timeout or client request
java.sql.SQLException: Query execution was interrupted
java.sql.BatchUpdateException: Statement cancelled due to timeout or client request
But there is not much load on database, so I think the problem is about my configuration. Here are some of my configs:
maxActive = 100
minIdle = 10
initialSize = 10
maxWait = 10000
maxIdle = 15
These are for some heavily loaded system. So I need to observe if my pool size is enough and other things like available connection count at any time. Is there any nice way of monitoring inside of connection pools?
For the timeout exceptions, consider the following pseudocode:
if(!connection.isValid())
connection = getNewConnection();
resultSet rs = connection.execute(qry);
Basically check if your connection has timed out before you execute the query.
If your query was interrupted, theres not much you can do other than to rollback the transaction and try again.
Enable JMX and tomcat will register your datasources with it's mbean server. You can then use jconsole to look at the connection pool details.
I have been handling a application which uses wicket+JPA+springs technologies.Recently we got many 5XX error in logs(greater than threshhold).During that time,There were some general problems due to unstable response times of the mainframe db2 which is backend for our application.
But after that once the mainframe is OK this application servers did not come to normal again.
There are a lot of hanging transactions (from my appplication).
There are many threads in the server that may be hung.
As users will go on keeping login or will access the links in aplication during that time the situation becomes worse.
When I look at webspehere logs I found following exceptions:
00000035 ThreadMonitor W WSVR0605W: Thread "WebContainer : 88" (000005ac)
has been active for 637111 milliseconds and may be hung.
There is/are 43 thread(s) in total in the server that may be hung.
In application logs i found following exceptions:
-->CouldNotLockPageException: Could not lock page 4. Attempt lasted 3 minutes
-->DefaultExceptionMapper - Connection lost, give up responding.
org.apache.wicket.protocol.http.servlet.ResponseIOException:
com.ibm.wsspi.webcontainer.ClosedConnectionException: OutputStream encountered error during
write.
--> JDBCExceptionReporter - [jcc][t4][2030][11211][3.67.27] A communication error occurred
during operations on the connection's underlying socket, socket input stream,
or socket output stream.
Error location: Reply.fill() - socketInputStream.read (-1). Message:
Connection reset. ERRORCODE=-4499, SQLSTATE=08001DSRA0010E: SQL State = 08001, Error Code = - 4.499
Now we are working on the solutions to this problem.The follwing are two solutions that we are thinking as of now.
1.I have gone through many forums and found that whenever we get CouldNotLockPageException then it would be better to invaidate the session and force user to login page.Currently We do not have session invalidation (logout) mechanism.So we will implement that one.
2.We need to implement transaction timeouts so that we can stop hanging transactions.
I need solution for this problem from java or server side.Here we are using wicket,jpa and springs frameworks.I have few queries.
1.How can we implement transaction timeouts in the above frameworks?
2.Will invalidating session can stop hanging transaction or threads that may hung?
Since you are already using Spring, it's as simple as that:
#Transactional(timeout = 300)
The Transaction annotation allow you to supply a timeout value(in seconds) and the transaction manager will forward it to the JTA transaction manager or your Data Source connection pool. It works nice with Bitronix Transaction Manager, which automatically picks it up.
You also need to make sure the java.sql.Conenction are always being closed and Transaction are always committed (when all operations succeeded) or rollbacked on failure.
Invalidating the user http session has nothing to do with jdbc connections. Your jdbc connection should always be committed/rollbacked and closed(which in case on connection pooling, will release the connection to the pool).
And make sure the max pool size is not greater than tour db max concurrent connections setting.