I have a service that consumed data from Redis streams every second and sends the data to the client via web socket. Everything worked just fine until two weeks ago (this code has been written two years ago). Since then i'm getting lots of RedisCommandTimeoutException errors. This is the full stack trace:
org.springframework.dao.QueryTimeoutException: Redis command timed out; nested exception is io.lettuce.core.RedisCommandTimeoutException: Command timed out after 20 second(s)
at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:70)
at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:41)
at org.springframework.data.redis.connection.lettuce.LettuceReactiveRedisConnection.lambda$translateException$0(LettuceReactiveRedisConnection.java:293)
at reactor.core.publisher.Flux.lambda$onErrorMap$28(Flux.java:7070)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onError(MonoFlatMapMany.java:255)
at reactor.core.publisher.FluxMap$MapSubscriber.onError(FluxMap.java:134)
at io.lettuce.core.RedisPublisher$ImmediateSubscriber.onError(RedisPublisher.java:891)
at io.lettuce.core.RedisPublisher$State.onError(RedisPublisher.java:712)
at io.lettuce.core.RedisPublisher$RedisSubscription.onError(RedisPublisher.java:357)
at io.lettuce.core.RedisPublisher$SubscriptionCommand.onError(RedisPublisher.java:797)
at io.lettuce.core.RedisPublisher$SubscriptionCommand.doOnError(RedisPublisher.java:793)
at io.lettuce.core.protocol.CommandWrapper.completeExceptionally(CommandWrapper.java:128)
at io.lettuce.core.protocol.CommandExpiryWriter.lambda$potentiallyExpire$0(CommandExpiryWriter.java:175)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:750)
Caused by: io.lettuce.core.RedisCommandTimeoutException: Command timed out after 20 second(s)
at io.lettuce.core.internal.ExceptionFactory.createTimeoutException(ExceptionFactory.java:59)
at io.lettuce.core.protocol.CommandExpiryWriter.lambda$potentiallyExpire$0(CommandExpiryWriter.java:176)
... 8 more
This error appears in each subscribed stream at the same time, let's say i have three streams that i'm subscribing to, with names "A", "B" and "C", all of them will have this error at the same time, although i'm using different subscribers for each stream.
Nothing has changed in this service, not in the code and not in the configurations. I looked at the CPU, heap memory, etc. of the service and everything looks fine. I checked AWS Elasticache as well and it seems like there are no problems too (I even talked with AWS support). There is no significant increase in number of streams or opened web sockets.
In addition, i have two other services that write to the Redis streams and they're working great.
I'm using ReactiveRedisConnectionFactory class from spring-data-redis library.
This is configurations:
#Bean(name = "streamRedisLettuceConnectionFactory")
public RedisConnectionFactory connectionFactory() {
log.info("Connecting to redis with configuration : {} ", chpStreamRedisConfiguration);
LettuceClientConfiguration configuration = getLettuceClientConfiguration();
return new LettuceConnectionFactory(redisStandaloneConfiguration, configuration);
}
#Bean(name = "streamRedisStandaloneConfiguration")
public RedisStandaloneConfiguration getRedisStandaloneConfiguration() {
return new RedisStandaloneConfiguration(chpStreamRedisConfiguration.getHost(), chpStreamRedisConfiguration.getPort());
}
#Bean(name = "streamLettuceClientConfiguration")
public LettuceClientConfiguration getLettuceClientConfiguration() {
return LettucePoolingClientConfiguration.builder()
.poolConfig(getLettucePoolConfig())
.commandTimeout(Duration.ofMillis(chpStreamRedisConfiguration.getTimeout()))
.build();
}
#Bean(name = "streamLettucePoolConfig")
public GenericObjectPoolConfig getLettucePoolConfig() {
GenericObjectPoolConfig genericObjectPoolConfig = new GenericObjectPoolConfig();
genericObjectPoolConfig.setMaxTotal(chpStreamRedisConfiguration.getConnectionPool().getMaxTotal());
genericObjectPoolConfig.setMaxIdle(chpStreamRedisConfiguration.getConnectionPool().getMaxTotal());
genericObjectPoolConfig.setMinIdle(chpStreamRedisConfiguration.getConnectionPool().getMinIdle());
return genericObjectPoolConfig;
}
I'm using:
spring-data-redis:2.3.9.RELEASE
lettuce-core:6.1.2.RELEASE
spring-boot:5.2.18.RELEASE
I tried to upgrade lettuce to the latest version (6.2.2) and it didn't help.
I used io.lettuce.core.protocol on debug and the only thing i got is this:
Completing command LatencyMeteredCommand [type=XREAD, output=StreamReadOutput [output=[], error='Command timed out after 20 second(s)'], commandType=io.lettuce.core.RedisPublisher$SubscriptionCommand]
I prefer not to increase the timeout because 20 seconds is already lot of time.
I am using Google cloud managed redis cluster(v5) via redisson(3.12.5)
Following are my SingleServer configurations in yaml file
singleServerConfig:
idleConnectionTimeout: 10000
connectTimeout: 10000
timeout: 3000
retryAttempts: 3
retryInterval: 1500
password: null
subscriptionsPerConnection: 5
clientName: null
address: "redis://127.0.0.1:6379"
subscriptionConnectionMinimumIdleSize: 1
subscriptionConnectionPoolSize: 50
connectionMinimumIdleSize: 40
connectionPoolSize: 250
database: 0
dnsMonitoringInterval: 5000
threads: 0
nettyThreads: 0
codec: !<org.redisson.codec.JsonJacksonCodec> {}
I am getting following exceptions when I increase the load on my application
org.redisson.client.RedisTimeoutException: Unable to acquire connection! Increase connection pool size and/or retryInterval settings Node source: NodeSource
org.redisson.client.RedisTimeoutException: Command still hasn't been written into connection! Increase nettyThreads and/or retryInterval settings. Payload size in bytes: 34. Node source: NodeSource
It seems there is no issue on redis cluster and i think i need to make tweaking in my client side redis connection pooling confs(mentioned above) to make it work.
Please suggest me the changes i need to make in my confs
I am also curious if I should close the Redis connection after making get/set calls. I have tried finding this but found nothing conclusive on how to close Redis connections
One last thing that I want to ask is that is there any mechanism to get Redis connection pool stats(active connection, idle connection etc ) in Redisson
Edit1:
I have tried by changing values following values in 3 different iterations
Iteration 1:
idleConnectionTimeout: 30000
connectTimeout: 30000
timeout: 30000
Iteration 2:
nettyThreads: 0
Iteration 3:
connectionMinimumIdleSize: 100
connectionPoolSize: 750
I have tried these things but nothing has worked for me
Any help is appreciated.
Thanks in advance
Assuming you are getting low memory alerts on your cache JVM.
You may have to analyze the traffic and determine 2 things
Too many parallel cache persists.
Huge chunk of data being persisted.
Both can be determined by the traffic on your server.
For option 1 configuring pool-size would solve you issue, but for option 2 you may have to refactor your code to persist data in smaller chunks.
Try to set nettyThreads = 64 settings
I have Hystrix working with Feign in my Spring Boot Application.
I have set the default timeout for Hystrix to 10000ms with:
feign:
hystrix:
enabled: true
hystrix:
command:
default:
execution:
isolation:
thread:
timeoutInMilliseconds: 10000
The problem is I have this one client, lets call it HeavyClient, that is a heavy call that sometimes takes more time and causes the circuit break.
I would like to increase the timeout cap in Hystrix for this one guy only. Is it possible?
I have tried with Feign properties like:
feign:
client:
config:
HeavyClient:
connectTimeout: 30000
readTimeout: 30000
But this doesn't work. I guess Hystrix does not check Feign properties.
I'm using:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-openfeign</artifactId>
<version>2.2.1.RELEASE</version>
</dependency>
Any help is appreciated, thanks!
After tweeking around and search more deeply I have found on this GitHub issue that default naming for Hystrix commands are something ugly like InterfaceName#methodNameSignature().
So, for instance, given the following #FeignClient:
#FeignClient(name = "heavyClient", url = "${heavyclient.url}")
public interface HeavyClient {
#RequestMapping("/process/{operation}")
public ResponseEntity<Response> process(#PathVariable("operation") String operation);
}
It would be configured as:
hystrix:
command:
default:
execution:
isolation:
thread:
timeoutInMilliseconds: 5000
HeavyClient#process(String):
execution:
isolation:
thread:
timeoutInMilliseconds: 30000
Unfortunelly, this did not work for me... Not sure why... =s
So to solve it, I registered a bean of type SetterFactory that would create the command keys given the #FeignClient name:
#Bean
public SetterFactory setterFactory() {
return (target, method) -> HystrixCommand.Setter
.withGroupKey(HystrixCommandGroupKey.Factory.asKey(target.name()))
.andCommandKey(HystrixCommandKey.Factory.asKey(target.name()));
}
And then I can use the configuration simply like:
hystrix:
command:
heavyClient:
execution:
isolation:
thread:
timeoutInMilliseconds: 30000
CircuitBreakerNameResolver to naming circuit as HystrixCommandKey, there is a default implementation DefaultCircuitBreakerNameResolver, which output key pattern is HardCodedTarget#methodName(ParamClass), so you can customize CircuitBreakerNameResolver instead of DefaultCircuitBreakerNameResolver.
We have a spring boot application with spring JPA and Sybase ASE database. We have used Hikari as the connection pool.
We have 1 stored procedure written which gets called every 4 minute.
Hikari configuration is:
datasource:
type: com.zaxxer.hikari.HikariDataSource
data-source-class-name: com.sybase.jdbc4.jdbc.SybDataSource
driver-class-name: com.sybase.jdbc4.jdbc.SybDriver
url: xxx-yyy-zzz
username: xxx-yyy-zzz
password: xxx-yyy-zzz
hikari:
maximum-pool-size: 10
minimum-idle: 1
The problem is, I am getting below error after every 10 calls to the stored procedure as I have configured pool size 10.
> Caused by: java.sql.SQLTransientConnectionException: HikariPool-2 -
> Connection is not available, request timed out after 30007ms. at
> com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:669)
>
> org.springframework.dao.DataAccessResourceFailureException: Unable to
> acquire JDBC Connection; nested exception is
> org.hibernate.exception.JDBCConnectionException: Unable to acquire
> JDBC Connection
Code is written as follows:
#Autowired
#Lazy
Smsxyz smsxyzImpl;
#PostConstruct
#Scheduled(fixedRate = 240000)
public void completeStatus() {
try {
smsxyzImpl.checkStatus(Constants.STATUS_PER_SCAN, Constants.STATUS_EXPIRY_DURATION, -1L);
} catch (Exception e) {
e.printStackTrace();
}
}
#Autowired
private EntityManager entityManager;
#Override
//#Transactional
public Integer checkStatus(Integer count, Integer expiry, Long userId) {
StoredProcedureQuery query = entityManager.createNamedStoredProcedureQuery("status.check");
query.setParameter("#status_count", count);
query.setParameter("#status_expiry", expiry);
query.setParameter("#user_id", userId);
Integer outputValue= (Integer) query.getOutputParameterValue("#return_status");
return outputValue;
}
If I remove #Transactional then it runs for 10 times (as hikari pool size is 10) and after that it throws an exception as mentioned above
If I keep #Transactional then it runs fine with no exception but it does not update the database.
If I run the stored procedure manually directly from the database, then it works perfectly fine that means no issue in the Stored procedure
My suspect is, it is not releasing the connection. So I closed the connection by entitymanager.close() but it didn't work
I spent many days for this issue but no luck. Could anyone let me know what I am doing wrong here?
I have a small Java application for testing purposes. I have moved to hikari recently. What I notice is that I keep getting this error.
java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:602)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:195)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:85)
Below is my settings for the hikari initially.
HikariConfig config = new HikariConfig();
config.setJdbcUrl("jdbc:mysql://localhost:3306/****");
config.setUsername("***");
config.setPassword("*****");
config.setMaximumPoolSize(20);
Hardly its being used my two devices and I ensure towards the end I do close it. So I don't know why it keep getting the error? What could be the issue or is there some settings which I need to change?
My hikari version is HikariCP-2.6.1.jar.
Your database is not obtaining connection within (30000 milliseconds that is default connectionTimeout property) because of network latency or some of the queries which are taking too long to execute(more than 30000 milliseconds).
Please try to increase value of property connectionTimeout.
YML configuration example:
spring:
datasource:
hikari:
minimumIdle: 2
maximumPoolSize: 10
idleTimeout: 120000
connectionTimeout: 300000
leakDetectionThreshold: 300000
Java Config example:
HikariConfig config = new HikariConfig();
config.setMaximumPoolSize(20);
config.setConnectionTimeout(300000);
config.setConnectionTimeout(120000);
config.setLeakDetectionThreshold(300000);
I am using spring boot and I was facing the same problem, and my solution was to get the connection like this "DataSourceUtils.getConnection(dataSource)". So I change from dataSource.getConnection() to DataSourceUtils.getConnection(dataSource).
In my case the code wasn't closing the connections.
Try-with-resources fixed it:
try (
Connection connection = dataSource.getConnection();
Statement statement = …
) {
…
}
In my case I was using JPA and hence using EntityManagerFactory for persistence and query for my springBoot project and got the same error.
The reason was in any CRUD operation I was not closing EntityManager once the operation is done hence exhausting the resources.
Hope this helps!!
EntityManager em = emf.createEntityManager();
Customer c = em.find(Customer.class , id);
em.close();
request timeout is not something that you can fix by increasing the timeout. Perhaps you'd need to evaluate all the queries from your service and implement indexing if it's needed
This can also happen if the client app is requesting lot of open connections and the database server setting has a max limit on number of pool connections. So the client app is unable to get any more connections from the database server. Check the database server connections pool to see if the max is exceeded during the time period of the errors.
Took forever to figure it out... In my case I used solution similar to #Andres Rincon:
try (Connection connection = DataSourceUtils.getConnection(jdbcTemplate.getDataSource())) {
// some code here
}
What fixed the issue in my case was to add proper indexing in the proper db tables. Take a look at the queries / transactions you're making to the db.
In my case the statement that was causing the latency was an UPDATE statement, e.g.
UPDATE table_name WHERE column1 = value1, column2 = value2;
What fixed the issue for me in this case was to add an index in that table for those two columns like:
CREATE INDEX index_name ON table_name (column1, column2);
Another good reason could be that you're not closing out your connections. You can close the connections with a try-with-resource statement like:
try( Connection connection = datasource.getConnection() ){
//your code
}
In my opinion, increasing the timeout as Girdhar Singh Rathore suggested is not ideal. It could temporarily fix the issue, but at some point you'll need to take care of proper indexing and closing connections management.
Hope this helps.
Generally opened and unclosed connections cause this problem.There is a limit of application servers to connect database and if you over this limit it will be crash your environment.
Connection must be stand on singleton pattern but if you really need to open a datasource or connect external datasource like reports you must close your connection in your finally block where you open connection block
connection.getConnection().rollback();
connection.getConnection().close();
You must also close if you are using PersistenceJpa without singleton
persistenceJPAConfig.dataSource().getConnection().rollback();
persistenceJPAConfig.dataSource().getConnection().close();
If you are using some stress test tools via creating threads to test your methods you probably get this error on your queries which take long time.It will be lead the way optimizing your queries or service instance size.
In my case a:
o.h.engine.jdbc.spi.SqlExceptionHelper: HikariPool-1 - Connection is not available, request timed out after 30019ms.
i.s.commons.web.error.ExceptionLogger: Internal Server Error
org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
Was caused by a too low spring.hikari.maximumPoolSize in the application properties, increasing from 5 to 20 solved the issue.
The log message is kind of miss-leading.
In my case I used solution similar to #Andres Rincon:
try (Connection conn = connectionManager.getDataConnection()) {
Statement stmt = conn.createStatement();
...
conn.close();
} catch (Exception e) {
e.printStackTrace();
}
I've fixed my issue using:
increase the minIdle and maxPool
spring.datasource.hikari.minimumIdle=20
spring.datasource.hikari.maximumPoolSize=30
spring.datasource.hikari.connectionTimeout=50000
To debug the issue/check if the values are ok, enable the logging for Hikari:
logging.level.com.zaxxer.hikari.HikariConfig=DEBUG
logging.level.com.zaxxer.hikari=TRACE
The logs will look like:
DEBUG 2023-01-06T16:12:31.932018849Z HikariPool-1 - Before cleanup stats (total=17, active=0, idle=17, waiting=0)
DEBUG 2023-01-06T16:12:31.932665522Z HikariPool-1 - After cleanup stats (total=17, active=0, idle=17, waiting=0)
DEBUG 2023-01-06T16:12:31.932733949Z HikariPool-1 - Fill pool skipped, pool is at sufficient level.
DEBUG 2023-01-06T16:12:32.495269726Z HikariPool-1 - After adding stats (total=17, active=0, idle=17, waiting=0)
DEBUG 2023-01-06T16:12:38.309953158Z HikariPool-1 - Fill pool skipped, pool is at sufficient level.
DEBUG 2023-01-06T16:12:39.200246897Z HikariPool-1 - Fill pool skipped, pool is at sufficient level.
DEBUG 2023-01-06T16:12:44.812065268Z HikariPool-1 - Before cleanup stats (total=18, active=0, idle=18, waiting=0)
DEBUG 2023-01-06T16:12:44.812822113Z HikariPool-1 - After cleanup stats (total=18, active=0, idle=18, waiting=0)
Good Luck ! :)