I have a question about springboot, quartz scheduler and HikariCP. I am relatively new to this domain and trying to understand the relations and working.
I have gone through many questions that are either related to Springboot HikariCP or Quartz scheduler using HikariCP but none of them is able to answer my questions.
I have an application with below configurations
#Database properties
spring.datasource.url = jdbc:mysql://localhost:3306/demo?user=root&password=root&useSSL=false&serverTimezone=UTC
spring.datasource.username = root
spring.datasource.password = root
spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.MySQL5InnoDBDialect
#Hikari
spring.datasource.hikari.minimumIdle=5
spring.datasource.hikari.maximumPoolSize=20
#quartz settings
spring.quartz.properties.org.quartz.jobStore.dataSource = quartzDataSource
spring.quartz.properties.org.quartz.dataSource.quartzDataSource.driver = com.mysql.cj.jdbc.Driver
spring.quartz.properties.org.quartz.dataSource.quartzDataSource.provider=hikaricp
spring.quartz.properties.org.quartz.dataSource.quartzDataSource.URL = jdbc:mysql://localhost:3306/demo?user=root&password=root&useSSL=false&serverTimezone=UTC
spring.quartz.properties.org.quartz.dataSource.quartzDataSource.user = root
spring.quartz.properties.org.quartz.dataSource.quartzDataSource.password = root
spring.quartz.job-store-type = jdbc
spring.quartz.properties.org.quartz.threadPool.threadCount=20
By default, springboot2 uses HikariCP. I have set the pool size to 20.
In quartz scheduler too, I have set it to use HikariCP.
Now my questions are
Whether springboot and quartz using the same connection pool or quartz is creating a new pool?
If quartz is creating a new pool, Is there any way to configure both such that both uses same connection pool created by springboot.
What should be the optimal connection pool for 1k,10k,50k users?
Thanks in advance.
Sorry, don't have enough time to come back with a complete answer, but maybe this will help:
you're giving the connection details in 2 different places, it's safe to assume you're creating 2 datasources with different pools.
Found this: https://www.candidjava.com/tutorial/quartz-reuse-existing-data-source-connection-pool/
The number of users can't be directly correlated to connection pool size. You should look at the number of concurrent requests you want to support: for 100 req/sec, each req taking 100 ms -> you need 10 connections. This is a very simplified way of calculating but it's a starting point, after that: monitoring and adjusting should help you.
Reusing Spring's datasource in Quartz is possible, and has been the case since Spring framework 4.x.
By default, Quartz creates a new connection pool based on the provided data source properties.
Even if you instruct Quartz to use a connection pooling provider (since it supports c3p0 and HikariCP out of the box), it will still create a new connection pool using the providers. It all comes down to the implementation details of the Quartz's JobStoreCMT class, which is usually the JobStore implementation used in Spring applications by default. JobStoreCMT will always create it's own pool.
Reusing Spring's datasource in Quartz is however very trivial, using the SchedulerFactoryBean in Spring. It accepts a Spring managed datasource through the setDataSource, as shown in the following snippet
#Configuration
public class SchedulerConfig {
#Autowired private DataSource dataSource;
#Bean
public SchedulerFactoryBean schedulerFactoryBean(){
SchedulerFactoryBean factory = new SchedulerFactoryBean();
factory.setDataSource(dataSource);
// ... set other properties
return factory;
}
Internally, Spring Framework instructs Quartz to use LocalDataSourceJobStore (a Spring provided job store that extends Quartz's JobStoreCMT) to manage jobs, when a datasource is provided to SchedulerFactoryBean. LocalDataSourceJobStore has a custom Quartz connection provider that reuses the provided datasource, instead of creating a new connection.
In Spring Boot 2, this is even simpler, since it does all of the auto-wiring, to use the application's default data source. One only needs to configure Quartz to use a JDBC store type:
spring.quartz.job-store-type=jdbc
Configuring Quartz to use a data source again in the properties file, might interfere with this autowiring behavior, and result in creation of a Quartz managed datasource with a new connection pool.
Related
I am not able to use other schema (than public) for quartz tables. This is my quartz setup:
spring.quartz.job-store-type=jdbc
spring.quartz.jdbc.initialize-schema=always
spring.quartz.properties.org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.PostgreSQLDelegate
spring.quartz.properties.org.quartz.jobStore.isClustered=true
spring.quartz.properties.org.quartz.jobStore.clusterCheckinInterval=2000
spring.quartz.properties.org.quartz.scheduler.instanceId=AUTO
spring.quartz.properties.org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX
spring.quartz.properties.org.quartz.jobStore.useProperties=false
spring.quartz.properties.org.quartz.jobStore.tablePrefix=QRTZ_
And config class:
#Bean
public SchedulerFactoryBean schedulerFactory(ApplicationContext applicationContext, DataSource dataSource, QuartzProperties quartzProperties) {
SchedulerFactoryBean schedulerFactoryBean = new SchedulerFactoryBean();
AutowireCapableBeanJobFactory jobFactory = new AutowireCapableBeanJobFactory(applicationContext.getAutowireCapableBeanFactory());
Properties properties = new Properties();
properties.putAll(quartzProperties.getProperties());
schedulerFactoryBean.setOverwriteExistingJobs(true);
schedulerFactoryBean.setDataSource(dataSource);
schedulerFactoryBean.setQuartzProperties(properties);
schedulerFactoryBean.setJobFactory(jobFactory);
return schedulerFactoryBean;
}
#Bean
public Scheduler scheduler(ApplicationContext applicationContext, DataSource dataSource, QuartzProperties quartzProperties)
throws SchedulerException {
Scheduler scheduler = schedulerFactory(applicationContext, dataSource, quartzProperties).getScheduler();
scheduler.start();
return scheduler;
}
This works fine, and the tables are getting created. However I would like to have the tables in a different schema. So I set quartz to use 'quartz' schema.
spring.quartz.properties.org.quartz.jobStore.tablePrefix=quartz.QRTZ_
This is the error I'm getting:
[ClusterManager: Error managing cluster: Failure obtaining db row lock: ERROR: current transaction is aborted, commands ignored until end of transaction block] [org.quartz.impl.jdbcjobstore.LockException: Failure obtaining db row lock: ERROR: current transaction is aborted, commands ignored until end of transaction block
Any ideas on how to solve it?
It was a bold hope that "tablePrefix" can also adjsut the "db schema", (and there is no documented property concerning "db schema"), but you could get more lucky, if you configure it on the datasource.
i.e. you would introduce/configure different (spring) datasource( bean)s for every user/schema used by your application ...
(like here:) Spring Boot Configure and Use Two DataSources
or here
, then you'd wire the scheduler factory with the appropriate datasource (quartz).
schedulerFactoryBean.setDataSource(quartzDataSource);
Or via (#Autowired) parameter injection, or method invocation : #Bean initialization - difference between parameter injection vs. direct method access?
UPDATE (regarding "wiring"):
..from current spring-boot doc:
To have Quartz use a DataSource other than the application’s main DataSource, declare a DataSourcebean, annotating its #Bean method with #QuartzDataSource. Doing so ensures that the Quartz-specific DataSource is used by both the SchedulerFactoryBean and for schema initialization.
Similarly, to have Quartz use a TransactionManager other than the application’s main ... declare a TransactionManager bean, ...#QuartzTransactionManager.
You can take even more control by customizing:
spring.quartz.jdbc.initialize-schema
Database schema initialization mode.
default: embedded (embedded|always|never)
spring.quartz.jdbc.schema
Path to the SQL file to use to initialize the database schema.
default: classpath:org/quartz/impl/jdbcjobstore/tables_##platform##.sql
... properties, where ##platform## refers to your db vendor.
But it is useless for your requirement... since looking at
and complying with the original schemes - they seem schema independent/free. (So the data source approach looks more promising, herefor.)
REFS:
https://www.quartz-scheduler.org/documentation/quartz-2.3.0/configuration/ConfigJobStoreTX.html
Spring Boot Configure and Use Two DataSources
https://stackoverflow.com/a/42360877/592355
https://www.baeldung.com/spring-annotations-resource-inject-autowire
#Bean initialization - difference between parameter injection vs. direct method access?
spring.quartz.jdbc.initialize-schema
What are the possible values of spring.datasource.initialization-mode?
spring.quartz.jdbc.schema
https://github.com/quartz-scheduler/quartz/tree/master/quartz-core/src/main/resources/org/quartz/impl/jdbcjobstore
https://docs.spring.io/spring-boot/docs/current/reference/html/features.html#features.quartz
https://docs.spring.io/spring-boot/docs/current/api/org/springframework/boot/autoconfigure/quartz/QuartzDataSource.html
So the idea is that Quartz doesn't create the tables using spring.quartz.properties.org.quartz.jobStore.tablePrefix
Table names are static. Eg qrtz_triggers. as #xerx593 pointed out.
What we can do is to create the tables (manual, flyway, liquibase) in a different schema, update tablePrefix=schema.qrtz_ and it will work.
Tested with Postgres
In my Spring boot(2.0.7 RELEASE) application I am not able to manually set/override the timeout for the database connections in the application.properites file. I am using JPA, Hibernate, Tomcat connection pool and Postgres.
I've researched thoroughly and found very similar questions :
Overriding timeout for database connection in properties file
JPA query timeout parameters ignored but #Transaction annotation works
The reason I ask new question is because neither of the questions above have an accepted answer nor a confirmed working solution. I tried including each proposed solution in my application.properties file with no success.
Also, as mentioned in question 2: if I add parameter 'timeout = someSeconds' in the #Transactional annotation, the connection timeouts as expected but if I try extracting it in the application.properties it fails and timeouts for the default time. The problem here is that I want all connections to timeout in the given time not only the transactions.
Things I've tried in the application.properties (The desired timeout is 4 seconds):
spring.jpa.properties.javax.persistence.query.timeout=4000
spring.jdbc.template.query-timeout=4
spring.transaction.defaultTimeout=4
spring.datasource.tomcat.validation-query-timeout=4
Materials I've read:
http://www.masterspringboot.com/configuration/web-server/configuring-tomcat-connection-pool-on-spring-boot
https://docs.spring.io/spring-boot/docs/current/reference/html/common-application-properties.html
https://www.baeldung.com/spring-boot-tomcat-connection-pool
https://www.objectdb.com/java/jpa/query/setting#Query_Hints_
Am I missing some property? Does anyone know why the timeout can't be overridden via the application.properties file?
Thanks in advance.
There are at least 3 time-outs to configure:
Transaction timeouts, which you already did. I declared mine in the transactionManager bean:
txManager.setDefaultTimeout(myDefaultValue);
Query timeouts(which obviously does not need #transactional), which you already did and also explained here
Network timeouts(Read this excellent article).
For my case, i am using Oracle, and my bean configuration is as follows:
#Bean
public HikariDataSource dataSource() {
HikariDataSource ds = new HikariDataSource();
ds.setDriverClassName(springDatasourceDriverClassName);
ds.setJdbcUrl(springDatasourceUrl);
ds.setUsername(springDatasourceUsername);
ds.setPassword(springDatasourcePassword);
ds.setDataSourceProperties(oracleProperties());
return ds;
}
Properties oracleProperties() {
Properties properties = new Properties();
properties.put("oracle.net.CONNECT_TIMEOUT", 10000);
properties.put("oracle.net.READ_TIMEOUT", 10000);
properties.put("oracle.jdbc.ReadTimeout", 10000);
return properties;
}
And if you do not want to configure a bean for the DataSource(which is what most people will do), you can configure the network timeout properties in application.properties:
spring.datasource.hikari.data-source-properties.oracle.net.CONNECT_TIMEOUT=10000
spring.datasource.hikari.data-source-properties.oracle.net.READ_TIMEOUT=10000
spring.datasource.hikari.data-source-properties.oracle.jdbc.ReadTimeout=10000
Depending on your datasource, but you can try this:
spring.datasource.hikari.max-lifetime=1000
spring.datasource.hikari.connection-timeout=1000
spring.datasource.hikari.validation-timeout=1000
spring.datasource.hikari.maximum-pool-size=10
I am looking for the right way to set a run-time parameter when a database connection is open. My run-time parameter is actually a time zone, but I think this should work for an arbitrary parameter.
I've found following solutions, but I feel like none of these is the right thing.
JdbcInterceptor
Because Spring Boot has Apache Tomcat connection pool as default I can use org.apache.tomcat.jdbc.pool.JdbcInterceptor to intercept connections.
I don't think this interceptor provides a reliable way to perform a statement when connection is open. Possibility to intercept every statement provided by this interceptor is unnecessary to set a parameter that should be set only once.
initSQL property
Apache's pooled connection has a build-in ability to initialise itself by a statement provided by PoolProperties.initSQL parameter. This is executed in ConnectionPool.createConnection(...) method.
Unfortunately official support for this parameter has been removed from Spring and no equivalent functionality has been introduced since then.
I mean, I can still use a datasource builder like in an example below, and then hack the property into a connection pool, but this is not a good looking solution.
// Thank's to property binders used while creating custom datasource,
// the datasource.initSQL parameter will be passed to an underlying connection pool.
#Bean
#ConfigurationProperties(prefix = "datasource")
public DataSource dataSource() {
return DataSourceBuilder.create().build();
}
Update
I was testing this in a Spring Boot 1.x application. Above statements are no more valid for Spring Boot 2 applications, because:
Default Tomcat datasource was replaced by Hikari which supports spring.datasource.hikari.connection-init-sql property. It's documentation says Get the SQL string that will be executed on all new connections when they are created, before they are added to the pool.
It seems that similar property was reintroduced for Tomcat datasource as spring.datasource.tomcat.init-s-q-l.
ConnectionPreparer & AOP
This is not an actual solution. It is more like an inspiration. The connection preparer was a mechanism used to initialise Oracle connections in Spring Data JDBC Extensions project. This thing has its own problems and is no more maintained but possibly can be used as a base for similar solution.
If your parameter is actually a time zone, why don't you find a way to set this property.
For example if you want to store or read a DateTime with a predefined timestamp the right way to do this is to set property hibernate.jdbc.time_zone in hibernate entityManager or spring.jpa.properties.hibernate.jdbc.time_zone in application.properties
I am using spring-boot in my web application and use spring-jpa to read/write from/to my database. It works very well but I want to understand how to manage the database connections. Below is my properties configuration for database:
spring.datasource.url=jdbc:mysql://localhost:3306/test?characterEncoding=utf8
spring.datasource.username=user
spring.datasource.password=pwd
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.max-active=500
I have set the maximum connections to 500. When a user makes a request on my spring application, a database connection will be opened for him. After finishing the request, will spring jpa close this connection? If not, when will it close the unused connections?
I have read through the spring jpa reference document from http://docs.spring.io/spring-data/jpa/docs/current/reference/html/. But it doesn't mention anything about the connections.
When using DB connection pooling, a call to sqlconnection.close() will not necessarily close the heavyweight connection to the database, instead most often will just release the connection as re-usable in the pool. That's why it is advisable to invoke the close() on connection as soon as possible when leveraging a client side connection pool.
In your configuration, the pool will contain a maximum number of 500 connections ( it would be also good to configure maxIdle, minIdle, and minEvictableIdleTimeMillis to tune the number of ready-to-use connections and how often to release them when not used).
Some more doc here
You have already found that you can configure this from application.properties
You can find all the possible properties here.
Notice that from Spring Boot 1.4 there are datasource properties for every datasource vendor that spring integrates with, out of the box. There is spring.datasource.dbcp.*,spring.datasource.tomcat.* and so on. See 1.4 docs
If that's not enought, and you need something very specific, you can declare the datasource bean yourself. Here is the example with Tomcat datasource:
#Bean
public DataSource dataSource(){
PoolProperties p = new PoolProperties();
p.setUrl("jdbc:mysql://localhost:3306/mysql");
p.setDriverClassName("com.mysql.jdbc.Driver");
p.setUsername("root");
p.setPassword("password");
p.setJmxEnabled(true);
p.setTestWhileIdle(false);
p.setTestOnBorrow(true);
p.setValidationQuery("SELECT 1");
p.setTestOnReturn(false);
p.setValidationInterval(30000);
p.setTimeBetweenEvictionRunsMillis(30000);
p.setMaxActive(100);
p.setInitialSize(10);
p.setMaxWait(10000);
p.setRemoveAbandonedTimeout(60);
p.setMinEvictableIdleTimeMillis(30000);
p.setMinIdle(10);
p.setLogAbandoned(true);
p.setRemoveAbandoned(true);
p.setJdbcInterceptors(
"org.apache.tomcat.jdbc.pool.interceptor.ConnectionState;"+
"org.apache.tomcat.jdbc.pool.interceptor.StatementFinalizer");
DataSource datasource = new DataSource();
datasource.setPoolProperties(p);
return datasource ;
}
We are migrating our Redis stack over to Redis Cluster.
In portions of our application, this has meant that we have had to replace the Jedis object with the JedisCluster object.
In our Spring client, we use the JedisConnectionFactory to persist sessions to redis. However, this class does not appear to support JedisCluster.
Any thoughts on how one would go about wiring up a Spring application to a Redis Cluster?
I noticed that this factory implements RedisConnectionFactory which requires an instance of RedisConnection to be returned. However, this assumes that only one connection to a Redis server would be required, which is not the case in RedisCluster (it takes a set of redis servers and creates connections for all of them). As a result, I am not sure what interfaces one would need to implement in order to bring Spring into our new stack.
Any help would be greatly appreciated. Thanks!
Spring Data Redis 1.7 will support Redis Cluster using the Jedis and the lettuce driver. Release-Date ETA first of April 2016.
Code samples for Spring Data Redis Cluster are already online: https://github.com/spring-projects/spring-data-examples/tree/master/redis/cluster
Operate through Sentinel:
> /** * jedis */ #Bean public RedisConnectionFactory jedisConnectionFactory() { RedisSentinelConfiguration sentinelConfig
> = new RedisSentinelConfiguration() .master("mymaster") .sentinel("127.0.0.1", 26379) .sentinel("127.0.0.1", 26380); return
> new JedisConnectionFactory(sentinelConfig); }
http://docs.spring.io/spring-data/redis/docs/current/reference/html/#redis:sentinel