I am using spring-boot in my web application and use spring-jpa to read/write from/to my database. It works very well but I want to understand how to manage the database connections. Below is my properties configuration for database:
spring.datasource.url=jdbc:mysql://localhost:3306/test?characterEncoding=utf8
spring.datasource.username=user
spring.datasource.password=pwd
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.max-active=500
I have set the maximum connections to 500. When a user makes a request on my spring application, a database connection will be opened for him. After finishing the request, will spring jpa close this connection? If not, when will it close the unused connections?
I have read through the spring jpa reference document from http://docs.spring.io/spring-data/jpa/docs/current/reference/html/. But it doesn't mention anything about the connections.
When using DB connection pooling, a call to sqlconnection.close() will not necessarily close the heavyweight connection to the database, instead most often will just release the connection as re-usable in the pool. That's why it is advisable to invoke the close() on connection as soon as possible when leveraging a client side connection pool.
In your configuration, the pool will contain a maximum number of 500 connections ( it would be also good to configure maxIdle, minIdle, and minEvictableIdleTimeMillis to tune the number of ready-to-use connections and how often to release them when not used).
Some more doc here
You have already found that you can configure this from application.properties
You can find all the possible properties here.
Notice that from Spring Boot 1.4 there are datasource properties for every datasource vendor that spring integrates with, out of the box. There is spring.datasource.dbcp.*,spring.datasource.tomcat.* and so on. See 1.4 docs
If that's not enought, and you need something very specific, you can declare the datasource bean yourself. Here is the example with Tomcat datasource:
#Bean
public DataSource dataSource(){
PoolProperties p = new PoolProperties();
p.setUrl("jdbc:mysql://localhost:3306/mysql");
p.setDriverClassName("com.mysql.jdbc.Driver");
p.setUsername("root");
p.setPassword("password");
p.setJmxEnabled(true);
p.setTestWhileIdle(false);
p.setTestOnBorrow(true);
p.setValidationQuery("SELECT 1");
p.setTestOnReturn(false);
p.setValidationInterval(30000);
p.setTimeBetweenEvictionRunsMillis(30000);
p.setMaxActive(100);
p.setInitialSize(10);
p.setMaxWait(10000);
p.setRemoveAbandonedTimeout(60);
p.setMinEvictableIdleTimeMillis(30000);
p.setMinIdle(10);
p.setLogAbandoned(true);
p.setRemoveAbandoned(true);
p.setJdbcInterceptors(
"org.apache.tomcat.jdbc.pool.interceptor.ConnectionState;"+
"org.apache.tomcat.jdbc.pool.interceptor.StatementFinalizer");
DataSource datasource = new DataSource();
datasource.setPoolProperties(p);
return datasource ;
}
Related
I want to create a one-shot database connection, based on spring classes.
To prevent spring loading the datasource on startup, I create it explicit when needed (only occasionally). I don't need any connection pooling, transactions and stuff like that.
DriverManagerDataSource ds = new DriverManagerDataSource(props.getUrl(), props.getUsername(), props.getPassword());
JdbcTemplate jdbc = new JdbcTemplate(ds);
jdbc.execute(...);
Question: how can I afterwards close/destroy the datasource and any open connections explicit?
I have a question about springboot, quartz scheduler and HikariCP. I am relatively new to this domain and trying to understand the relations and working.
I have gone through many questions that are either related to Springboot HikariCP or Quartz scheduler using HikariCP but none of them is able to answer my questions.
I have an application with below configurations
#Database properties
spring.datasource.url = jdbc:mysql://localhost:3306/demo?user=root&password=root&useSSL=false&serverTimezone=UTC
spring.datasource.username = root
spring.datasource.password = root
spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.MySQL5InnoDBDialect
#Hikari
spring.datasource.hikari.minimumIdle=5
spring.datasource.hikari.maximumPoolSize=20
#quartz settings
spring.quartz.properties.org.quartz.jobStore.dataSource = quartzDataSource
spring.quartz.properties.org.quartz.dataSource.quartzDataSource.driver = com.mysql.cj.jdbc.Driver
spring.quartz.properties.org.quartz.dataSource.quartzDataSource.provider=hikaricp
spring.quartz.properties.org.quartz.dataSource.quartzDataSource.URL = jdbc:mysql://localhost:3306/demo?user=root&password=root&useSSL=false&serverTimezone=UTC
spring.quartz.properties.org.quartz.dataSource.quartzDataSource.user = root
spring.quartz.properties.org.quartz.dataSource.quartzDataSource.password = root
spring.quartz.job-store-type = jdbc
spring.quartz.properties.org.quartz.threadPool.threadCount=20
By default, springboot2 uses HikariCP. I have set the pool size to 20.
In quartz scheduler too, I have set it to use HikariCP.
Now my questions are
Whether springboot and quartz using the same connection pool or quartz is creating a new pool?
If quartz is creating a new pool, Is there any way to configure both such that both uses same connection pool created by springboot.
What should be the optimal connection pool for 1k,10k,50k users?
Thanks in advance.
Sorry, don't have enough time to come back with a complete answer, but maybe this will help:
you're giving the connection details in 2 different places, it's safe to assume you're creating 2 datasources with different pools.
Found this: https://www.candidjava.com/tutorial/quartz-reuse-existing-data-source-connection-pool/
The number of users can't be directly correlated to connection pool size. You should look at the number of concurrent requests you want to support: for 100 req/sec, each req taking 100 ms -> you need 10 connections. This is a very simplified way of calculating but it's a starting point, after that: monitoring and adjusting should help you.
Reusing Spring's datasource in Quartz is possible, and has been the case since Spring framework 4.x.
By default, Quartz creates a new connection pool based on the provided data source properties.
Even if you instruct Quartz to use a connection pooling provider (since it supports c3p0 and HikariCP out of the box), it will still create a new connection pool using the providers. It all comes down to the implementation details of the Quartz's JobStoreCMT class, which is usually the JobStore implementation used in Spring applications by default. JobStoreCMT will always create it's own pool.
Reusing Spring's datasource in Quartz is however very trivial, using the SchedulerFactoryBean in Spring. It accepts a Spring managed datasource through the setDataSource, as shown in the following snippet
#Configuration
public class SchedulerConfig {
#Autowired private DataSource dataSource;
#Bean
public SchedulerFactoryBean schedulerFactoryBean(){
SchedulerFactoryBean factory = new SchedulerFactoryBean();
factory.setDataSource(dataSource);
// ... set other properties
return factory;
}
Internally, Spring Framework instructs Quartz to use LocalDataSourceJobStore (a Spring provided job store that extends Quartz's JobStoreCMT) to manage jobs, when a datasource is provided to SchedulerFactoryBean. LocalDataSourceJobStore has a custom Quartz connection provider that reuses the provided datasource, instead of creating a new connection.
In Spring Boot 2, this is even simpler, since it does all of the auto-wiring, to use the application's default data source. One only needs to configure Quartz to use a JDBC store type:
spring.quartz.job-store-type=jdbc
Configuring Quartz to use a data source again in the properties file, might interfere with this autowiring behavior, and result in creation of a Quartz managed datasource with a new connection pool.
I have a doubt of how the spring boot JDBC template works. I have read the documentation , but could not understand clearly :( When does the template opens connection , when does it gets closed . How does the transactions are handled . Does it gets opened and closed for every query execution ?
When does the template opens connection, when does it gets closed
For building JdbcTemplate you should specify the JDBC DataSource to obtain connections from:
public JdbcTemplate(DataSource dataSource)
Or:
public JdbcTemplate()
JdbcAccessor.setDataSource(javax.sql.DataSource)
Conclusively, JdbcTemplate works with this DataSource.
DataSource, depending on the implementation, may return new standard Connection objects that are not pooled or Connection objects that participate in connection pooling which can be an be recycled.
JdbcTemplate has pooled connections and releases them back to DataSource.
How does the transactions are handled
JdbcTemplate relies on database transactions.
If you want to operate transactions on service layer/business logic you need transaction management.
The simplest way is to annotate services with #Transactional or use org.springframework.transaction.support.TransactionTemplate.
I'm confused about the difference between DataSource and SessionFactory.
I think SessionFactory is a manager for retrieve Sessions (which are in fact connections to a database I guess).
DataSource has the method getConnection()
"Attempts to establish a connection with the data source that this DataSource object represents."
It means a DataSource object always works directly with the database?
If I have a pool of connections, DataSource will ask to the pool for a connection? Or will it try to get another connection to the database.
Are in fact DataSource and SessionFactory the same? Both try to give sessions/connections?
Datasource maintains the pool of connections, establishes the connection, knows how to connect etc.,
User configures data source either in the server(like websphere/weblogic) or in the spring/hibernate configuration file. Either way you give the handle of this data source to the sessionfactory during the app startup.
SessionFActory is an interface which hides all the details like opening the connection / closing the connections. Through out your development time you dont worry about these small things like loading driver/closing connection/opening etc., so you can concentrate on business logic/other important stuff.
What is the difference between javax.sql.DataSource and javax.sql.ConnectionPoolDataSource? I don't know which resource type to use when I am creating a connection pool in GlassFish for MySQL.
ConnectionPoolDataSource is just a DataSource (as it inherit's CommonDataSource which is also inherited by DataSource) with capability of Connection Pooling -
you asked : I don't know which resource type to use
It depend's on your application, many database drivers take a long time to create a new connection with database, If your application is going to create too many connection's (very frequently). use connection pooling.