I have following settings for my database (I have multiple databases, so they are configured in spring.datasource hierarchy.
spring:
datasource:
db-write:
url: jdbc:sqlserver://whatever.database.windows.net:1433;database=dbname;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;
username: 'myusername'
password: 'mynotsosecretpassword'
driver-class-name: com.microsoft.sqlserver.jdbc.SQLServerDriver
Then I am configuring my datasource here
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories(basePackages = "com.datasources.dbwrite.repository",
entityManagerFactoryRef = "entityManagerFactoryDbWrite",
transactionManagerRef= "transactionManagerDbWrite"
)
public class DataSourceConfigurationDbWrite {
#Bean
#Primary
#ConfigurationProperties("spring.datasource.db-write")
public DataSourceProperties dataSourcePropertiesDbWrite() {
return new DataSourceProperties();
}
#Bean
#Primary
#ConfigurationProperties("spring.datasource.db-write.configuration")
public DataSource dataSourceDbWrite() {
return dataSourcePropertiesDbWrite().initializeDataSourceBuilder()
.type(HikariDataSource.class).build();
}
#Primary
#Bean(name = "entityManagerFactoryDbWrite")
public LocalContainerEntityManagerFactoryBean entityManagerFactoryDbWrite(EntityManagerFactoryBuilder builder) {
return builder
.dataSource(dataSourceDbWrite())
.packages("com.datasources.dbwrite.models")
.build();
}
#Primary
#Bean
public PlatformTransactionManager transactionManagerDbWrite(
final #Qualifier("entityManagerFactoryDbWrite") LocalContainerEntityManagerFactoryBean entityManagerFactoryDbWrite) {
return new JpaTransactionManager(Objects.requireNonNull(entityManagerFactoryDbWrite.getObject()));
}
}
I am configuring my hikari datasource in dataSourceDbWrite method based on the properties i read in dataSourcePropertiesDbWrite method. I believe i need to configure properties in specific hierarchy so that dataSourceDbWrite method can easily detect which properties are needed for hikari. Is that correct?
What that hierarchy would be?
Moreover, how can and where can i find what properties i can configure for hikari? connection-timeout? connection pool size etc?
Me personally prefer application.yml than code to configurate Hikari:
spring:
datasource:
driver-class-name: com.mysql.cj.jdbc.Driver
url: {JDBC URL}
username: {USERNAME}
password: {PASSWORD}
type: com.zaxxer.hikari.HikariDataSource
hikari:
minimum-idle: 5
idle-timeout: 600000
maximum-pool-size: 10
auto-commit: true
pool-name: HikariCorePool
max-lifetime: 1800000
connection-timeout: 30000
connection-test-query: select * from information_schema.tables limit 1
(BTW, that piece of configuration was originally writen by a colleague years ago. We didn't change it and just copy-and-paste into any new projects those years.😆)
If you want to check out all configurable fields, those spring.datasource.hikari.* keys inorg.springframework.boot:spring-boot-autoconfigure:{VERSION}/META-INF/spring/spring-configuration-metadata.json may could help.
And javadoc in com.zaxxer.hikari.HikariConfigMXBean could help too.
See example in article, the properties hierarchy are according to #ConfigurationProperties's value
If we want to configure Hikari, we just need to add a #ConfigurationProperties to the data source definition:
#Bean
#ConfigurationProperties("spring.datasource.todos.hikari")
public DataSource todosDataSource() {
return todosDataSourceProperties()
.initializeDataSourceBuilder()
.build();
}
Then, we can insert the following lines into the application.properties file:
spring.datasource.todos.hikari.connectionTimeout=30000
spring.datasource.todos.hikari.idleTimeout=600000
spring.datasource.todos.hikari.maxLifetime=1800000
See relevant hikari's spring properties
spring.datasource.hikari.connection-timeout
spring.datasource.hikari.data-source-class-name
spring.datasource.hikari.data-source-properties
spring.datasource.hikari.driver-class-name
spring.datasource.hikari.idle-timeout
spring.datasource.hikari.initialization-fail-timeout
spring.datasource.hikari.jdbc-url
spring.datasource.hikari.leak-detection-threshold
spring.datasource.hikari.login-timeout
spring.datasource.hikari.max-lifetime
spring.datasource.hikari.maximum-pool-size
spring.datasource.hikari.minimum-idle
spring.datasource.hikari.validation-timeout
And explanation on each property in HikariCP, for example
connectionTimeout
This property controls the maximum number of milliseconds that a client (that's you) will wait for a connection from the pool. If this time is exceeded without a connection becoming available, a SQLException will be thrown. Lowest acceptable connection timeout is 250 ms. Default: 30000 (30 seconds)
Notice that camelCase hikari properties (connectionTimeout) is shown as snake-case in spring (connection-timeout)
Related
I am working with spring boot. I have properties defined in application.yml.
spring:
datasource:
username: username
password: password
username and password values are stored externally which program fetches during startup. let's say the bean which fetches them during startup is dbConfig
How can I inject values from dbConfgig to application.yml?
I am using spring-data-jpa autoconfigure which automatically connects to database at startup. I want these values to be loaded to application.yml before spring connects to database.
There is no need to inject the user/password in application.yml. You can set them programmatically like this:
#Configuration
public class DataSourceConfig {
#Bean
public DataSource getDataSource() {
DataSourceBuilder dataSourceBuilder = DataSourceBuilder.create();
dataSourceBuilder.driverClassName("org.h2.Driver");
dataSourceBuilder.url("jdbc:h2:mem:test");
// Take the values from external source, then set them
dataSourceBuilder.username("username");
dataSourceBuilder.password("password");
return dataSourceBuilder.build();
}
}
I think that first, you must create a thread to detect the change at your db Config file and then may you must re-init your bean (data source) to make your change effect.
See:
how-to-reinitialize-a-spring-bean
You may also try spring cloud to store properties. And you can then use with the help of placeholders.
https://cloud.spring.io/spring-cloud-config/reference/html/
Application has a default spring data source specified in application.yml
spring:
datasource:
type: com.zaxxer.hikari.HikariDataSource
url: jdbc:oracle:thin:#localhost:1521:xe
username: system
password: oracle
hikari:
poolName: Hikari
auto-commit: false
I have added configuration options for a second data source, used for a completely difference (JDBCTemplate purpose).
faas20:
ds:
url: jdbc:oracle:thin:#tldb0147vm.group.net:1760:tdb
username: ...
password: ...
Then, I add two data sources, one named, and the other default. Without the default one, liquibase fails to start.
#Configuration
public class LegacyConfiguration {
#Bean(name = "faas20")
#ConfigurationProperties(prefix = "faas20.ds")
public DataSource legacyDataSource() {
return DataSourceBuilder
.create()
.build();
}
#Bean
public DataSource defaultDataSource() {
return DataSourceBuilder
.create()
.build();
}
}
Startup of the application fails though.
The application now cannot build the default EntityManagerFactory.
Why would that be affected?
Parameter 0 of constructor in impl.OrderServiceImpl required a bean named 'entityManagerFactory' that could not be found.
Consider defining a bean named 'entityManagerFactory' in your configuration.
Without the two data sources present the application and liquibase start up as they should.
edit
I am not clear on how to configure two separate data sources,
Default Data Source for JPA
Additional Data Source for use in JDBC (and potentially other JPA classes)
It seems Hikaricp was not used.
For example, spring.datasource.maximum-pool-size is always effected.
spring.datasource.hikari.maximum-pool-size is not affected.
I set the following:
in application.yml
spring:
datasource:
....
maximum-pool-size: 10
hikari:
connection-timeout: 60000
maximum-pool-size: 5
And then I checked the number of connection by netstat command.
There ware 10 connections.
It seems that maximum-pool-size of hikari doesn't work.
Even if I deleted the spring.datasource.maximum-pool-size, the
maximum-pool-size of hikari still doesn't work.
Moreover, I set the follwoing log event, but there were no log about HikariCP.
logging:
level:
ROOT: NOTE
org.springframework: DEBUG
Of course, I built with the follwoing dependencies to make sure it exculded tomcat-jdbc:
compile("org.springframework.boot:spring-boot-starter-data-jpa") {
exclude group: 'org.apache.tomcat', module: 'tomcat-jdbc'
}
compile("org.springframework.boot:spring-boot-starter-jdbc") {
exclude group: 'org.apache.tomcat', module: 'tomcat-jdbc'
}
compile("com.zaxxer:HikariCP:2.6.0")
Could you help to how to find the problem?
I got this working in my spring boot app which needed two database connections.
Here are my Configuration beans:
#Bean
#Primary
#ConfigurationProperties("spring.datasource.primary")
public DataSourceProperties dataSourcePropertiesPrimary() {
return new DataSourceProperties();
}
#Bean
#Primary
#ConfigurationProperties("spring.datasource.primary.hikari")
public HikariDataSource dataSourcePrimary() {
return dataSourcePropertiesPrimary()
.initializeDataSourceBuilder()
.type(HikariDataSource.class)
.build();
}
And my application.properties file:
spring.datasource.primary.hikari.minimum-idle=1
spring.datasource.primary.hikari.maximum-pool-size=3
You can use Jolokia to confirm before and after the pool size.
Alternatively you can confirm by running your app in debug mode and break pointing the file HikariConfig.java on the private method validateNumerics where maxPoolSize is set.
As pointed out here, you can define two Datasources in Spring Boot in the following way:
#first db
spring.datasource.url = [url]
spring.datasource.username = [username]
spring.datasource.password = [password]
spring.datasource.driverClassName = oracle.jdbc.OracleDriver
#second db ...
spring.secondDatasource.url = [url]
spring.secondDatasource.username = [username]
spring.secondDatasource.password = [password]
spring.secondDatasource.driverClassName = oracle.jdbc.OracleDriver
#Bean
#Primary
#ConfigurationProperties(prefix="spring.datasource")
public DataSource primaryDataSource() {
return DataSourceBuilder.create().build();
}
#Bean
#ConfigurationProperties(prefix="spring.secondDatasource")
public DataSource secondaryDataSource() {
return DataSourceBuilder.create().build();
}
This seems to work fine. But when you are doing Atomikos XA-Transactions, each datasource has to has a unique resource name for the case that a recovery is necessary.
The Boot documentation defines a property for this:
spring.jta.atomikos.datasource.unique-resource-name=dataSource # The unique name used to identify the resource during recovery.
How do I provide a unique-resource-name for the primary and a different one for the secondary resource in Spring Boot?
Looks to me like XA-Transactions are supported in Boot, but only between JMS and DB ressources...
I want to be able to leverage the Spring Boot datasource auto-configuration. However it doesn't support all the features I'm using, logValidationErrors in particular.
spring:
datasource:
driverClassName: oracle.jdbc.OracleDriver
url: jdbc:jtds:sqlserver://111.11.11.11/DataBaseName
username: someuser
password: somepass
testOnBorrow: true
testWhileIdle: true
validationQuery: select /* validationQuery */ 1 from dual
minEvictableIdleTimeMillis: 1000
validationInterval: 30000
These aren't currently used:
logValidationErrors: true
maxAge: 1800000 # //30 Minute idle age
removeAbondoned: true
Can I just grab the created DataSource bean and set those values manually? Or is there a better way to extend or wrap the autoconfiguration?
See here for more about logValidationErrors, etc: https://tomcat.apache.org/tomcat-8.0-doc/jdbc-pool.html
I solved this with a BeanPostProcessor, similar to Dave Syer's suggestion:
#Override public Object postProcessBeforeInitialization(Object bean, String beanName) throws BeansException {
if (bean instanceof DataSource) {
DataSource ds = (DataSource) bean;
ds.setLogValidationErrors(true);
ds.setRemoveAbandoned(true);
ds.setMaxAge(1800000);
}
return bean;
}
I'll also likely submit a PR to get the properties added to Spring Boot itself.
There are multiple ways you can fix this issue.
First, you may want to submit a pull request for TomcatDataSourceConfiguration against the spring boot project. Adding those dependencies is straightforward (look at validationInterval in the source code for an example.
Or, you could create your own datasource the way you want. If a DataSource bean is present, boot will not attempt to create its own. Then you could just extend from TomcatDataSourceConfiguration and add any property you want by overriding dataSource. Finally, you should import your extended class so that the bean is registered, which will disable auto-configuration for it.
If you chose that last option and that works for you, it might be worthwhile to still report an issue for those properties if you believe they could be interesting to a wider audience.
If I were you I would just #Autowire the existing DataSource, downcast as necessary, and then set the extra property. It can be dangerous up do that for some beans (if they initialize themselves from their properties), but I doubt if that will be a problem here.