Develop Reference

Spring Boot/Spring Batch: Using and configuring two DataSource via app properties

I'm working with Spring Boot/Spring Batch, and need to provide two jdbc Data Sources.
I can't find a way to automatically load the config parameters from my application.properties.
//BatchConfiguration which uses both Data Sources:
#Configuration
#EnableBatchProcessing
public class BatchConfiguration {
#Bean
public JdbcBatchItemWriter<Customer> writer(#Qualifier("dataSourceOne") DataSource dataSource) {
return new JdbcBatchItemWriterBuilder<Customer>()
.itemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>())
.sql("..."))
.dataSource(dataSource)
.build();
}
#Bean
public JdbcCursorItemReader<Customer> reader(#Qualifier("dataSourceTwo") DataSource dataSource) {
return new JdbcCursorItemReaderBuilder<Customer>()
.dataSource(dataSource)
.name("myItemReader")
.sql("...")
.rowMapper(new CustomerRowMapper())
.build();
}
Above code works with the following Configuration, which has the DataSource provided directly:
//WORKING SOLUTION
#Configuration(proxyBeanMethods = false)
public class DataSourcesConfiguration {
#Bean("dataSourceOne")
#Primary
public DataSource dataSource() {
DataSourceBuilder dataSourceBuilder = DataSourceBuilder.create();
dataSourceBuilder.url("jdbc:h2:mem:AAA-h2");
dataSourceBuilder.username("AAA");
dataSourceBuilder.password("AAA");
dataSourceBuilder.driverClassName("org.h2.Driver");
return dataSourceBuilder.build();
}
#Bean("dataSourceTwo")
public DataSource dataSourceTwo() {
DataSourceBuilder dataSourceBuilder = DataSourceBuilder.create();
dataSourceBuilder.url("jdbc:oracle:thin:#AAA");
dataSourceBuilder.username("AAA");
dataSourceBuilder.password("AAA");
dataSourceBuilder.driverClassName("oracle.jdbc.driver.OracleDriver");
return dataSourceBuilder.build();
}
}
Now if I try to create the DataSource from my application properties via annotation #ConfigurationProperties, it will not work:
//NOT WORKING
#Configuration(proxyBeanMethods = false)
public class DataSourcesConfiguration {
#Bean("dataSourceOne")
#Primary
#ConfigurationProperties(prefix="data1.datasource")
public DataSource dataSource() {
return DataSourceBuilder.create().build();
}
#Bean("dataSourceTwo")
#ConfigurationProperties(prefix="data2.datasource")
public DataSource dataSourceTwo() {
return DataSourceBuilder.create().build();
}
}
application.properties:
data1.datasource.url=jdbc:h2:mem:AAA-h2
data1.datasource.username=AAA
data1.datasource.password=AAA
data1.datasource.driverClassName=org.h2.Driver
data2.datasource.url=jdbc:oracle:thin:#AAA
data2.datasource.username=AAA
data2.datasource.password=AAA
data2.datasource.driverClassName=oracle.jdbc.driver.OracleDriver
I receive the error Unable to detect database type in the non working version.
How can I correctly configure and provide a DataSource through my properties in application.properties?

The values to use in creating a DataSource are actually part of another class DataSourceProperties
which you can also use to create a DataSource.
Following is a snippet from TaskDbConfig (this class also contains information if you need more functions).
import org.springframework.boot.autoconfigure.jdbc.DataSourceProperties;
import com.zaxxer.hikari.HikariDataSource;
#Bean
#ConfigurationProperties("data1.datasource")
public DataSourceProperties data1DataSourceProperties() {
return new DataSourceProperties();
}
#Bean
#ConfigurationProperties("data1.datasource.hikari")
public HikariDataSource data1DataSource() {
// Different types are documented at
// https://docs.spring.io/spring-boot/docs/current/api/org/springframework/boot/jdbc/DataSourceBuilder.html
return data1DataSourceProperties().initializeDataSourceBuilder()
.type(HikariDataSource.class).build();
}
application.yml
data1.datasource.platform: h2
data1.datasource.url: 'jdbc:h2:mem:test'
# Hikari config properties: https://github.com/brettwooldridge/HikariCP#configuration-knobs-baby
data1.datasource.hikari.minimumIdle: 1
data1.datasource.hikari.maximumPoolSize: 4

Cyclic dependency when there are multiple DataSource defined

I wanted to declare two DataSource beans and use one of them dynamically using AbstractRoutingDataSource, which is declared as #Primary bean. Surprisingly, I was not able to run my application because of cyclic dependency:
org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaConfiguration
┌─────┐
| dataSource defined in <myclass>
↑ ↓
| readOnlyDataSource defined in <myclass>
↑ ↓
| org.springframework.boot.autoconfigure.jdbc.DataSourceInitializerInvoker
└─────┘
It is cause because of this implementation:
#SpringBootApplication
public class DemoApplication {
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
#Bean
#Primary
DataSource dataSource(#Qualifier("firstDS") DataSource firstDS,
#Qualifier("secondDS") DataSource secondDS) {
MyRoutingDataSource ds = new MyRoutingDataSource();
ds.setCurrentDS(firstDS);
return ds;
}
#Bean("firstDS")
public DataSource firstDS(DataSourceProperties properties) {
return properties.initializeDataSourceBuilder().build();
}
#Bean("secondDS")
public DataSource secondDs(DataSourceProperties properties) {
return properties.initializeDataSourceBuilder().build();
}
class MyRoutingDataSource extends AbstractRoutingDataSource {
private DataSource currentDS;
public void setCurrentDS(DataSource currentDS) {
this.currentDS = currentDS;
}
#Override
protected Object determineCurrentLookupKey() {
return currentDS;
}
}
}
Please note that I don't want to exclude DataSourceAutoConfiguration - it provides some additional functionally that I want to use in my project (e.g. DataSourceInitializer).
Could you please explain to me why it does not work? I feel that this error message is misleading. There is no cyclic dependency between HibernateJpaConfiguration and DataSourceInitializerInvoker. Both of them uses DataSource which primary definition I provide.
There is full project with that issue: https://github.com/kozub/spring-dependency-management-bug

I ran into the same problem you have, with the difference that I am not using DataSourceAutoConfiguration.
I'm not a Spring expert, so I can't tell you the root cause. But I was able to get my code to work by going from something like this, which is similar to what you posted:
#Bean
#Primary
DataSource dataSource(#Qualifier("firstDS") DataSource firstDS,
#Qualifier("secondDS") DataSource secondDS) {
MyRoutingDataSource ds = new MyRoutingDataSource();
ds.setFirstDS(firstDS);
ds.setSecondDs(secondDS);
return ds;
}
#Bean("firstDS")
public DataSource firstDS() {
return /*create first DS*/
}
#Bean("secondDS")
public DataSource secondDs(DataSourceProperties properties) {
return /*create second DS*/
}
To this:
#Bean
DataSource dataSource() {
DataSource first = /*create first DS*/
DataSource second = /*create second DS*/
MyRoutingDataSource ds = new MyRoutingDataSource();
ds.setFirstDS(first);
ds.setSecondDs(second);
return ds;
}
As you can see, I was able to solve the problem by only having one Spring bean of type DataSource. I created the two "first" and "second" DataSources inside the method which creates the routing datasource so that they don't have to be Spring beans. Having only one bean of type DataSource got rid of my circular dependency error.
This solved my problem, but you also want to use DataSourceAutoConfiguration.
I think you may be able to achieve that with something like this:
#Bean
DataSource dataSource(#Qualifier("firstDSproperties") DataSourceProperties firstDSprops,
#Qualifier("secondDSproperties") DataSourceProperties secondDSprops) {
DataSource first = firstDSprops.initializeDataSourceBuilder().build();
DataSource second = secondDSprops.initializeDataSourceBuilder().build();
MyRoutingDataSource ds = new MyRoutingDataSource();
ds.setCurrentDS(firstDS);
return ds;
}
#Bean("firstDSproperties")
#ConfigurationProperties("datasource.first")
public DataSourceProperties firstDataSourceProperties() {
return new DataSourceProperties();
}
#Bean("secondDSproperties")
#ConfigurationProperties("datasource.second")
public DataSourceProperties secondDataSourceProperties() {
return new DataSourceProperties();
}
What this code does is to make two beans of type DataSourceProperties rather than type DataSource. With DataSourceProperties beans you can still let Spring autowire your config without (hopefully) having the cyclical dependency problem caused by having multiple beans of type DataSource depending on each other.
I haven't tested doing this with DataSourceProperties since I am not using DataSourceAutoConfiguration in my code. But based on your code I think it might work.

There is slight a mistake here. Let me explain you with foo and bar DB example along with git repo reference.
Here is how your application.properties look like
# Oracle DB - "foo"
spring.datasource.url=jdbc:oracle:thin:#//db-server-foo:1521/FOO
spring.datasource.username=fooadmin
spring.datasource.password=foo123
spring.datasource.driver-class-name=oracle.jdbc.OracleDriver
# PostgreSQL DB - "bar"
bar.datasource.url=jdbc:postgresql://db-server-bar:5432/bar
bar.datasource.username=baradmin
bar.datasource.password=bar123
bar.datasource.driver-class-name=org.postgresql.Driver
Set the SQL Dialect to “default” in your application.properties to let Spring autodetect the different SQL Dialects of each datasource
spring.jpa.database=default
Package should look something like
src/main/java
- com.foobar
- foo
- domain
- repo
- bar
- domain
- repo
Here is the main part. Configuration classes
Foo Configuration class (Oracle)
package com.foobar;
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories(
entityManagerFactoryRef = "entityManagerFactory",
basePackages = { "com.foobar.foo.repo" }
)
public class FooDbConfig {
#Primary
#Bean(name = "dataSource")
#ConfigurationProperties(prefix = "spring.datasource")
public DataSource dataSource() {
return DataSourceBuilder.create().build();
}
#Primary
#Bean(name = "entityManagerFactory")
public LocalContainerEntityManagerFactoryBean
entityManagerFactory(
EntityManagerFactoryBuilder builder,
#Qualifier("dataSource") DataSource dataSource
) {
return builder
.dataSource(dataSource)
.packages("com.foobar.foo.domain")
.persistenceUnit("foo")
.build();
}
#Primary
#Bean(name = "transactionManager")
public PlatformTransactionManager transactionManager(
#Qualifier("entityManagerFactory") EntityManagerFactory
entityManagerFactory
) {
return new JpaTransactionManager(entityManagerFactory);
}
}
Bar Configuration class (postgres)
package com.foobar;
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories(
entityManagerFactoryRef = "barEntityManagerFactory",
transactionManagerRef = "barTransactionManager",
basePackages = { "com.foobar.bar.repo" }
)
public class BarDbConfig {
#Bean(name = "barDataSource")
#ConfigurationProperties(prefix = "bar.datasource")
public DataSource dataSource() {
return DataSourceBuilder.create().build();
}
#Bean(name = "barEntityManagerFactory")
public LocalContainerEntityManagerFactoryBean
barEntityManagerFactory(
EntityManagerFactoryBuilder builder,
#Qualifier("barDataSource") DataSource dataSource
) {
return
builder
.dataSource(dataSource)
.packages("com.foobar.bar.domain")
.persistenceUnit("bar")
.build();
}
#Bean(name = "barTransactionManager")
public PlatformTransactionManager barTransactionManager(
#Qualifier("barEntityManagerFactory") EntityManagerFactory
barEntityManagerFactory
) {
return new JpaTransactionManager(barEntityManagerFactory);
}
}
Your repositories would look something like
package com.foobar.bar.repo;
#Repository
public interface BarRepository extends JpaRepository<Bar, Long> {
Bar findById(Long id);
}
package com.foobar.foo.repo;
#Repository
public interface FooRepository extends JpaRepository<Foo, Long> {
Foo findById(Long id);
}
And you are done here.
You can refer code on github here

I am using multiple databases in Spring Boot, but creating domains in primary Database

I am trying to connect with two databases in Spring Boot. But two domains are creating in the primary Database. I uploaded my code. If anyone knows. I am not getting any console errors.But two domains creating in primary DB.
May I know what is region behind this?
(First Databse)
#EnableTransactionManagement
#EnableJpaRepositories(entityManagerFactoryRef = "dbEntityManagerFactory", transactionManagerRef = "dbTransactionManager", basePackages = {
"com.multipledatasources.db.domain", "com.multipledatasources.db.repository" })
public class DBConfigure {
#Autowired
private DatabaseSettings databaseSettings;
#Bean
#Primary
#ConfigurationProperties("spring.datasource1")
public DataSourceProperties productProperties() {
return new DataSourceProperties();
}
#Primary
#Bean(name = "DB")
public DataSource dataSource(DataSourceProperties dataSourceProperties) {
return dataSourceProperties.initializeDataSourceBuilder().build();
}
#Primary
#Bean(name = "dbEntityManagerFactory")
public LocalContainerEntityManagerFactoryBean dbEntityManagerFactory(EntityManagerFactoryBuilder builder,
#Qualifier("DB") DataSource DBDataSource) {
Map<String, String> map = databaseSettings.setting();
return builder.dataSource(DBDataSource)
.packages("com.multipledatasources.db.domain", "com.multipledatasources.db.repository").properties(map)
.persistenceUnit("DB").build();
}
#Primary
#Bean(name = "dbTransactionManager")
public PlatformTransactionManager dbTransactionManager(
#Qualifier("dbEntityManagerFactory") EntityManagerFactory dbEntityManagerFactory) {
return new JpaTransactionManager(dbEntityManagerFactory);
}
(Second Database)
#EnableTransactionManagement
#EnableJpaRepositories(transactionManagerRef = "sysTransactionManager", entityManagerFactoryRef = "sysEntityManagerFactory", basePackages = {
"com.multipledatasources.sys.domain", "com.multipledatasources.sys.repository" })
public class SysConfigure {
#Autowired
private DatabaseSettings databaseSettings;
#Bean
#ConfigurationProperties("spring.datasource2")
public DataSourceProperties sysProperties() {
return new DataSourceProperties();
}
#Bean(name = "sys")
public DataSource sysDataSource(DataSourceProperties dataSourceProperties) {
return dataSourceProperties.initializeDataSourceBuilder().build();
}
#Bean(name = "sysEntityManagerFactory")
public LocalContainerEntityManagerFactoryBean sysEntityManagerFactory(
EntityManagerFactoryBuilder entityManagerFactory, #Qualifier("sys") DataSource dataSource) {
Map<String, String> map = databaseSettings.setting();
return entityManagerFactory.dataSource(dataSource).properties(map)
.packages("com.multipledatasources.sys.domain", "com.multipledatasources.sys.repository")
.persistenceUnit("sys").build();
}
#Bean(name = "sysTransactionManager")
public PlatformTransactionManager sysTransactionManager(
#Qualifier("sysEntityManagerFactory") EntityManagerFactory sysEntityManagerFactory) {
return new JpaTransactionManager(sysEntityManagerFactory);
}
(Setting class)
Map<String, String> map = new HashMap<>();
map.put(AvailableSettings.SHOW_SQL, "true");
map.put(AvailableSettings.HBM2DDL_AUTO, "update");
map.put(AvailableSettings.DIALECT, "org.hibernate.dialect.MySQL5InnoDBDialect");
return map;
}
(Application.properties File)
server.port=4456
# Hibernate ddl auto (create, create-drop, validate, update)
spring.jpa.hibernate.ddl-auto=create-drop
spring.jpa.generate-ddl=true
## Spring DATASOURCE (DataSourceAutoConfiguration & DataSourceProperties)
spring.datasource1.url=jdbc:mysql://localhost:3306/DB?useSSL=false
spring.datasource1.username=root
spring.datasource1.password=sarat
## Spring DATASOURCE (DataSourceAutoConfiguration & DataSourceProperties)
spring.datasource2.url=jdbc:mysql://localhost:3306/sys?userSSL=false
spring.datasource2.username=root
spring.datasource2.password=sarat
Project Strectures

How to configure global transactionmanager in spring-boot with atomikos and hikari

This i my database configuration and app.properties
#Configuration
public class DatabaseConfig {
#Bean
#ConfigurationProperties(prefix = "datasource.two")
public HikariConfig hikariConfig2() {
return new HikariConfig();
}
#Bean
public DataSource dataSource2() {
HikariDataSource hds = new HikariDataSource(new hikariConfig2());
return hds;
}
#Bean
public JdbcTemplate jdbcTemplate2() {
return new JdbcTemplate(dataSource2());
}
#Bean
#ConfigurationProperties(prefix = "datasource.one")
public HikariConfig hikariConfig2() {
return new HikariConfig();
}
#Bean
public DataSource dataSource1() {
HikariDataSource hds = new HikariDataSource(hikariConfig1());
return hds;
}
#Bean
public JdbcTemplate jdbcTemplate1() {
return new JdbcTemplate(dataSource1());
}
}
and
datasource.one.jdbc-url=...
datasource.one.username=...
datasource.one.password=...
datasource.one.driver-class-name=...
datasource.two.jdbcUrl=...
datasource.two.username=...
datasource.two.password=...
datasource.two.driver-class-name=...
How should I create global transaction manager which will support two phase commit. I want to run this app inside docker with Tomcat so no JavaEE.
Also no hibernate. I am using sql-processor with spring-stack:
PS: In some very old article I found this but idk if it rigth sollution
#Bean
public DataSourceTransactionManager tm1() {
DataSourceTransactionManager txm = new DataSourceTransactionManager(refDataSource());
return txm;
}
#Bean
public DataSourceTransactionManager tm2() {
DataSourceTransactionManager txm = new DataSourceTransactionManager(logDataSource());
return txm;
}
#Bean(name = "transactionManager")
#Primary
public PlatformTransactionManager transactionManager() throws Throwable {
return new ChainedTransactionManager(tm1(), tm2());
}

Tables not getting created in multiple databases in spring boot application

I am working on spring boot multi tenancy application. I have configured multi datasources as shown below :
application.properties
spring.multitenancy.datasource1.url=jdbc:mysql://localhost:3306/db1
spring.multitenancy.datasource1.username=root
spring.multitenancy.datasource1.password=****
spring.multitenancy.datasource1.driver-class-name=com.mysql.jdbc.Driver
spring.jpa.hibernate.ddl-auto=update
spring.multitenancy.datasource2.url=jdbc:mysql://localhost:3306/db2
spring.multitenancy.datasource2.username=root
spring.multitenancy.datasource2.password=****
spring.multitenancy.datasource2.driver-class-name=com.mysql.jdbc.Driver
spring.jpa.hibernate.ddl-auto=update
spring.multitenancy.datasource3.url=jdbc:mysql://localhost:3306/db3
spring.multitenancy.datasource3.username=root
spring.multitenancy.datasource3.password=****
spring.multitenancy.datasource3.driver-class-name=com.mysql.jdbc.Driver
spring.jpa.hibernate.ddl-auto=update
DataSourceBasedMultiTenantConnectionProviderImpl.java
#Component
public class DataSourceBasedMultiTenantConnectionProviderImpl extends AbstractDataSourceBasedMultiTenantConnectionProviderImpl {
private static final long serialVersionUID = 8168907057647334460L;
private static final String DEFAULT_TENANT_ID = "tenant_1";
#Autowired
private DataSource dataSource1;
#Autowired
private DataSource dataSource2;
#Autowired
private DataSource dataSource3;
private Map<String, DataSource> map;
#PostConstruct
public void load() {
map = new HashMap<>();
map.put("tenant_1", dataSource1);
map.put("tenant_2", dataSource2);
map.put("tenant_3", dataSource3);
}
#Override
protected DataSource selectAnyDataSource() {
return map.get(DEFAULT_TENANT_ID);
}
#Override
protected DataSource selectDataSource(String tenantIdentifier) {
return map.get(tenantIdentifier);
}
}
MultitenancyProperties.java
#ConfigurationProperties("spring.multitenancy")
public class MultitenancyProperties {
#NestedConfigurationProperty
private DataSourceProperties datasource1;
#NestedConfigurationProperty
private DataSourceProperties datasource2;
#NestedConfigurationProperty
private DataSourceProperties datasource3;
public DataSourceProperties getDatasource1() {
return datasource1;
}
public void setDatasource1(DataSourceProperties datasource1) {
this.datasource1 = datasource1;
}
public DataSourceProperties getDatasource2() {
return datasource2;
}
public void setDatasource2(DataSourceProperties datasource2) {
this.datasource2 = datasource2;
}
public DataSourceProperties getDatasource3() {
return datasource3;
}
public void setDatasource3(DataSourceProperties datasource3) {
this.datasource3 = datasource3;
}
}
MultiTenancyJpaConfiguration.java
#Configuration
#EnableConfigurationProperties(JpaProperties.class)
public class MultiTenancyJpaConfiguration {
#Autowired
private DataSource dataSource;
#Autowired
private JpaProperties jpaProperties;
#Autowired
private MultiTenantConnectionProvider multiTenantConnectionProvider;
#Autowired
private CurrentTenantIdentifierResolver currentTenantIdentifierResolver;
#Bean
public LocalContainerEntityManagerFactoryBean entityManagerFactory(EntityManagerFactoryBuilder builder) {
Map<String, Object> hibernateProps = new LinkedHashMap<>();
hibernateProps.putAll(jpaProperties.getHibernateProperties(dataSource));
hibernateProps.put(Environment.MULTI_TENANT, MultiTenancyStrategy.DATABASE);
hibernateProps.put(Environment.MULTI_TENANT_CONNECTION_PROVIDER, multiTenantConnectionProvider);
hibernateProps.put(Environment.MULTI_TENANT_IDENTIFIER_RESOLVER, currentTenantIdentifierResolver);
hibernateProps.put(Environment.DIALECT, "org.hibernate.dialect.MySQLDialect");
return builder.dataSource(dataSource).packages(HotelEntity.class.getPackage().getName()).properties(hibernateProps).jta(false).build();
}
}
Application launcher
#SpringBootApplication(exclude = DataSourceAutoConfiguration.class)
#EnableConfigurationProperties(MultitenancyProperties.class)
public class Application {
public static void main(String[] args) {
ApplicationContext ctx = SpringApplication.run(Application.class, args);
}
}
When I run the boot application, all tables are created in only first data source.
1) How can I create tables in all data sources on application startup?
2) How to see connections opened/closed for each of the data sources?
3) Is there a better way of configuring multi tenancy application using spring boot for better performance?

As #Alex said, you need differnt EntityManagers, TransactionManager and Datasources. Here is how I would do it
#Configuration
#EnableJpaRepositories(
entityManagerFactoryRef = "dataSource1EntityManagerFactory",
transactionManagerRef = "dataSource1TransactionManager",
basePackageClasses = dataSource1Repository.class)
public class DataSource1Config extends SqlConfig{// Put all common code in base class SqlConfig. If not remove it
#Bean
#Primary
public DataSource dataSource1() {
//create dataSource using MultitenancyProperties::getDataSource1
}
#Primary
#Bean(name = "dataSource1TransactionManager")
PlatformTransactionManager dataSource1TransactionManager(EntityManagerFactory dataSource1EntityManagerFactory) {
JpaTransactionManager txManager = new JpaTransactionManager();
txManager.setEntityManagerFactory(dataSource1EntityManagerFactory);
return txManager;
}
#Primary
#Bean(name = "dataSource1EntityManagerFactory")
LocalContainerEntityManagerFactoryBean dataSource1EntityManagerFactory() {
LocalContainerEntityManagerFactoryBean em = new LocalContainerEntityManagerFactoryBean();
em.setDataSource(dataSource1());
em.setPackagesToScan(dataSource1Repository.class.getPackage().getName(), dataSource1ModelClass.class.getPackage().getName());
em.setPersistenceUnitName("dataSource1Db");
HibernateJpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter();
vendorAdapter.setGenerateDdl(false);
em.setJpaVendorAdapter(vendorAdapter);
return em;
}
}
You can create two other classes like this. Remember to use #Primary on only one instace of datasource, transactionmanger and entitymanager(doesn't matter which one). Another word of caution, make sure Repository classes are in different packages for all three databases.

Try adding the following properties:
spring.jpa.generate-ddl=true

You need 2 different persistence factories, not one, each should produce different EntityManagers for different datasources. Also each entity mapping should be marked to been used only with one entity manager.
See full solution here:
http://www.baeldung.com/spring-data-jpa-multiple-databases