Develop Reference

Spring boot multiple external datasource stored in internal database

I have a spring boot project and I have one internal database with the configuration on the application.properties. In this database I have a Company table which contains the connection informations to external databases (all the external databases have same structure).
I created a class which create datasource when we need :
public class PgDataSource {
private static Map<Long, DataSource> dataSourceMap = new HashMap<>();
private static void createDataSource(Company company) {
HikariConfig hikariConfig = new HikariConfig();
hikariConfig.setMaximumPoolSize(10);
hikariConfig.setMinimumIdle(1);
hikariConfig.setJdbcUrl("jdbc:postgresql://"+company.getUrl()+"/"+company.getIdClient());
hikariConfig.setUsername(company.getUsername());
hikariConfig.setPassword(company.getPassword());
dataSourceMap.put(company.getId(), new HikariDataSource(hikariConfig));
}
public static DataSource getDataSource(Company company) {
if (!dataSourceMap.containsKey(company.getId()))
createDataSource(company);
return dataSourceMap.get(company.getId());
}
}
Could you tell me if this solution is the best and if I can use JPA with this solution ?
Thanks

The difficulty in your setup is not multiple datasources, but the fact that they are dynamic, i.e. determined at run time.
In addition to a DataSource JPA uses EntityManagerFactory and TransactionManager, which are determined statically i.e. at compile time. So it's not easy to use JPA with dynamic data sources.
In Spring Boot 2 you can try the AbstractRoutingDataSource, which allows to route JPA calls to a different datasource based on some (thread-bound) context. Here's an example of how it can be used and a demo application.
Alternatively you can turn your setup into a static setup, then use the regular multiple datasource approach. The downside is that the list of "companies" will be fixed at compile time, and so is probably not what you want.

Thanks it's work fine !
My solution :
I create a first datasource for my local database with #Primary annotation.
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories(
entityManagerFactoryRef = "localEntityManagerFactory",
basePackages = {"fr.axygest.akostaxi.local"}
)
public class LocalConfig {
#Primary
#Bean(name = "dataSource")
#ConfigurationProperties(prefix = "spring.datasource")
public DataSource dataSource() {
return DataSourceBuilder.create().build();
}
#Primary
#Bean(name = "localEntityManagerFactory")
public LocalContainerEntityManagerFactoryBean entityManagerFactory(
EntityManagerFactoryBuilder builder,
#Qualifier("dataSource") DataSource dataSource) {
return builder
.dataSource(dataSource)
.packages("fr.axygest.akostaxi.local.model")
.persistenceUnit("local")
.build();
}
#Primary
#Bean(name = "transactionManager")
public PlatformTransactionManager transactionManager(
#Qualifier("localEntityManagerFactory") EntityManagerFactory
entityManagerFactory
) {
return new JpaTransactionManager(entityManagerFactory);
}
}
Next, for the x external databases save in the company table of the local database, I use AbstractRoutingDataSource
I store a current context as a ThreadLocal :
public class ThreadPostgresqlStorage {
private static ThreadLocal<Long> context = new ThreadLocal<>();
public static void setContext(Long companyId) {
context.set(companyId);
}
public static Long getContext() {
return context.get();
}
}
I defined the RoutingSource to extend the AbstractRoutingDataSource :
public class RoutingSource extends AbstractRoutingDataSource
{
#Override
protected Object determineCurrentLookupKey() {
return ThreadPostgresqlStorage.getContext();
}
}
And the config class create all the databases connection saved in company table :
#Configuration
#EnableJpaRepositories(
basePackages = {"fr.axygest.akostaxi.postgresql"},
entityManagerFactoryRef = "pgEntityManager"
)
#EnableTransactionManagement
public class PgConfig {
private final CompanyRepository companyRepository;
#Autowired
public PgConfig(CompanyRepository companyRepository) {
this.companyRepository = companyRepository;
}
#Bean(name = "pgDataSource")
public DataSource pgDataSource() {
RoutingSource routingSource = new RoutingSource();
List<Company> companies = companyRepository.findAll();
HashMap<Object, Object> map = new HashMap<>(companies.size());
companies.forEach(company -> {
map.put(company.getId(), createDataSource(company));
});
routingSource.setTargetDataSources(map);
routingSource.afterPropertiesSet();
return routingSource;
}
#Bean(name = "pgEntityManager")
public LocalContainerEntityManagerFactoryBean pgEntityManager(
EntityManagerFactoryBuilder builder,
#Qualifier("pgDataSource") DataSource dataSource) {
return builder
.dataSource(dataSource)
.packages("fr.axygest.akostaxi.postgresql.model")
.persistenceUnit("pg")
.properties(jpaProperties())
.build();
}
private DataSource createDataSource(Company company) {
HikariConfig hikariConfig = new HikariConfig();
hikariConfig.setMaximumPoolSize(10);
hikariConfig.setMinimumIdle(1);
hikariConfig.setJdbcUrl("jdbc:postgresql://" + company.getUrl() + "/" + company.getIdClient());
hikariConfig.setUsername(company.getUsername());
hikariConfig.setPassword(company.getPassword());
return new HikariDataSource(hikariConfig);
}
private Map<String, Object> jpaProperties() {
Map<String, Object> props = new HashMap<String, Object>();
props.put("hibernate.dialect", "org.hibernate.dialect.MySQL5Dialect");
return props;
}
}

Spring Annotation Config with Multiple JPA Persistence Units Doesn't Persist

I started out with this guide, to migrate our xml config to an annotation config.
The current problem is, that my test a persist seems to not actually write the data (= no transaction commit). This results in the next check to fail. The environment currently has five persistence units and according entityManagerFactories and transactionManagers.
MyTest.java
#RunWith(SpringRunner.class)
#ContextConfiguration(classes = {PersistenceJPAConfig.class}, loader = AnnotationConfigContextLoader.class)
public class MyTest {
#Autowired
private MyDao testable;
#Transactional(transactionManager = "tm1") // tried with name= and without
#Test
public void crudTest() throws Exception {
// assert that the table is empty
List<MyDO> all = testable.getAll();
assertTrue(all.isEmpty());
// write one entity
MyDO anEntity = new MyDO();
testable.persist(anEntity);
// load all entities and assert that the details match
List<MyDO> allAfterInsert = testable.getAll();
// THIS FAILS
assertFalse("The database result should not be empty.", allAfterInsert.isEmpty());
}
}
PersistenceJPAConfig.java
#Configuration
#EnableTransactionManagement
#ComponentScan("my.package")
public class PersistenceJPAConfig {
#PersistenceContext(unitName = "myPersistenceUnit") // also tried
#Bean(name = "myEntityManager")
public LocalContainerEntityManagerFactoryBean entityManagerFactory() {
LocalContainerEntityManagerFactoryBean em = new LocalContainerEntityManagerFactoryBean();
em.setDataSource(dataSource());
em.setPackagesToScan(new String[] {"my.package"});
em.setPersistenceUnitName("myPersistenceUnit");
JpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter();
em.setJpaVendorAdapter(vendorAdapter);
em.setJpaProperties(additionalProperties());
return em;
}
// ... the same for the other persistenceUnits with increasing names
#Bean
public DataSource dataSource() {
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName("org.h2.Driver");
dataSource.setUrl("jdbc:h2:mem:test;DB_CLOSE_DELAY=-1");
dataSource.setUsername("sa");
dataSource.setPassword("");
return dataSource;
}
#Bean(name = "tm1")
public PlatformTransactionManager transactionManager() {
JpaTransactionManager transactionManager = new JpaTransactionManager();
transactionManager.setEntityManagerFactory(entityManagerFactory().getNativeEntityManagerFactory());
transactionManager.setPersistenceUnitName("myPersistenceUnit");
transactionManager.afterPropertiesSet();
return transactionManager;
}
// ... the same for the other persistenceUnits with increasing names
#Bean
public PersistenceExceptionTranslationPostProcessor exceptionTranslation() {
return new PersistenceExceptionTranslationPostProcessor();
}
#Bean
public PersistenceAnnotationBeanPostProcessor persistenceAnnotationBeanPostProcessor() {
return new PersistenceAnnotationBeanPostProcessor();
}
Properties additionalProperties() {
Properties properties = new Properties();
properties.setProperty("hibernate.hbm2ddl.auto", "update");
properties.setProperty("hibernate.dialect", "org.hibernate.dialect.H2Dialect");
return properties;
}
}
MyDao.java
public class MyDao {
#PersistenceContext(unitName = "myPersistenceUnit")
EntityManager em;
#Override
#Transactional(transactionManager = "tm1") // also tried with different variations like above
public Long persist(final MyDO entity) {
em.persist(entity); // tried to add an em.flush(), but that throws a TransactionRequiredException: no transaction is in progress
// handling transactions would throw IllegalStateException: Not allowed to create transaction on shared EntityManager - use Spring transactions or EJB CMT instead
return entity.getId();
}
}
The persistence units are located in the file META-INF/persistence.xml.
The tests worked with the xml configuration, but don't work with my current annotation config. Is there something I forgot? What more information could I provide?

I found a solution: Instead of entityManagerFactory().getNativeEntityManagerFactory() I used entityManagerFactory().getObject() and everything works as expected.
#Bean(name = "tm1")
public PlatformTransactionManager transactionManager() {
JpaTransactionManager transactionManager = new JpaTransactionManager();
transactionManager.setEntityManagerFactory(entityManagerFactory().getObject());
transactionManager.setPersistenceUnitName("myPersistenceUnit");
transactionManager.afterPropertiesSet();
return transactionManager;
}
I am not sure why though.

Spring Data - data is loaded but not persisted

I'm trying to use Spring Data for the first time. I've created my repository interface for Spring to implement. I use findAll() method, and it works well. However, save() and delete() methods don't work as I expect: no data is persisted, though no exception is thrown.
I tried to debug through Hibernate code, and I see event EntityDeleteAction[com.my.test.City#2] is added to the ActionQueue, but its execute() method is not called. And I did not manage to find out why.
Could you help me please? Below is database creation code, java config, my entity and repository code:
CREATE TABLE cities(
id SERIAL PRIMARY KEY,
name VARCHAR(45) NOT NULL UNIQUE
);
INSERT INTO cities(id, name) VALUES (1, 'Moscow');
INSERT INTO cities(id, name) VALUES (2, 'St.Petersburg');
City.java
#Entity
#Table(name = "cities")
public class City {
#Id #GeneratedValue(strategy = GenerationType.IDENTITY)
public Long id;
public String name;
}
CityRepository.java:
#Repository
public interface CityRepository extends CrudRepository<City, Long> {}
AppConfig.java (part)
#EnableWebMvc
#Configuration
#ComponentScan(value = ...)
#EnableJpaRepositories("com.my.test.repository")
#Import(value = {
SecurityConfig.class,
WebMvcConfig.class
})
#PropertySource("classpath:db.properties")
#EnableTransactionManagement
public class AppConfig extends WebSecurityConfigurerAdapter
#Bean
DataSource dataSource() {
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName(env.getProperty("db.driver"));
dataSource.setUrl(env.getProperty("db.url"));
dataSource.setUsername(env.getProperty("db.username"));
dataSource.setPassword(env.getProperty("db.password"));
return dataSource;
}
#Bean
SessionFactory sessionFactory() {
LocalSessionFactoryBuilder sessionBuilder = new LocalSessionFactoryBuilder(dataSource());
sessionBuilder.scanPackages("com.my.test");
sessionBuilder.addProperties(getHibernateProperties());
return sessionBuilder.buildSessionFactory();
}
#Bean
PlatformTransactionManager transactionManager() {
return new HibernateTransactionManager(sessionFactory());
}
#Bean
LocalContainerEntityManagerFactoryBean entityManagerFactory() {
LocalContainerEntityManagerFactoryBean em = new LocalContainerEntityManagerFactoryBean();
em.setDataSource(dataSource());
em.setPackagesToScan(
"com.my.test.repository",
"com.my.test.entity"
);
em.setJpaVendorAdapter(new HibernateJpaVendorAdapter());
return em;
}
private Properties getHibernateProperties() {
Properties properties = new Properties();
properties.put("hibernate.show_sql", env.getProperty("hibernate.show_sql"));
properties.put("hibernate.dialect", env.getProperty("hibernate.dialect"));
return properties;
}
Method I execute:
cityRepository.delete(2);
Thank you for help!

As M. Deinum wrote, the issue was in incorrect transaction manager. This was the solution:
delete bean sessionFactory()
replace beans:
#Bean
public JpaTransactionManager transactionManager() {
return new JpaTransactionManager(entityManagerFactory().getObject());
}
#Bean
public LocalContainerEntityManagerFactoryBean entityManagerFactory() {
LocalContainerEntityManagerFactoryBean em = new LocalContainerEntityManagerFactoryBean();
em.setDataSource(dataSource());
em.setPersistenceProviderClass(HibernatePersistenceProvider.class);
em.setPackagesToScan(
"com.my.test.repository",
"com.my.test.entity"
);
em.setJpaProperties(getHibernateProperties());
return em;
}

Spring JPA and LazyInitializationException

I'm using spring-jpa. I have 2 tests.
#Test
#Transactional
public void testFindAll() {
List<Engine> eList = engineService.findAll();
Engine e = eList.get(0); //this engine id=3
List<Translation> tList = e.getTranslations();
for(Translation t : tList) {
...
}
}
This method fails with this exception:
org.hibernate.LazyInitializationException: failed to lazily initialize
a collection of role: xxx.Engine.translations, could not initialize
proxy - no Session
However, this method works just fine:
#Test
#Transactional
public void testFindOne() {
Engine e = engineService.findOne(3);
List<Translation> tList = e.getTranslations();
for(Translation t : tList) {
...
}
}
Why translation list is successfully loaded in one case, but not in another?
EDIT: service/repo code:
public interface EngineRepository extends JpaRepository<Engine, Integer>
{
}
.
#Service
#Transactional
public class EngineService
{
#Autowired
private EngineRepository engineRepository;
public List<Engine> findAll()
{
return engineRepository.findAll();
}
public Engine findOne(Integer engId)
{
return engineRepository.findOne(engId);
}
}
.
public class Engine implements Serializable {
...
#OneToMany
#JoinColumn(name="ID", referencedColumnName="TRAN_ID", insertable=false, updatable=false, nullable=true)
#LazyCollection(LazyCollectionOption.EXTRA)
private List<Translation> translations;
...
}
Config:
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories(basePackages = {"xxx.dao"})
#ComponentScan(basePackages = {"xxx.dao", "xxx.service", "xxx.bean"})
#PropertySource("classpath:application.properties")
public class SpringDataConfig {
#Autowired
private Environment environment;
#Bean
public DataSource dataSource() {
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setUrl(environment.getProperty("db.url"));
dataSource.setDriverClassName(environment.getProperty("db.driverClass"));
dataSource.setUsername(environment.getProperty("db.username"));
dataSource.setPassword(environment.getProperty("db.password"));
return dataSource;
}
#Bean
public LocalContainerEntityManagerFactoryBean entityManagerFactory() throws NamingException {
HibernateJpaVendorAdapter hibernateJpaVendorAdapter = new HibernateJpaVendorAdapter();
hibernateJpaVendorAdapter.setDatabase(Database.POSTGRESQL);
Properties properties = new Properties();
properties.put("hibernate.dialect", environment.getProperty("hibernate.dialect"));
properties.put("hibernate.show_sql", environment.getProperty("hibernate.showSQL"));
properties.put("hibernate.format_sql", environment.getProperty("hibernate.formatSQL"));
LocalContainerEntityManagerFactoryBean entityManagerFactoryBean = new LocalContainerEntityManagerFactoryBean();
entityManagerFactoryBean.setDataSource(dataSource());
entityManagerFactoryBean.setPackagesToScan("xxx.model");
entityManagerFactoryBean.setJpaVendorAdapter(hibernateJpaVendorAdapter);
entityManagerFactoryBean.setJpaProperties(properties);
return entityManagerFactoryBean;
}
#Bean
public JpaTransactionManager transactionManager(EntityManagerFactory entityManagerFactory) {
JpaTransactionManager transactionManager = new JpaTransactionManager();
transactionManager.setEntityManagerFactory(entityManagerFactory);
return transactionManager;
}
}

I think the problem here is that the session gets closed after the first line in the first case. You should check out the JpaRepository implementation of findAll().

Integration Testing with Spring
It seems you're failing to provide a Spring Context within your TestCase, what that means? The #Transactional is being ignored. Therefore you end up with the closed session exception, because there is no transaction.
Take a look how to configure a TestCase with a Spring Context here
#RunWith(SpringJUnit4ClassRunner.class)
// ApplicationContext will be loaded from AppConfig and TestConfig
#ContextConfiguration(classes = {AppConfig.class, TestConfig.class})
public class MyTest {
#Autowired
EngineService engineService;
#Test
#Transactional
public void testFindOne() {}
#Test
#Transactional
public void testFindAll() {}
}

CrudRepository save method not returning updated entity

I'm working with Spring and in it spring-data-jpa:1.7.0.RELEASE and hibernate-jpa-2.1-api:1.0.0.Final. My database is a MySQL. In a integration test, when I save an entity, the entityRepository.save() method does not return a update version of it (with the auto-incremented id populated for example), thus I can see it is in the database.
Here's my configuration class:
//#EnableTransactionManagement
#EnableJpaRepositories(basePackages = "core.repository")
#Configuration
#PropertySource("classpath:application.properties")
public class JPAConfiguration {
#Autowired
private Environment env;
private
#Value("${db.driverClassName}")
String driverClassName;
private
#Value("${db.url}")
String url;
private
#Value("${db.username}")
String user;
private
#Value("${db.password}")
String password;
#Bean
public static PropertySourcesPlaceholderConfigurer propertySourcesPlaceholderConfigurer() {
return new PropertySourcesPlaceholderConfigurer();
}
#Bean
public DataSource dataSource() {
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName(driverClassName);
dataSource.setUrl(url);
dataSource.setUsername(user);
dataSource.setPassword(password);
return dataSource;
}
#Bean
public LocalContainerEntityManagerFactoryBean entityManagerFactory() {
HibernateJpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter();
vendorAdapter.setGenerateDdl(true);
vendorAdapter.setDatabasePlatform(env.getProperty("hibernate.dialect"));
final LocalContainerEntityManagerFactoryBean em =
new LocalContainerEntityManagerFactoryBean();
em.setJpaVendorAdapter(vendorAdapter);
em.setPackagesToScan(new String[]{"core.persistence"});
em.setDataSource(dataSource());
em.afterPropertiesSet();
return em;
}
#Bean
public EntityManager entityManager(EntityManagerFactory entityManagerFactory) {
return entityManagerFactory.createEntityManager();
}
#Bean
public JpaTransactionManager transactionManager(final EntityManagerFactory emf) {
final JpaTransactionManager transactionManager = new JpaTransactionManager();
transactionManager.setEntityManagerFactory(emf);
return transactionManager;
}
#Bean
public HibernateExceptionTranslator hibernateExceptionTranslator() {
return new HibernateExceptionTranslator();
}
}
Here's my test class:
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(classes = {JPAConfiguration.class})
//#Transactional
//#TransactionConfiguration(defaultRollback = true)
public class AgencyRepositoryIntegrationTests {
#Autowired
AgencyRepository agencyRepository;
#Test
public void testThatAgencyIsSavedIntoRepoWorks() throws Exception {
Agency c = DomainTestData.getAgency();
Agency d = agencyRepository.save(c);
// here d equals to c but both have id 0.
Collection<Agency> results = Lists.newArrayList(agencyRepository.findAll());
//here results != null and it does contain the agency object.
assertNotNull(results);
}
}
I end up commenting the Transactional annotations (both in the test and in the configuration class) due to research other questions in stackoverflow, but it did not work.
Thanks

Try to uncomment your #Transactional annotation and run. I can't see any visible problem in your snippet but there is something inside me (and it's not hunger) saying that your transaction demarcation is not right. I had similar problems like this before and it was just #transaction annotations missing or transaction misconfiguration.

Use saveAndFlush method to make sure that the entity is in the database.
Make sure to add #Transactional to your methods if a NoTransactionException appears.