How to query and retrieve results from ~100 customer databases using SpringBoot? - java

I have the scenario where I have to run 10 different queries on 100 customer databases with similar structure and then push the results to an ElasticSearch cluster for analysis. All the database connections are configured inside my applications.properties file. I decided to use Spring-Boot for the project and Java High Level Rest Client as the ElasticSearch API. However I found that in Spring-Boot I have to create an entity class for every entity and create a separate class and methods for each database connection. I am new to Spring Boot and am also not understanding the concepts of entitymanager or rowmapper. It is quite different from Java connection-statement-query-resultset format. Kindly help me
I have tried creating this Database configuration class where I tried to configure a single database reading from the properties file. I have created the basic datasource() and jdbctemplate() methods
package elasticsearch;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.jdbc.DataSourceBuilder;
import org.springframework.boot.orm.jpa.EntityManagerFactoryBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.data.jpa.repository.config.EnableJpaRepositories;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.orm.jpa.JpaTransactionManager;
import org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.annotation.EnableTransactionManagement;
import javax.persistence.EntityManagerFactory;
import javax.sql.DataSource;
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories(
entityManagerFactoryRef = "entityManagerFactory",
basePackages = { "elasticsearch" }
)
public class DatabaseConfig {
#Bean(name = "dataSource")
#ConfigurationProperties(prefix = "primary.datasource")
public DataSource dataSource() {
return DataSourceBuilder.create().build();
}
#Bean(name = "jdbcTemplate")
public JdbcTemplate jdbcTemplate(#Qualifier("dataSource") DataSource dataSource){
return new JdbcTemplate(dataSource);
}
#Bean(name = "entityManagerFactory")
public LocalContainerEntityManagerFactoryBean entityManagerFactory(
EntityManagerFactoryBuilder builder,
#Qualifier("dataSource") DataSource dataSource) {
return builder
.dataSource(dataSource)
.packages("elasticsearch")
.persistenceUnit("elasticsearch")
.build();
}
#Bean(name = "transactionManager")
public PlatformTransactionManager transactionManager(
#Qualifier("entityManagerFactory") EntityManagerFactory
entityManagerFactory
) {
return new JpaTransactionManager(entityManagerFactory);
}
}
I do not want to create an entity class for every object because the queries and also the results can vary. Also I do not want to create rowmappers because I have already written Json mappers for the rows retrieved to push it to elasticsearch

I am new to Spring Boot and am also not understanding the concepts of
entitymanager or rowmapper.
You don't show any of the details of your schema, but I don't think you need both JPA and JdbcTemplate. I'd recommend one or the other.
My preference would be JdbcTemplate.
JPA/Hibernate is overkill and complexity that you don't need. JdbcTemplate will be fine if you are comfortable with writing SQL SELECTs.
It is quite different from Java connection-statement-query-resultset format.
Not really. JdbcTemplate helps you with the boilerplate, but it's still JDBC underneath.
100 client databases will require 100 sets of URL and credentials, one for each. That is a lot of configuration. You can't get around that.
The problem is intractable if the schemas are not identical for all customers.
I would separate the two problems: querying for customer data and pushing to Elastic Search.
You only need a single RowMapper per query if the schema and query are identical for all customers.
I think a single repository/data access object can be used. You only need write and test it once, but you need to instantiate a new instance at runtime for each database connection.

Related

Spring boot hibernate no transaction is in progress

I'm using spring boot and it perfectly makes me entity manager. And I decided to test getting session factory from the entity manager and to use it for an example. But I get the next problem:javax.persistence.TransactionRequiredException: no transaction is in progress
properties
spring.datasource.url= jdbc:postgresql://localhost:5432/ring
spring.datasource.username=postgres
spring.datasource.password=root
spring.jpa.show-sql = false
spring.jpa.properties.hibernate.format_sql=false
#Note: The last two properties on the code snippet above were added to suppress an annoying exception
# that occurs when JPA (Hibernate) tries to verify PostgreSQL CLOB feature.
spring.jpa.database-platform=org.hibernate.dialect.PostgreSQL9Dialect
spring.jpa.properties.hibernate.temp.use_jdbc_metadata_defaults = false
spring.jpa.properties.hibernate.current_session_context_class = org.springframework.orm.hibernate5.SpringSessionContext
service class
package kz.training.springrest.service;
import kz.training.springrest.entity.User;
import org.hibernate.Session;
import org.hibernate.SessionFactory;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import javax.persistence.EntityManager;
import javax.persistence.EntityManagerFactory;
import javax.persistence.PersistenceContext;
import javax.persistence.PersistenceUnit;
#Service
public class UserService {
#PersistenceContext
private EntityManager entityManager;
#Transactional
public void insertUser(User user) {
SessionFactory sessionFactory = entityManager.unwrap(Session.class).getSessionFactory();
Session session = sessionFactory.getCurrentSession();
session.save(user);
}
}
runner
package kz.training.springrest.run;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.domain.EntityScan;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.data.jpa.repository.config.EnableJpaAuditing;
import org.springframework.transaction.annotation.EnableTransactionManagement;
#SpringBootApplication
#EntityScan("kz.training.springrest.entity")
#EnableTransactionManagement
#ComponentScan(basePackages="kz.training.springrest")
public class SpringrestApplication {
public static void main(String[] args) {
SpringApplication.run(SpringrestApplication.class, args);
}
}
Do you have any ideas how to solve it?
I don't quite understand why you're making your service method so unnecessarily complex. You should simply be able to do it this way
#Transactional
public void insertUser(User user) {
entityManager.persist( user );
}
If there are points where you need access to the native Hibernate Session you can simply unwrap and use the Session directly like this:
#Transactional
public void doSomethingFancyWithASession() {
Session session = entityManager.unwrap( Session.class );
// use session as needed
}
The notion here is that Spring provides you an already functional EntityManager instance by you using the #PersistenceContext annotation. That instance will safely be usable by the current thread your spring bean is being executed within.
Secondly, by using #Transactional, this causes Spring's transaction management to automatically make sure that the EntityManager is bound to a transaction, whether that is a RESOURCE_LOCAL or JTA transaction is based on your environment configuration.
You're running into your problem because of the call to #getCurrentSession().
What is happening is Spring creates the EntityManager, then inside your method when you make the call to #getCurrentSession(), you're asking Hibernate to create a second session that is not bound to the transaction started by your #Transactional annotation. In short its essentially akin to the following:
EntityManager entityManager = entityManagerFactory.createEntityManager();
entityManager.getTransaction().begin();
Session aNewSession = entityManager.unwrap( Session.class )
.getFactory()
.getCurrentSession();
// at this point entityManager is scoped to a transaction
// aNewSession is not scoped to any transaction
// this also likely uses 2 connections to the database which is a waste
So follow the paradigm I mention above and you should no longer run into the problem. You should never need to call #getCurrentSession() or #openSession() in a Spring environment if you're properly allowing Spring to inject your EntityManager instance for you.
I have same your error, when I deploy my spring boot app to WebLogic Server.
(Even It works fine if I run it directly via Eclipse (Or deploy to Tomcat) ).
I solved the problem by adding #EnableTransactionManagement to UserService.
salved my problem using in hibernate conf "hibernate.allow_update_outside_transaction","true"

Spring Boot JPA: How do I connect multiple databases?

I currently have one database connected and it is working. I would like to connect another (and eventually 2 more) databases. How do I do so? There should be a solution using only annotations and properties files.
I read this
Profile Specific Properties
and it sort of helps but I still don't know how switch from one profile to the other in the code during runtime. I'm assuming I need to be connected to one profile at a time before I try to retrieve/persist things from different databases.
I also read this question, How to use 2 or more databases with spring?, but I dont know how it works too well/ if it will apply. I'm not using a controller class and I dont know what that does. I'm also not sure how the config class they mention in the answer actually connects to the specific DO.
This is my application.properties file: (marked out username and password but its there in my file)
hibernate.dialect=org.hibernate.dialect.SQLServer2012Dialect
hibernate.show_sql=true
hibernate.format_sql=true
hibernate.default_schema=dbo
hibernate.packagesToScan=src.repositories.LMClientRepository.java
spring.jpa.generate-ddl=true
spring.jpa.hibernate.naming-strategy=org.hibernate.cfg.DefaultNamingStrategy
spring.datasource.username=***
spring.datasource.password=***
spring.datasource.url=jdbc:sqlserver://schqvsqlaod:1433;database=dbMOBClientTemp;integratedSecurity=false;
spring.datasource.testOnBorrow=true
spring.datasource.validationQuery=SELECT 1
spring.jpa.database=dbMOBClientTemp
spring.jpa.show-sql=true
spring.jpa.hibernate.dialect=org.hibernate.dialect.SQLServer2012Dialect
spring.datasource.driverClassName=com.microsoft.sqlserver.jdbc.SQLServerDriver
This is my application file:
package testApplication;
import java.util.ArrayList;
import java.util.List;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.orm.jpa.EntityScan;
import org.springframework.cache.annotation.EnableCaching;
import org.springframework.context.annotation.Bean;
import org.springframework.data.jpa.repository.config.EnableJpaRepositories;
import fileRetrieval.InputFileParse;
import lmDataObjects.LMClientDO;
import lmDataObjects.LoadMethodDO;
import repositories.LMClientRepository;
import repositories.LoadMethodRepository;
#SpringBootApplication
#EnableJpaRepositories(basePackageClasses = LoadMethodRepository.class)
#EntityScan(basePackageClasses = LoadMethodDO.class)
#EnableCaching
public class Application {
private static final Logger log = LoggerFactory.getLogger(Application.class);
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
#Bean
public CommandLineRunner demo(LoadMethodRepository lm_repo, LMClientRepository lmc_repo) {
return (args) -> {
List<LMClientDO> lmlist = InputFileParse.getMultiGroupfile();
List<String> uniqueMediaIds = new ArrayList(InputFileParse.getUniqueMediaIds());
for (int i = 0; i < InputFileParse.getUniqueMediaIds().size(); i ++){
lm_repo.save(new LoadMethodDO(uniqueMediaIds.get(i)));
}
for (int i = 0; i < lmlist.size(); i++){
lmc_repo.save(new LMClientDO(lmlist.get(i).getClientId(), lmlist.get(i).getMediaId()));
}
//Here is where I would like to do stuff with data from the other database that I have not connected yet
};
}
}
I also made a new properties file called application-MTS.properties and I put data for the new database in there. Still unsure of what to do with it.
spring.datasource.username=***
spring.datasource.password=***
spring.datasource.url=jdbc:sqlserver://SCHQVSQLCON2\VSPD:1433;database=dbMTS;integratedSecurity=false;
You will need to define multiple DataSource beans that each represent the various database connection resources you plan to use.
You will then need to add a TransactionManager and EntityManagerFactory bean definition for each of those DataSource beans.
If you intend to have each DataSource participate in a JTA transaction, you'll need to also consider configuring a JTA transaction manager rather than individual resource local transaction managers.

How to get Spring Data to handle multiple heterogeneous DataSources?

I've been successful using using the Accessing Data With JPA tutorial for Spring. I've gotten a CrudRepository of my own to work automatically by just configuring a specific DataSource #Bean, and the internal connections between these are managed by Spring Data (or Spring Boot, it's hard to tell which).
However, I can't figure out how to get that automated plumbing to handle a second DataSource #Bean. Injecting a second one causes the autoconfiguration classes to explode during startup.
Any thoughts as to how to do this? The searches I've done for this resulted in articles discussing multiple homogeneous DataSources for load balancing or other purposes, which is really not what I need. I have multiple databases with completely separate content that I need to pull into this app and I'd really like to avoid having to replicate all that automated configuration just because a second database entered the mix.
I'm hoping this is simple, but I'm fearful that it's an unsupported edge case in the autoconfiguration.
You can create two datasources and entitymanagers, one bean of them mark as #Primary
#Configuration
#EnableJpaRepositories(basePackages = "io.eddumelendez.springdatajpa.repository1")
public class FirstConfiguration {
#ConfigurationProperties(prefix = "datasource.postgres")
#Bean
#Primary
public DataSource postgresDataSource() {
return DataSourceBuilder.create().
build();
}
#Bean(name = "entityManagerFactory")
#Primary
public LocalContainerEntityManagerFactoryBean emf1(EntityManagerFactoryBuilder builder){
return builder
.dataSource(postgresDataSource())
.packages("io.eddumelendez.springdatajpa.domain1")
.persistenceUnit("users")
.build();
}
}
Configuration for another datasource:
#Configuration
#EnableJpaRepositories(basePackages = "io.eddumelendez.springdatajpa.repository2", entityManagerFactoryRef = "emf2")
public class SecondConfiguration {
#Bean
#ConfigurationProperties(prefix = "datasource.mysql")
public DataSource mysqlDataSource() {
return DataSourceBuilder.create().build();
}
#Bean
public LocalContainerEntityManagerFactoryBean emf2(EntityManagerFactoryBuilder builder){
return builder
.dataSource(mysqlDataSource())
.packages("io.eddumelendez.springdatajpa.domain2")
.persistenceUnit("customers")
.build();
}
}
Your application.properties should looks like this:
datasource.mysql.url=jdbc:mysql://localhost:3306/mysql_demo
datasource.mysql.username=root
datasource.mysql.password=root
datasource.postgres.url=jdbc:postgresql://localhost:5432/postgres_demo
datasource.postgres.username=postgres
datasource.postgres.password=postgres

How it works together these 2 Spring Java configuration classes?

I am studying for the Spring Core certification and I have the followind doubt with an exercice related to the beans configuration using the Java configuration way.
So I have the following RewardsConfig class that configure my beans (this class is into the application folder src/main/java):
package config;
import javax.sql.DataSource;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import rewards.RewardNetwork;
import rewards.internal.RewardNetworkImpl;
import rewards.internal.account.AccountRepository;
import rewards.internal.account.JdbcAccountRepository;
import rewards.internal.restaurant.JdbcRestaurantRepository;
import rewards.internal.restaurant.RestaurantRepository;
import rewards.internal.reward.JdbcRewardRepository;
import rewards.internal.reward.RewardRepository;
#Configuration
public class RewardsConfig {
#Autowired
DataSource dataSource;
#Bean
public RewardNetwork rewardNetwork(){
return new RewardNetworkImpl(accountRepository(), restaurantRepository(), rewardRepository());
}
#Bean
public AccountRepository accountRepository(){
JdbcAccountRepository repository = new JdbcAccountRepository();
repository.setDataSource(dataSource);
return repository;
}
#Bean
public RestaurantRepository restaurantRepository(){
JdbcRestaurantRepository repository = new JdbcRestaurantRepository();
repository.setDataSource(dataSource);
return repository;
}
#Bean
public RewardRepository rewardRepository(){
JdbcRewardRepository repository = new JdbcRewardRepository();
repository.setDataSource(dataSource);
return repository;
}
}
As you can see I declare 4 methods that are used to create 4 beans and that specify the dependency that occurs among these beans.
So I have a RewardNetwork bean that is implemented by RewardNetworkImpl class that depends from the following 3 beans: AccountRepository, RestaurantRepository and RewardRepository.
Is it the correct interpretation of the Java configuration is Spring?
Can I say for example that RewardNetwork is the declared bean and that RewardNetworkImpl its the current implementation of this bean?
All the 3beans (AccountRepository, RestaurantRepository and RewardRepository) depends by another bean dataSource that, as you can see in the previous code snippet, is declared as #Autowired:
#Autowired
DataSource dataSource;
This bean is not declared in this configuration class because it changes according to the environment (test, developt, production).
So, in my case it is declared into the unit test folder src/test/java:
package rewards;
import javax.sql.DataSource;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseBuilder;
#Configuration
public class TestInfrastructureConfig {
/**
* Creates an in-memory "rewards" database populated
* with test data for fast testing
*/
#Bean
public DataSource dataSource(){
return
(new EmbeddedDatabaseBuilder())
.addScript("classpath:rewards/testdb/schema.sql")
.addScript("classpath:rewards/testdb/test-data.sql")
.build();
}
}
So the dataSource bean define a datasource that is valid only for the test environment (used when I perform a unit test).
Now my doubt is: I have 2 different configuration classes and the dataSource bean is not definied into the RewardsConfig configuration class that contains the 3 beans that use it. Why I can't not use the #Import annotation to use it into RewardsConfig?
Something like it:
#Import(TestInfrastructureConfig.class)
How it work exactly?
Tnx
You don't have to import beans to make them available for autowiring. #Import is used to add extra configuration classes.
You really don't want to hard-import a test configuration class, because then your production code is referring to test-only code (and, in this case, always activating it). Instead, think of your configuration class more like an abstract class: declare autowired beans, but don't worry about how they get there. The downstream (runtime) configuration will supply them, and you don't need to know how. Maybe you're supplying an in-memory H2 for testing and using Spring Cloud Connectors for actual runs, doesn't matter.

Multiple jpa:repositories in xml config, how to configure with #EnableJPARepositories using Spring java config?

I have researched and found an explaination and sample code as to how to use spring data jpa with multiple datasources which refers to configuring multiple jpa:repositories in the xml configuration as follows:
<jpa:repositories base-package="org.springframework.data.jpa.repository.sample"
entity-manager-factory-ref="entityManagerFactory">
<repository:exclude-filter type="assignable" expression="org.springframework.data.jpa.repository.sample.AuditableUserRepository" />
</jpa:repositories>
<jpa:repositories base-package="org.springframework.data.jpa.repository.sample"
entity-manager-factory-ref="entityManagerFactory-2"
transaction-manager-ref="transactionManager-2">
<repository:include-filter type="assignable" expression="org.springframework.data.jpa.repository.sample.AuditableUserRepository" />
</jpa:repositories>
How would you declare both of the above jpa:repositories configurations using java configuration and the #EnableJpaRepositories annotation?
The annotation seems to support only one set of attributes (i.e. for one jpa:repository only) and it is not possible to declare the annotation multiple times.
I created a 'minimal' multiple datasource project to help me work out how to do this. There are 7 Java classes and other config in there, so I will only post key extracts in this answer. You can get the full project from GitHub: https://github.com/gratiartis/multids-demo
The demo sets up two JPA entities:
#Entity public class Foo { /* Constructors, fields and accessors/mutators */ }
#Entity public class Bar { /* Constructors, fields and accessors/mutators */ }
Associated with these we will create two repositories. Thanks to the awesomeness of Spring Data, we can get ourselves some pretty full-featured repositories purely by defining interfaces which extend JpaRepository:
public interface FooRepository extends JpaRepository<Foo, Long> {}
public interface BarRepository extends JpaRepository<Bar, Long> {}
Now we need to ensure that each of these maps to a table in its own database.
To achieve this, we will need two separate entity managers, each of which has a different datasource. However, in a Spring Java config #Configuration class, we can only have one #EnableJpaRepositories annotation and each such annotation can only reference one EntityManagerFactory. To achieve this, we create two separate #Configuration classes: FooConfig and BarConfig.
Each of these #Configuration classes will define a DataSource based on an embedded HSQL database:
#Bean(name = "fooDataSource")
public DataSource dataSource() {
return new EmbeddedDatabaseBuilder()
.setName("foodb").setType(EmbeddedDatabaseType.HSQL).build();
}
#Bean(name = "barDataSource")
public DataSource dataSource() {
return new EmbeddedDatabaseBuilder()
.setName("bardb").setType(EmbeddedDatabaseType.HSQL).build();
}
#Bean(name = "barEntityManagerFactory")
public EntityManagerFactory entityManagerFactory() {
LocalContainerEntityManagerFactoryBean lef =
new LocalContainerEntityManagerFactoryBean();
lef.setDataSource(dataSource());
lef.setJpaVendorAdapter(jpaVendorAdapter);
lef.setPackagesToScan("com.sctrcd.multidsdemo.domain.bar");
lef.setPersistenceUnitName("barPersistenceUnit");
lef.afterPropertiesSet();
return lef.getObject();
}
#Bean(name = "fooEntityManagerFactory")
public EntityManagerFactory entityManagerFactory() {
LocalContainerEntityManagerFactoryBean lef =
new LocalContainerEntityManagerFactoryBean();
lef.setDataSource(dataSource());
lef.setJpaVendorAdapter(jpaVendorAdapter);
lef.setPackagesToScan("com.sctrcd.multidsdemo.domain.foo");
lef.setPersistenceUnitName("fooPersistenceUnit");
lef.afterPropertiesSet();
return lef.getObject();
}
Each configuration should define an EntityManagerFactory, as above, which references its own dataSource() #Bean method. It also defines a path to the #Entity beans which it manages. You need to make sure that #Entity beans for different data sources are in different packages.
At this point it's worth noting that if each of these configurations uses the default namings for key persistence beans (i.e. entityManagerFactory), then Spring will see that there are two beans with the EntityManager interface, both of which have the same name. So one will be chosen. This leads to errors such as:
Not an managed type: class com.sctrcd.multidsdemo.domain.bar.Bar
This can be seen in the branch of the demo project here: https://github.com/gratiartis/multids-demo/tree/1-unnamed-entitymanager-beans
This is because in that example, Spring has wired up the beans relating to the "foodb" database, and Bar is not an entity in that database. Unfortunately the BarRepository has been wired up with the Foo entity manager.
We resolve this issue by naming all our beans in each of config class. i.e.
#Bean(name = "fooDataSource") public DataSource dataSource() { .. }
#Bean(name = "fooEntityManager") public EntityManager entityManager() { .. }
At this point if you were to run the tests in the project, you might see warnings such as:
No bean named 'entityManagerFactory' is defined.
This is because ... drumroll ... we do not have an EntityManagerFactory with the default name "entityManagerFactory". We have one called "fooEntityManagerFactory" and another called "barEntityManagerFactory". Spring is looking for something with a default name, so we need to instruct it to wire things up differently.
As it turns out, this is incredibly simple to do. We just need to put the correct references in the #EnableJpaRepositories annotation for each #Configuration class.
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories(
entityManagerFactoryRef = "fooEntityManagerFactory",
transactionManagerRef = "fooTransactionManager",
basePackages = {"com.sctrcd.multidsdemo.integration.repositories.foo"})
public class FooConfig {
// ...
}
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories(
entityManagerFactoryRef = "barEntityManagerFactory",
transactionManagerRef = "barTransactionManager",
basePackages = { "com.sctrcd.multidsdemo.integration.repositories.bar" })
public class BarConfig {
// ...
}
As you can see, each of these #EnableJpaRepositories annotations defines a specific named EntityManagerFactory and PlatformTransactionManager. They also specify which repositories should be wired up with those beans. In the example, I have put the repositories in database-specific packages. It is also possible to define each individual repository by name, by adding includeFilters to the annotation, but by segregating the repositories by database, I believe that things should end up more readable.
At this point you should have a working application using Spring Data repositories to manage entities in two separate databases. Feel free to grab the project from the link above and run the tests to see this happening. Hopefully this answer is useful to more folks, as I have spent a decent amount of time working out to do this as cleanly as possible with as little code as I could manage. Any ideas for improvement of the answer or demo project are welcome.
You may try put it on two #Configuration classes (one #EnableJpaRepositories per #Configuration).

Categories

Resources