Thanks for reading this.
I have 2 MySQL databases - master for writes, slave for reads. The perfect scenario I imagine is that my app uses connection to master for readOnly=false transactions, slave for readOnly=true transactions.
In order to implement this I need to provide a valid connection depending on the type of current transaction. My data service layer should not know about what type of connection it uses and just use the injected SqlMapClient (I use iBatis) directly. This means that (if I get it right) the injected SqlMapClients should be proxied and the delegate should be chosen at runtime.
public class MyDataService {
private SqlMapClient sqlMap;
#Autowired
public MyDataService (SqlMapClient sqlMap) {
this.sqlMap = sqlMap;
}
#Transactional(readOnly = true)
public MyData getSomeData() {
// an instance of sqlMap connected to slave should be used
}
#Transactional(readOnly = false)
public void saveMyData(MyData myData) {
// an instance of sqlMap connected to master should be used
}
}
So the question is - how can I do this?
Thanks a lot
It's an interesting idea, but you'd have a tough job on your hands. The readOnly attribute is intended as a hint to the transaction manager, and isn't really consulted anywhere meaningful. You'd have to rewrite or extend multiple Spring infrastructure classes.
So unless you're hell-bent on getting this working a you want, your best option is almost certainly to inject two separate SqlMapClient objects into your DAO, and for the methods to pick the appropriate one. The #Transactional annotations would also need to indicate which transaction manager to use (assuming you're using DataSourceTransactionManager rather than JpaTransactionManager), taking care to match the transaction manager to the DataSource used by the SqlMapClient.
Related
I have 2 datasources say dataSourceA and dataSourceB but based upon few calculations, I need to execute the same query in different schemas. Also, it is going to be executed in either of the schemas.
so, at DAO layer , I have one setDataSource() method which is #autowired to the dataSourceA, thus,returning the JDBCTemplate with former DataSource. How can I implement the dataSourceB changes using the same JDBCTemplate as it will be difficult to change at every DAO layer as entire application change will be required.
You could you inject both datasources and select the datasource inside your method according to your logic:
public class SomeDaoImpl implements SomeDao {
private final JdbcTemplate jdbcTemplateA;
private final JdbcTemplate jdbcTemplateB;
#Autowired
public SomeDaoImpl(JdbcTemplate jdbcTemplateA, JdbcTemplate jdbcTemplateB) {
// injecting both JdbcTemplate instances
this.jdbcTemplateA = jdbcTemplateA;
this.jdbcTemplateB = jdbcTemplateB;
}
public void businessLogicMethod(...) {
// choosing the actual template to be used according to your logic
JdbcTemplate jdbcTemplate = chooseTemplate(...);
// now using the template to execute a query
jdbcTemplate.execute(...);
}
}
Another option would be to instantiate two SomeDaoImpl instances and inject one JdbcTemplate into each of them, and select the DAO instance in your service layer.
But both these solutions have a flaw: transaction is usually initiated in the service layer (with an interceptor, for example), and it has no idea that you are going to route your requests to another datasource; so it could happen that a transaction starts on one datasource, but the query is executed on another one.
So the clearest solution would be to go one level up and instantiate 2 services, in each of them DAOs with different JdbcTemplate instances. Of course, 2 transaction managers will have to be configured and carefully wired (for example, via #Transactional("transactionManagerA")). More information on this here Spring - Is it possible to use multiple transaction managers in the same application?
I have a question regarding #Transactional annotation.
Nothing special defined, so as I understand is PROPAGATION_REQUIRED
Let’s say I have a transactional annotation which on both service and dao layer.
Service
#Transactional
public long createStudentInDB(Student student) {
final long id = addStudentToDB (student);
addStudentToCourses (id, student.getCourseIds());
return id;
}
private long addStudentToDB (Student student) {
StudentEntity entity = new StudentEntity ();
convertToEntity(student, entity);
try {
final id = dao.create(entity);
} catch (Exception e){
//
}
return id;
}
private void addStudentToCourses (long studentId, List<String> coursesIds){
//add user to group
if(coursesIds!= null){
List<StudentCourseEntity> studentCourses = new ArrayList<>();
for(String coursesId: coursesIds){
StudentCourseEntity entity = new StudentCourseEntity ();
entity.setCourseId(coursesId);
entity.setStudentId(userId);
studentCourses.add(studentId);
}
anotherDao.saveAll(studentCourses);
}
}
DAO
#Transactional
public UUID create(StudentEntity entity) {
if ( entity == null ) { throw new Exception(//…); }
getCurrentSession().save(entity);
return entity.getId();
}
ANOTHER DAO:
#Transactional
public void saveAll(Collection< StudentCourseEntity > studentCourses) {
List< StudentCourseEntity > result = new ArrayList<>();
if(studentCourses!= null) {
for (StudentCourseEntity studentCourse : studentCourses) {
if (studentCourse!= null) {
save(studentCourse);
}
}
}
}
Despite the fact that’s not optimal, it seems it causing deadlocks.
Let’s say I have max 2 connections to the database.
And I am using 3 different threads to run the same code.
Thread-1 and thread-2 receive a connection, thread-3 is not getting any connection.
More than that, it seems that thread-1 become stuck when trying to get a connection in dao level, same for thread-2. Causing a deadlock.
I was sure that by using propagation_required this would not happen.
Am I missing something?
What’s the recommendation for something like that? Is there a way I can have #transactional on both layers? If not which is preferred?
Thanks
Fabrizio
As the dao.doSomeStuff is expected to be invoked from within other transactions I would suggest you to configure this method as:
#Transaction(propagation=REQUIRES_NEW)
Thanks to that the transaction which is invoking this method will halted until the one with REQUIRES_NEW will be finished.
Not sure if this is the fix for your particular deadlock case but your example fits this particular set-up.
You are right, Propagation.REQUIRED is the default. But that also means that the second (nested) invocation on dao joins / reuses the transaction created on service level. So there is no need to create another transaction for the nested call.
In general Spring (on high level usage) should manage resource handling by forwarding it to the underlying ORM layer:
The preferred approach is to use Spring's highest level template based
persistence integration APIs or to use native ORM APIs with
transaction- aware factory beans or proxies for managing the native
resource factories. These transaction-aware solutions internally
handle resource creation and reuse, cleanup, optional transaction
synchronization of the resources, and exception mapping. Thus user
data access code does not have to address these tasks, but can be
focused purely on non-boilerplate persistence logic.
Even if you handle it on your own (on low level API usage) the connections should be reused:
When you want the application code to deal directly with the resource
types of the native persistence APIs, you use these classes to ensure
that proper Spring Framework-managed instances are obtained,
transactions are (optionally) synchronized, and exceptions that occur
in the process are properly mapped to a consistent API.
...
If an existing transaction already has a connection synchronized
(linked) to it, that instance is returned. Otherwise, the method call
triggers the creation of a new connection, which is (optionally)
synchronized to any existing transaction, and made available for
subsequent reuse in that same transaction.
Source
Maybe you have to find out what is happening down there.
Each Session / Unit of Work will be bound to a thread and released (together with the assigned connection) after the transaction has ended. Of course when your thread gets stuck it won't release the connection.
Are you sure that this 'deadlock' is caused by this nesting? Maybe that has another reason. Do you have some test code for this example? Or a thread dump or something?
#Transactional works by keeping ThreadLocal state, which can be accessed by the (Spring managed) Proxy EntityManager. If you are using Propagation.REQUIRED (the default), and you have a non-transactional method which calls two different DAOs (or two Transactional methods on the same DAO), you will get two transactions, and two calls to acquire a pooled connection. You may get the same connection twice or two different connections, but you should only use one connection at the time.
If you call two DAOs from a #Transactional method, there will only be one transaction, as the DAO will find and join the existing transaction found in the ThreadLocal state, again you only need one connection from the pool.
If you get a deadlock then something is very wrong, and you may want to debug when your connections and transaction are created. A transaction is started by calling Connection.setAutoCommit(false), in Hibernate this happens in org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor#begin(). Connections are managed by a class extending org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor so these are some good places to put a break point, and trace the call-stack back to your code to see which lines created connections.
I've been struggling for a few days to get this working, but it seems that I cannot find a solution to it. That's why I'd like to ask it here.
Short version
I have a multi tenant implementation which works with Spring boot, Spring Data JPA and Hibernate. This works like a charm. But now I'd like to implement a functionality where I switch the database (data source) during a single transaction. For example I use similar code in my service class
#Autowired
private CustomRepository customRepository;
#Autorwired
private CustomTenantIdentifierResolver customResolver;
#Transactional
public Custom getCustom(String name) {
// Set the datasource to "one";
this.customResolver.setIdentifier("one");
Custom result = this.customRepository.findOneByName(name);
//If the result is null, switch datasource to default and try again
this.customResolver.setIdentifier("default");
result = this.customRepository.findOneByName(name);
return result;
}
The problem is, my data source does not switch. It uses the same source for the second request. I guess I'm doing something terribly wrong here.
What is the correct way to switch the data source during a single transaction?
EDIT (07-06-2016)
Since I noticed that switching the data source for a single transaction is not going to work, I'll add a followup.
Would it be possible to switch the data source in between two transactions for a single user request? If so, what would be the correct way to do this?
Long Version
Before moving on, I'd like to mention that my multi tenant implementation is based on the tutorial provided on this blog.
Now, my goal is to use the default data source as a fallback when the dynamic one (chosen by a custom identifier) fails to find a result. All this needs to be done in a single user request. It doesn't make a difference in the solution uses a single or multiple transactional annotated methods.
Until now I tried a couple things, one of them is described above, another includes the use of multiple transaction managers. That implementation uses a configuration file to create two transaction manager beans which each a different data source.
#Configuration
#EnableTransactionManagement
public class TransactionConfig {
#Autowired
private EntityManagerFactory entityManagerFactory;
#Autowired
private DataSourceProvider dataSourceProvider;
#Bean(name = "defaultTransactionManager")
public PlatformTransactionManager defaultTransactionManager() {
JpaTransactionManager jpaTransactionManager = new JpaTransactionManager();
jpaTransactionManager.setEntityManagerFactory(entityManagerFactory);
jpaTransactionManager.setDataSource(dataSourceProvider.getDefaultDataSource());
jpaTransactionManager.afterPropertiesSet();
return jpaTransactionManager;
}
#Bean(name = "dynamicTransactionManager")
public PlatformTransactionManager dynamicTransactionManager() {
JpaTransactionManager jpaTransactionManager = new JpaTransactionManager();
jpaTransactionManager.setEntityManagerFactory(entityManagerFactory);
jpaTransactionManager.afterPropertiesSet();
return jpaTransactionManager;
}
}
Next I split the service method into two separate ones and added the #Transactional annotation including the right bean name
#Transactional("dynamicTransactionManager")
public Custom getDynamicCustom(String name) {
...stuff...
}
#Transactional("defaultTransactionManager")
public Custom getDefaultCustom(String name) {
...stuff...
}
But it didn't make any difference, the first data source was still used for the second method call (which should use the default transaction manager).
I hope someone can help me find a solution to this.
Thanks in advance.
Spring provides a variation of DataSource, called AbstractRoutingDatasource. It can be used in place of standard DataSource implementations and enables a mechanism to determine which concrete DataSource to use for each operation at runtime. All you need to do is to extend it and to provide an implementation of an abstract determineCurrentLookupKey method.
Keep in mind that determineCurrentLookupKey method will be called whenever TransactionsManager requests a connection. So, if you want to switch DataSource, you just need to open new transaction.
You can find example here
http://fedulov.website/2015/10/14/dynamic-datasource-routing-with-spring/
You can't just move a transaction over to another datasource. While there is a concept of distributed (or XA) transactions, it consists of separate transactions (in separate data sources) that are treated as if they were part of a single (distributed) transaction.
I do not know if it is possible, but i think you should try to avoid switching source during a transaction, for the following reason:
If an error occurs during the second request you will want to roll back the entire transaction, which means switching back to the old source. In order to be able to do that you will need to hold an open connection to that old source: When the transaction is complete you will need to confirm the transaction to that old source.
I would recommend to rethink if you really want this, beside the point if it is possible at all.
Let's say there are #Service and #Repository interfaces like the following:
#Repository
public interface OrderDao extends JpaRepository<Order, Integer> {
}
public interface OrderService {
void saveOrder(Order order);
}
#Service
public class OrderServiceImpl implements OrderService {
#Autowired
private OrderDao orderDao;
#Override
#Transactional
public void saveOrder(Order order) {
orderDao.save(order);
}
}
This is part of working application, everything is configured to access single database and everything works fine.
Now, I would like to have possibility to create stand-alone working instance of OrderService with auto-wired OrderDao using pure Java with jdbcUrl specified in Java code, something like this:
final int tenantId = 3578;
final String jdbcUrl = "jdbc:mysql://localhost:3306/database_" + tenantId;
OrderService orderService = someMethodWithSpringMagic(appContext, jdbcUrl);
As you can see I would like to introduce multi-tenant architecture with tenant per database strategy to existing Spring-based application.
Please note that I was able to achieve that quite easily before with self-implemented jdbcTemplate-like logic also with JDBC transactions correctly working so this is very valid task.
Please also note that I need quite simple transaction logic to start transaction, do several requests in service method in scope of that transaction and then commit it/rollback on exception.
Most solutions on the web regarding multi-tenancy with Spring propose specifying concrete persistence units in xml config AND/OR using annotation-based configuration which is highly inflexible because in order to add new database url whole application should be stopped, xml config/annotation code should be changed and application started.
So, basically I'm looking for a piece of code which is able to create #Service just like Spring creates it internally after properties are read from XML configs / annotations. I'm also looking into using ProxyBeanFactory for that, because Spring uses AOP to create service instances (so I guess simple good-old re-usable OOP is not the way to go here).
Is Spring flexible enough to allow this relatively simple case of code reuse?
Any hints will be greatly appreciated and if I find complete answer to this question I'll post it here for future generations :)
HIbernate has out of the box support for multi tenancy, check that out before trying your own. Hibernate requires a MultiTenantConnectionProvider and CurrentTenantIdentifierResolver for which there are default implementations out of the box but you can always write your own implementation. If it is only a schema change it is actually pretty simple to implement (execute a query before returning the connection). Else hold a map of datasources and get an instance from that, or create a new instance.
About 8 years ago we already wrote a generic solution which was documented here and the code is here. It isn't specific for hibernate and could be used with basically anything you need to switch around. We used it for DataSources and also some web related things (theming amongst others).
Creating a transactional proxy for an annotated service is not a difficult task but I'm not sure that you really need it. To choose a database for a tenantId I guess that you only need to concentrate in DataSource interface.
For example, with a simple driver managed datasource:
public class MultitenancyDriverManagerDataSource extends DriverManagerDataSource {
#Override
protected Connection getConnectionFromDriverManager(String url,
Properties props) throws SQLException {
Integer tenant = MultitenancyContext.getTenantId();
if (tenant != null)
url += "_" + tenant;
return super.getConnectionFromDriverManager(url, props);
}
}
public class MultitenancyContext {
private static ThreadLocal<Integer> tenant = new ThreadLocal<Integer>();
public static Integer getTenantId() {
return tenant.get();
}
public static void setTenatId(Integer value) {
tenant.set(value);
}
}
Of course, If you want to use a connection pool, you need to elaborate it a bit, for example using a connection pool per tenant.
I've read about AbstractRoutingDataSource and the standard ways to bind a datasource dynamically in this article:
public class CustomerRoutingDataSource extends AbstractRoutingDataSource {
#Override
protected Object determineCurrentLookupKey() {
return CustomerContextHolder.getCustomerType();
}
}
It uses a ThreadLocal context holder to "set" the DataSource:
public class CustomerContextHolder {
private static final ThreadLocal<CustomerType> contextHolder =
new ThreadLocal<CustomerType>();
public static void setCustomerType(CustomerType customerType) {
Assert.notNull(customerType, "customerType cannot be null");
contextHolder.set(customerType);
}
public static CustomerType getCustomerType() {
return (CustomerType) contextHolder.get();
}
// ...
}
I have a quite complex system where threads are not necessarily in my control, say:
Scheduled EJB reads a job list from the database
For each Job it fires a Spring (or Java EE) batch job.
Each job have its origin and destination databases (read from a central database).
Multiple jobs will run in parallel
Jobs may be multithreaded.
ItemReader will use the origin data source that was set for that specific job (origin data source must be bound to some repositories)
ItemWriter will use the destination data source that was set for that specific job (destination data source must also be bound to some repositories).
So I'm feeling somewhat anxious about ThreadLocal, specially, I'm not sure if the same thread will be used to handle multiple jobs. If that happens origin and destination databases may get mixed.
How can I "store" and bind a data source dynamically in a safe way when dealing with multiple threads?
I could not find a way to setup Spring to play nice with my setup and inject the desired DataSource, so I've decided to handle that manually.
Detailed solution:
I changed my repositories to be prototypes so that a new instance is constructed every time that I wire it:
#Repository
#Scope(BeanDefinition.SCOPE_PROTOTYPE)
I've introduced new setDataSource and setSchema methods in top level interfaces / implementations that are supposed to work with multiple instances / schemas.
Since I'm using spring-data-jdbc-repository my setDataSource method simple wraps the DataSource with a new JdbcTemplate and propagate the change.
setJdbcOperations(new JdbcTemplate(dataSource));
My implementation is obtaining the DataSources directly from the application server:
final Context context = new InitialContext();
final DataSource dataSource = (DataSource) context.lookup("jdbc/" + dsName);
Finally, for multiples schemas under the same database instance, I'm logging in with a special user (with the correct permissions) and using a Oracle command to switch to the desired schema:
getJdbcOperations().execute("ALTER SESSION SET CURRENT_SCHEMA = " + schema);
While this goes against the Dependency inversion principle it works and is handling my concurrency requirements very well.