Develop Reference

How to Rollback the entire process when database error occurs within the loop?

Consider two MicroServices ( 1 & 2), 1st Service retrieves the data from the user and sends the data to the 2nd Service where the data gets stored in the database. This takes place in a loop.
For example, consider the length of the loop as 5. When each time it iterates, it calls the rest service and saves the data to the database. Suppose, while iterating for the 3rd time, the service2 throws some error and the execution stops, and the remaining data are not stored in the Database. Now my question is there any way I can remove the other two data which got saved to Database successfully without writing a separate function for removing the data individually?
Is there any way to roll back the entire process when an error occurs within the loop?
Microservice-1
class MicroService1 {
#Autowired
private RestTemplate restTemplate;
void rollBack(){
String [] name={"hello","hai","hey"};
for(String n: name){
MyObject object=new MyObject();
object.setName( n );
try{
restTemplate.postForEntity("micorservice-2",object,String.class);
} catch(){
.....
}
}
}
}
Microservice-2
class MicroService2 {
#Autowired
private MyRepository repo;
//Template call comes to this function and saves the data to the database
void saveData(MyObject obj){
try{
repository.save(obj);
} catch(Exception ex) {
....
}
}
}

Every external rest call is technically independent from each other, and every request starts a new transaction in MicroService2. These transactions are also independent and unaware of each other.
Your business transaction (saving entities in a loop) spans multiple technical transactions.
is there any way I can remove the other two data which got saved to Database successfully without writing a separate function for removing the data individually? - So no, there is no out of the box solution. You have to write your custom rollback logic, to delete the saved entities from previous transactions.
If your use case allows it, you could wait for the loop to end in Microservice1 and then initiate a single external rest call to Microservice2, the payload of the call would be a list of entities, it would be a bulk save. This way, either all entities are saved, or none, plus if the save fails in Microservice2, you could also rollback Microservice1.

Try using "Transactions" or "Saga" : https://microservices.io/patterns/data/saga.html

Multiple entityManager in Spring application. Persistence of duplicate objects issue

My Spring component gets a request from a client, asks a web-service about some data and saves received objects to a database.
I identify all objects and save only new ones.
The issue occurs when the client makes two or more same requests in the same time (or due to even different user requests I receive same objects from web-service).
To describe the issue with persistence here some details. For each client request my component starts execution in a separate thread, I get a new entityManager, begin a transaction, receive a data from web-service, then I identify objects and persist new ones using given entityManager in a current transaction.
If in separate transactions I receive the same objects from web-service and if they are new ones that are not yet in database I am not able to identify them in not-commited transactions and so they are persisted in all transactions. Then all duplicate objects will be commited and saved to database.
What could be good solutions in this case? Is there any way to identify new objects properly even in different transactions? Or what approaches can be applied?
May be Spring provides some approaches to manage transactions or entityManagers so that it can help with this issue...
Note. Of course I can use database instruments to avoid saving duplicate objects but in this case it is not a very good solution.

Check if objects are present in a database before saving.
Use #UniqueConstraint or #Column(unique = true) to prevent duplicate rows, handle exceptions appropriately.
Use #Version to manage concurrent modification for existing entities. More about optimistic and pesimistic locking: Chapter 5. Locking. Related discussions: Hibernate Automatic Versioning and When to use #Version and #Audited in Hibernate?
You may use thread locks / synchronization mechanisms to ensure that requests for the same user will happen in order. However, this won't work if your service in running on more than 1 node.

So the solution in my case is the following:
Make transactions pretty small and commit every object separately.
Make unique constraints in database to prevent duplicating of
objects. This point will not help us a lot but needed for point 3.
Every commit() method we insert in try-catch block. If we try to
commit duplicate object in parallel transactions then we will receive an exception and in catch block we can check the database, select the object that is already there and work with it futher.
The example:
boolean reidentifyNeed = false;
try {
DofinService.getEntityManagerThreadLocal().getTransaction().begin();
DofinService.getEntityManagerThreadLocal().persist(entity);
try {
DofinService.getEntityManagerThreadLocal().getTransaction().commit();
//if commit is successfull
entityIdInDB = (long) entity.getId();
DofinService.getEntityManagerThreadLocal().clear();
} catch (Exception ex) {
logger.error("Error committing " + entity.getClass().getSimpleName() + " in DB. Possibly duplicate object. Will try to re-identify object. Error: " + ex.toString());
reidentifyNeed = true;
}
if(reidentifyNeed){
//need clear entityManager, because if duplicated object was persisted then during *select* an object flush() method will be executed and it will thrown ConstrainViolationException
DofinService.getEntityManagerThreadLocal().clear();
CheckSimilarObject checkSimilarObject = new CheckSimilarObject();
long objectId = checkSimilarObject.checkObject(dofinObject);
logger.warn("Re-identifying was done. EntityId = " + objectId);
entityIdInDB = objectId;
}
} catch (Exception ex) {
logger.error("Error persisting and commiting object: " + ex.toString());
}

How to write correct/reliable transactional code with JAX-RS and Spring

Basically, I am trying to understand how to write correct (or "to correctly write"?) transactional code, when developing REST service with Jax-RS and Spring. Also, we're using JOOQ for data-access. But that shouldn't be very relevant...
Consider simple model, where we have some organisations, that have these fields: "id", "name", "code". All of which must be unique. Also there's a status field.
Organization might be removed at some point. But we don't want to remove the data altogether, because we want to save it for analytical/maintenance purposes. So we just set organization 'status' field to 'REMOVED'.
Because we don't delete the organization row from the table, we can't simply put the unique constraint on the "name" column, because, we might delete organization and then create a new one with the same name. But let's assume that codes has to be unique globally, so we DO have a unique constraint on the code column.
So with that, let's see this simple example, that creates organization, performing some checks along the way.
Resource:
#Component
#Path("/api/organizations/{organizationId: [0-9]+}")
#Consumes(MediaType.APPLICATION_JSON)
#Produces(MediaTypeEx.APPLICATION_JSON_UTF_8)
public class OrganizationResource {
#Autowired
private OrganizationService organizationService;
#Autowired
private DtoConverter dtoConverter;
#POST
public OrganizationResponse createOrganization(#Auth Person person, CreateOrganizationRequest request) {
if (organizationService.checkOrganizationWithNameExists(request.name())) {
// this throws special Exception which is intercepted and translated to response with 409 status code
throw Responses.abortConflict("organization.nameExist", ImmutableMap.of("name", request.name()));
}
if (organizationService.checkOrganizationWithCodeExists(request.code())) {
throw Responses.abortConflict("organization.codeExists", ImmutableMap.of("code", request.code()));
}
long organizationId = organizationService.create(person.user().id(), request.name(), request.code());
return dtoConverter.from(organization.findById(organizationId));
}
}
DAO service looks like that:
#Transactional(DBConstants.SOME_TRANSACTION_MANAGER)
public class OrganizationServiceImpl implements OrganizationService {
#Autowired
#Qualifier(DBConstants.SOME_DSL)
protected DSLContext context;
#Override
public long create(long userId, String name, String code) {
Organization organization = new Organization(null, userId, name, code, OrganizationStatus.ACTIVE);
OrganizationRecord organizationRecord = JooqUtil.insert(context, organization, ORGANIZATION);
return organizationRecord.getId();
}
#Override
public boolean checkOrganizationWithNameExists(String name) {
return checkOrganizationExists(Tables.ORGANIZATION.NAME, name);
}
#Override
public boolean checkOrganizationWithCodeExists(String code) {
return checkOrganizationExists(Tables.ORGANIZATION.CODE, code);
}
private boolean checkOrganizationExists(TableField<OrganizationRecord, String> checkField, String checkValue) {
return context.selectCount()
.from(Tables.ORGANIZATION)
.where(checkField.eq(checkValue))
.and(Tables.ORGANIZATION.ORGANIZATION_STATUS.ne(OrganizationStatus.REMOVED))
.fetchOne(DSL.count()) > 0;
}
}
This brings some questions:
Should I put #Transactional annotation on Resource's createOrganization method? Or should I create one more service that talks to DAO and put #Transactional annotation to it's method? Something else?
What would happen if two users concurrently send request with the same "code" field. Before first transaction is commited the checks are successfully passed, so no 409 respones will be sent. Than first transaction will be committed properly, but the second one will violate DB constraint. This will throw SQLException. How to gracefully handle that? I mean I still want to show nice error message on the client side, saying that name is already used. But I can't really parse SQLException or smth.. can I?
Similar to the previous one, but this time "name" is not unique. In this case, second transaction will not violate any constraints, which leads to having two organization with the same name, that violates our buisness constraints.
Where can I see/learn tutorials/code/etc., that you consider great examples on how to write correct/reliable REST+DB code with complicated buisness logic. Github/books/blogs, whatever. I've tried to find something like that myselft, but most examples just focus on the plumbing - add these libs to maven, use these annotations, there is your simple CRUD, the end. They don't contain any transactional considirations at all. I.e.
UPDATE:
I know about isolation level and the usual error/isolation matrix (dirty reads, etc..). The problem I have is finding some "production-ready" sample to learn from. Or a good book on a subject. I still don't really get how to handle all the errors properly.. I guess I need to retry a couple of times, if transaction failed.. and than just throw some generic error and implement client, that handles that.. But do I really have to use SERIALIZABLE mode, whenever I use range queries? Because it will affect performance greatly. But otherwise how can I garantee that transaction will fail..
Anyway I've decided that for now I need more time to learn about transactions and db management in general to tackle this problem...

Generally, without talking about transactionality, endpoint should only grab parameters from the request and call the Service. It shouldn't do business logic.
It seems your checkXXX methods are part of the business logic, because they throw errors about domains-specific conflicts. Why not put them into the Service into one method, which is by the way transactional?
//service code
public Organization createOrganization(String userId, String name, String code) {
if (this.checkOrganizationWithNameExists(request.name())) {
throw ...
}
if (this.checkOrganizationWithCodeExists(code)) {
throw ...
}
long organizationId = this.create(userId, name, code);
return dao.findById(organizationId);
}
I took as your parameters are Strings, but they can be anything. I'm not sure you want to throw Responses.abortConflict in the service layer because it seems to be a REST concept, but you can define your own exception types for it if you want.
Endpoint code should look like this, however, it might contain additional try-catch block which converts the thrown exceptions to Error responses:
//endpoint code
#POST
public OrganizationResponse createOrganization(#Auth Person person, CreateOrganizationRequest request) {
String code = request.code();
String name = request.name();
String userId = person.user().id();
return dtoConverter.from(organizationService.createOrganization(userId, name, code));
}
As for question 2 and 3, transaction isolation levels are your friends. Put isolation level high enough. I think 'repeatable read' is the suitable one in your case. Your checkXXX methods will detect if some other transaction commits entities with the same name or code and it's guaranteeed that the situations stays by the time 'create' method is executed. One more useful read regarding Spring and transaction isolation levels.

As per my understanding the best way to handle DB level transaction you must use Spring's Isolation trnsaction in effective way in the dao layer. Below is sample industry standard codde in your case...
public interface OrganizationService {
#Retryable(maxAttempts=3,value=DataAccessResourceFailureException.class,backoff=#Backoff(delay = 1000))
public boolean checkOrganizationWithNameExists(String name);
}
#Repository
#EnableRetry
public class OrganizationServiceImpl implements OrganizationService {
#Transactional(isolation = Isolation.READ_COMMITTED)
#Override
public boolean checkOrganizationWithNameExists(String name){
//your code
return true;
}
}
Please pinch me if I'm wrong in here

Separation of concern :
Jax-rs resource (endpoint) layer : just handle the request, invoke the service and wrap the potential exception in appropriate response code (just catch and wrap manually or use exception mapper).
Service / business layer : expose a transactional method for each unit of work, business error must be handled as checked exception, operational ones as unchecked (subclasses of RuntimeException).
Data access layer: just handle the data access stuff (i.e. get db context, executes query and eventually map the result).
I insist on one thing, the good place to have transaction boundaries is the place where your business methods are defined. A transaction scope must be a business unit of work.
Regarding the concurrency issue, there is 2 way to handle this kind of concurrency problem : pessimistic or optimistic locking.
Pessimistic :
Lock
do your stuff
Update
Release lock
Optimistic :
check version
do your stuff
update if version is same, fail otherwise
Pessimistic is an issue regarding scalability and performance, optimistic problem is that you sometimes end by sending an operating error to the end-user.
I would personally go with optimistic locking in your case, JOOQ support it

First off the DAO layer should not even know it's being fronted by a REST webservice. Be sure to separate responsibilities.
Keep the #Transactional on the DAO. If you are issuing only a single statement than you need to decide if you are OK with dirty reads. Basically, figure out what the lowest Isolation Level is for your application. Every method will start a new Transaction (unless called from another method that already had one started) and if any Exceptions are thrown it will rollback any calls. You can setup a custom ExceptionHandler in your Controller to handle SQLDataIntegrityExceptions (like you're "code" insert example).
Use an Aggregate Primary Key that covers (id, name, code, status) so you can have an org with the same name but one will be "CURRENT" and one will be "REMOVED"

How to create Junit for the code that uses envers?

I am to write a JUnit to check that version is being maintained or not(on an event). Here is what I did using JUnit:
#Test
Public void testAudit() {
try {
//create Dao code
dao.save(); //This will create entry in AUD- and REVINFO-tables perfectly
SomeObject obj = SomeHelper.getAuditData(dao));
/*Method to be tested which generates audit message using envers i.e(dao created)*/
//Some logic to check if output is as expected
}
catch(Exception e) {
Assert.fail();
}
finally {
dao.delete(); //delete the data saved by JUnit (Problem starts here )
}
}
Calling the delete for dao would cause
UnsupportedOperationException: Can't write to a readonly object
I use Ehcache for caching. I googled for the problem and came to know that it might be because of CacheConcurrencyStrategy wrongly set for domain object which I want to delete. I checked.
For domain object there was no CacheConcurrencyStrategy. But nested object had CacheConcurrencyStrategy set as READ_WRITE (This might be real culprit).
But I don't want to change existing domain and existing code. Is it any way to bypass CacheConcurrencyStrategy for JUnit? If not, is there any possible way out without changing the existing code?

The ENVERs data is written post the transactions commit, so your code will never access the audit record, because one does not exist yet. If you want to test ENVERs, you need to manage the transactions yourself. Here is an example;
#Before
public void setup() {
// Envers audit information is written via post-event listeners and therefore the transaction needs to be
// committed.
PlatformTransactionManager txMgr = applicationContext.getBean(PlatformTransactionManager.class);
TransactionStatus status = txMgr.getTransaction(new DefaultTransactionDefinition(TransactionDefinition.PROPAGATION_REQUIRES_NEW));
Account account = accountDAO.getByUsername(UPDATE);
if (account != null) {
accountDAO.delete(account);
}
account = createAccount();
account.setUsername(INITIAL);
accountDAO.update(account);
txMgr.commit(status);
status = txMgr.getTransaction(new DefaultTransactionDefinition(TransactionDefinition.PROPAGATION_REQUIRES_NEW));
account.setUsername(UPDATE);
accountDAO.update(account);
txMgr.commit(status);
}
Then in your test, you can query out the audit information anyway you want (raw SQL, via the AuditReader, etc).

Hibernate pattern for transaction reuse

I have a base method that I'm writing in order to not repeat the same hibernate session/transaction logic over and over. It's fairly simple, but there's a specific issue that I'm not sure can be solved with this approach.
Imagine that you have a User entity and a Permission entity. If a request is made to save a user along with its matching permissions, then I think that it would make sense to perform both operations in a single transaction, since being able to save only one of those entities could be considered data corruption. For example, failing to save the user's permissions would warrant a rollback on previously inserted user data.
I made the following method to allow generic hibernate operations that could work with the current transaction if it were necessary, although I now think that in its current form it won't work since calling session.beginTransaction(); will probably return a new transaction even if the previous hasn't been commited (is this the case?). Suppose that I changed it in order to have it return the current session and transaction if it was specified that there would be more operations for the current transaction, do you think it would work? Would it be advisable to do something like this, or would you recommend a change of approach? Thanks
protected <T> void baseOperation(Class<T> entityClass, List<T> instances, BaseHibernateDAO.Operations operation, boolean isLastOperation) throws Exception
{
Session session = null;
Transaction transaction = null;
boolean caughtException = false;
//get session from factory
session = HibernateSessionFactory.getSession();
try
{
//get current transaction
transaction = session.beginTransaction();
for (Object instance : instances) //perform operation on all instances
{
log.debug(String.format("Will perform %s operation on %s instance.", operation.name(), entityClass.getName()));
switch (operation) //perform requested operation
{
case SAVE:
session.save(instance);
break;
case UPDATE:
session.update(instance);
break;
case SAVEORUPDATE:
session.saveOrUpdate(instance);
break;
case DELETE:
session.saveOrUpdate(instance);
break;
}
log.debug(String.format("%s operation on %s instance was succesful.", operation.name(), entityClass.getName()));
}
session.flush(); //synchronize
if (isLastOperation) //if this is the last operation of the transaction
{
transaction.commit();
log.debug("Transaction commited succesfully.");
}
}
catch (Exception e) //error occurred
{
caughtException = true;
//roll-back if transaction exists
if (transaction != null)
{
transaction.rollback();
}
//log and re-throw
log.error("An error occurred during transaction operation.", e);
throw e;
}
finally //cleanup tasks
{
if (isLastOperation || caughtException) //close session if there are no more pending operations or if an error occurred
{
HibernateSessionFactory.closeSession();
}
}
}

"Advisable" would be to stop trying to rewrite code that's already been written, debugged, dragged through the mud, debugged more, and deployed thousands of times. I.e, the issues and considerations you're encountering have been encountered and overcome before, and the solutions are proven. Further, having been extensively used and improved, they require much less effort to use than what you're putting into your custom solution. Check out Spring's Hibernate support, especially "Implementing DAOs based on plain Hibernate 3 API" and "Declarative transaction demarcation". For further reading, there's a whole chapter on transaction management.
I have a sample project on github where you can see a very simple example of using Spring to manage Hibernate Sessions and transactions in the context of a webapp (using Spring MVC).
Update: For those who come along later, so they don't have to dig through the comments:
There are three general ways to use Spring's transaction handling: declaratively defining which methods are transactional with XML, declaratively annotating methods as #Transactional, or programmatically using TransactionTemplate.