I have my service layer 'S' calling data layer 'D_OLD'. Now I am upgrading my D_OLD database version to D_NEW. But I want to test stability for a few days and if not stable, I want to rollback. But if I rollback, I must be able to replay all the transactions happened on new database to old database.
What I am planning is to have two instances of D, one with new database and other running with old database.
I am planning to intercept all the PUT/POST/DELETE requests from S to D_NEW and put it in a kafka topic. I will have a job to replay that requests into D_OLD.
Now my S is a spring application using ribbon for making requests to D_NEW. I have tried ClientHttpRequestInterceptor, but seems it work only with RestTemplate.
Is there any way to intercept outgoing requests in ribbon/spring/tomcat?
Related
I have a requirement to update a database using events in Springboot microservice. Each microservice has its own persistent layer. Microservices are communicating with each other using REST API's.
Scenario:
I have two microservices - Vendor microservice with vendor DB and Order microservice with order DB. When a request is received by the vendor microservice, it will update the vendor Db and also add an order in the order DB and all this should be done in one transaction.
I cannot use a REST API for calling the vendor service to update the order. If any transaction fails, everything should be rolled back. How can I achieve this using events or something similar?
You can use a message queue like Kafka or RabbitMQ. Because you faced an issue known as two-phase commit. You can use transactional outbox pattern that is widely used in projects which consist of microservices.
Spring already provides a consistent programming model for transactions. You can use
#Transactional
to achieve this.
You can read more about transactional annotation in the official documentation
https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/transaction/annotation/Transactional.html
We have an application behind a load balancer that only supports round-robin, no sticky sessions.
The Spring Boot application runs in an OpenShift cluster, with a number of pods in a deployment. We have at least two pods, but might scale this up to 10 or 20, depending on load.
The application uses Spring Session and Spring Security. Spring Session is configured to use MongoDB (not Redis), since the application is already using MongoDB for other purposes.
During functional testing with low to moderate load, we have noticed issues with session attributes "going missing": the code that updates these entries runs successfully, but after the request is finished, the older contents of the attributes is in the session. This happens randomly.
Testing with a single instance of the application, no such observations were made.
To me, this smells like a race condition between the write back of the session object to Mongo, with some HTTP request on one pod racing the write back in another pod, and the "wrong one" winning.
Is this a valid usage scenario for Spring Session with MongoDB? In other words, is this supposed to work?
If it is supposed to work, how can I find out what's happening, and what can I do to solve the issue?
If it's not supposed to work, is there a Spring Session setup that would allow a cross application server sharing of session state without race conditions?
We spent a lot of time with the DB team trying to figure out if this had anything to do with the MongoDB client connection or server configuration, but after some more thorough research, I've found the culprit: it's spring-session-data-mongodb, because it fails to implement delta updates.
https://github.com/spring-projects/spring-session-data-mongodb/issues/106
The problem is that there is no logic to check if a write to the session repository is necessary, or any tracking which attributes of the session have been changed. The session is written back at the end of every request unconditionally.
If you have multiple concurrent requests, like any normal web application has, the session state survives from the request that was started last and finished first. So a simple image retrieval (that is handled through a Spring handler) will cause the session to be written back. If you have, like we do, a login handler that takes a substantial amount of time (up to 2 seconds) because it retrieves a bunch of user info from a backend system, the request for the image will have started after the login request, but will have finished before it. spring-session-data-mongodb then decides that the session state from the login handler is stale, and fails to save it.
So until that bug is fixed, we will need to use a different repository like Redis.
The current situation
Got a project using: spring-boot, spring-cloud, postgresql, as a microservice system.
There are 2 services, say SA and SB, they operate on 2 RDBMS databases respectively, say DA and DB.
Now, there is an operation contains 2 sub steps:
Http client would make a request to service SA, to save a record RA, into DA.
Then, SA send a request to service SB, to save a record RB, into DB.
As a whole, the 2 sub steps should either both commit, or both rollback.
Analysis
If move both operations into a single service, then could use Spring's distributed transaction to sovled it with JTA (based on 2PC protocol).
But here, the 2 operations are in 2 services, and they communicated via http REST protocol. Maybe could use mq + compensation to solve this, but I am not sure is there a better approach.
The questions are
In this case, does JTA (based on 2PC protocol) still work?
If not, what is the preferred solution?
Possible solutions I can guess:
Refactor code to move the 2 operations into a single service.
Implementat the mq + compensation architecture to support this.
Maybe this project is helpful for you https://github.com/apache/servicecomb-pack
Apache ServiceComb Pack is an eventually data consistency solution for micro-service applications. ServiceComb Pack currently provides TCC and Saga distributed transaction co-ordination solutions by using Alpha as a transaction coordinator and Omega as an transaction agent
My query is, how to invoke a method in spring boot project when a table is being updated in MySQL DB.
Case 1: Let's suppose I have a Mysql Db that is shared with two application and any particular table is updated frequently.
So, my issue is how can I notify another application (Spring-boot) to be aware of that update so that it can process that data.
Note I don't want to use the scheduler.
Thanks,
There could be many solutions depending on which technology you use. If you are using Spring-data, then you can use events and from within eventhandler notify other application (you can use rest integration, amqp, ...)
This question has nothing to do with Springboot or any other application.
Question here is how should two applications(web or non web) communicate with each other.
With respect to communication, it depends on the requirement, whether you need Synchronous communication or Asynchronous communication.
If its Synchronous communication, you may use Rest Template or any other, but it blocks the current request, if other application is running as service.
If its Asynchronous communication(where you don't want to wait for result and client is not blocked), you use Message Brokers like Kafka, ActiveMq
I think you can either make async call feature of spring boot application, but consider what should you if the request fails?
I was asked the following question in an interview and couldn't answer that.
How do you include a jdbc operation,a web service call and a JMS operation into one single transaction. That means if one of them fails all has to be roll backed.
I have heard about two-phase commit protocol and oracl XA in case of database transactions involving multiple databases. But not sure whether the same can be used here.
The critical factor is that the web services you connect to have been built using a web services framework that supports transactions. JBoss Narayana is one such web services framework. Once the web services endpoint you are connecting to is on such a framework, it's just a matter of configuring spring to use the appropriate client.
In the case of Narayana, the spring config (from http://bgshinhung.blogspot.ca/2012/10/integrating-spring-framework-jetty-and.html) for transactions with web services:
You are never going to be able to do this in a completely bomb-proof way as the systems are separate. A failure in one stage of the system (for example between the SQL commit and the JMS commit the power on your server gets turned off) will leave the SQL commit in place.
The only way to resolve that would be to keep some record of partial commits somewhere and scan that on startup to fix any resulting problems but now what happens if you have a failure processing or keeping that list.
Essentially the solution is to do your own implementation of the multiple-stage-commit and rollback process wrapping the three operations you need to make. If any of the operations fails then you need to reverse (preferably using an internal transaction mechanism, if not then by issuing reversing commands) any that have been done so far.
There are a lot of corner cases and potential ways for a system like this to fail though, so really the first approach should be to consider whether you can redesign the system so you don't need to do this at all!
It may be trick question and the right answer is "it can not be done".
But I would try to pseodo-code something like this:
try{
jdbc.startTransaction();
Savepoint saveJdbc = jdbc.setSavepoint();
JMS.startTransaction();
Savepoint saveJMS = JMS.setSavepoint();
jdbs.doSomeStuff();
JMS.doSomeStuff();
jdbc.commit();
JMS.commit();
if(webServise.doSomeStuff() == fail){throw new Exception();}
}
catch(Eception e){
jdbc.rollback(saveJdbc);
JMS.rollback(saveJMS);
}
You prepare one servise that has roll back. You prepare second servise that has roll back. You will try web servise and if web servise fail you will roll back those two which have rollback.
May be it is a way to implement rollback to your web servise.
We had same situation like web service will push the data, we have to read the xml stream and persist to db(oracle). implementation we followed is.
Web service send soap message and that will contain xml stream data.
all request soap messages pushed to jms.
respective listner will read the stream and persist the data into 'Temporary tables'.
if request processed successfully then move data from temp table to actual table.
if any error roll back.
hope above points may help.
To my mind, it looks like interviewer liked to understand your ability to think in terms of enterprise wide distribution. Few points:
JDBC is used for Database connectivity
WebService is probably a mechanism to send control command to a
server from any client.
JMS is mainly used for alerts of what is being happened in the
system.
My guess is your interviewer might be having a typical scenario with him that they wish to suffice the following situation:
Data is on one tier ( cluster, or machine )
Clients may be any kind, mobile, app, ios, objective c, browser pay, etc.
JMS is configured to listen to topics. Or is that he wishes he could do that.
Now probably the best approach is to write a JMS Subscriber which decides what to do in the onMessage() method. As an example, suppose a web service is initiated a payment request from client. This will initiate a JMS publisher to tell a DAO do the necessary internal connection to database and when transaction is in middle and when it finishes, one message will be published to subscriber. You will have full grain control of every step as that would be configured to be published through JMS. Though this is difficult to achieve, this could be your interviewer's expected approach from you. (This is Only my guess, and please note.)