I have Service.class with start() and asychronous() method :
public ResponseEntity<Object> start() throws APICommandConstructionException, APICommunicationException, APIReplyParseException,
APIErrorResponse, IOException {
List<Company> companiesList = dbHandler.retrieveCompaniesList();
Company company = null;
for (int i = 0; i < companiesList.size(); i++) {
asychronousMethod(companiesList, i, company);
}
return new ResponseEntity<Object>("Start method has Finished", HttpStatus.OK);
}
#Async("threadPoolTaskExecutor")
public void asychronousMethod(List<Company> companiesList, int i, Company company) throws APICommandConstructionException, APIReplyParseException, APICommunicationException, APIErrorResponse, IOException {
company = companiesList.get(i);
company = utils.websiteScrap(company);
companiesRepository.save(company);
}
Everything that is inside a loop doesn't run in parallel, but it starts second loop after first finished. Why is that? How to do it parallel?
In brief: you shouldn't call explicitly methods with Spring annotations.
More detailed:
Spring creates special proxies that on back-stage do 'magic' for you. So if you have async annotation, that means that (depending on compile- and runtime-configuration of Spring) there was some hidden part of code that is not executed when you invoke this.asynchronousMethod.
How to fix:
First of all method should match public CompletableFuture<Void> - such way you know when thread is complete.
Second instead of this you need resolve self-instance as Spring proxy.
The simplest way over #Autowired:
#Autowired
MyClass zhis;
.... //in for loop:
future = zhis.asychronousMethod(companiesList, i, company);
P.s please see good example at https://spring.io/guides/gs/async-method/
Related
I'm having a problem when I trying to delete data from the DB using multiple threads with Hibernate.
Repo:
#Modifying
#Query("DELETE FROM Customer cus WHERE cus.customerId in :customerIds")
public void deleteByCustomerIds(#Param("customerIds") List<Long> customerIds);
Service:
public runDelete (List<Long> customerIds) {
List<List<Long>> partitions = Lists.partition(customerIds, 5000);
for(int i = 0; i < partitions.size(); i++ ) {
final int index = i;
Runnable thread = () -> deleteCustomersInBatches(partitions.get(index));
new Thread(thread).start();
}
}
#Transactional(propagation = Propagation.REQUIRES_NEW)
private void deleteCustomerInBatches(List<Long> customerIds) {
for (List<Long> batch : Lists.partition(oldCalcIds, 1000)) {
customerRepo.deleteByCustomerIds(batch);
}
}
This is how code looks like, I have the #Transactional tag on the service layer where the repo call is being made.
at java.lang.Thread.run(Thread.java:748) Caused by:
javax.persistence.TransactionRequiredException: Executing an
update/delete query
at org.hibernate.jpa.spi.AbstractQueryImpl.executeUpdate(AbstractQueryImpl.java:54)
I keep getting this error. Any help is appreciated.
It's because you're invoking #Transactional method from within same bean.
#Transactional only works on methods invoked on proxies created by spring. It means, that when you create a #Service or other bean, method called from the outside will be transactional. If invoked from within bean, nothing will happen, as it doesn't pass through proxy object.
The easiest solution would be to move the method to another bean. If you really want to keep it within same component, then you need to invoke it, so that it gets wrapped in proxy by spring AOP. You can do this like that:
private YourClass self;
#Autowired
private ApplicationContext applicationContext;
#PostConstruct
public void postContruct(){
self = applicationContext.getBean(YourClass.class);
}
Then invoking method on self would result in opening a transaction.
The title might be incorrect, but I will try to explain my issue. My project is a Spring Boot project. I have services which do calls to external REST endpoints.
I have a service method which contains several method calls to other services I have. Every individual method call can be successful or not. Every method call is done to a REST endpoint and there can be issues that for example the webservice is not available or that it throws an unknown exception in rare cases. What ever happens, I need to be able to track which method calls were successful and if any one of them fails, I want to rollback to the original state as if nothing happened, see it a bit as #Transactional annotation. All REST calls are different endpoints and need to be called separately and are from an external party which I don't have influence on. Example:
public MyServiceImpl implements MyService {
#Autowired
private Process1Service;
#Autowired
private Process2Service;
#Autowired
private Process3Service;
#Autowired
private Process4Service;
public void bundledProcess() {
process1Service.createFileRESTcall();
process2Service.addFilePermissionsRESTcall();
process3Service.addFileMetadataRESTcall(); <-- might fail for example
process4Service.addFileTimestampRESTcall();
}
}
If for example process3Service.addFileMetadataRESTcall fails I want to do something like undo (in reverse order) for every step before process3:
process2Service.removeFilePermissionsRESTcall();
process1Service.deleteFileRESTcall();
I read about the Command pattern, but that seems to be used for Undo actions inside an application as a sort of history of actions performed, not inside a Spring web application. Is this correct for my use case too or should I track per method/webservice call if it was successful? Is there a best practice for doing this?
I guess however I track it, I need to know which method call failed and from there on perform my 'undo' method REST calls. Although in theory even these calls might also fail of course.
My main goal is to not have files being created (in my example) which any further processes have not been performed on. It should either be all successful or nothing. A sort of transactional.
Update1: improved pseudo implementation based on comments:
public Process1ServiceImpl implements Process1Service {
public void createFileRESTcall() throws MyException {
// Call an external REST api, pseudo code:
if (REST-call fails) {
throw new MyException("External REST api failed");
}
}
}
public class BundledProcessEvent {
private boolean createFileSuccess;
private boolean addFilePermissionsSuccess;
private boolean addFileMetadataSuccess;
private boolean addFileTimestampSuccess;
// Getters and setters
}
public MyServiceImpl implements MyService {
#Autowired
private Process1Service;
#Autowired
private Process2Service;
#Autowired
private Process3Service;
#Autowired
private Process4Service;
#Autowired
private ApplicationEventPublisher applicationEventPublisher;
#Transactional(rollbackOn = MyException.class)
public void bundledProcess() {
BundleProcessEvent bundleProcessEvent = new BundleProcessEvent();
this.applicationEventPublisher.publishEvent(bundleProcessEvent);
bundleProcessEvent.setCreateFileSuccess = bundprocess1Service.createFileRESTcall();
bundleProcessEvent.setAddFilePermissionsSuccess = process2Service.addFilePermissionsRESTcall();
bundleProcessEvent.setAddFileMetadataSuccess = process3Service.addFileMetadataRESTcall();
bundleProcessEvent.setAddFileTimestampSuccess = process4Service.addFileTimestampRESTcall();
}
#TransactionalEventListener(phase = TransactionPhase.AFTER_ROLLBACK)
public void rollback(BundleProcessEvent bundleProcessEvent) {
// If the last process event is successful, we should not
// be in this rollback method even
//if (bundleProcessEvent.isAddFileTimestampSuccess()) {
// remove timestamp
//}
if (bundleProcessEvent.isAddFileMetadataSuccess()) {
// remove metadata
}
if (bundleProcessEvent.isAddFilePermissionsSuccess()) {
// remove file permissions
}
if (bundleProcessEvent.isCreateFileSuccess()) {
// remove file
}
}
Your operation looks like a transaction, so you can use #Transactional annotation. From your code I can't really tell how you are managing HTTP response calls for each of those operations, but you should consider having your service methods to return them, and then do a rollback depending on response calls. You can create an array of methods like so, but how exactly you want your logic to be is up to you.
private Process[] restCalls = new Process[] {
new Process() { public void call() { process1Service.createFileRESTcall(); } },
new Process() { public void call() { process2Service.addFilePermissionsRESTcall(); } },
new Process() { public void call() { process3Service.addFileMetadataRESTcall(); } },
new Process() { public void call() { process4Service.addFileTimestampRESTcall(); } },
};
interface Process {
void call();
}
#Transactional(rollbackOn = Exception.class)
public void bundledProcess() {
restCalls[0].call();
... // say, see which process returned wrong response code
}
#TransactionalEventListener(phase = TransactionPhase.AFTER_ROLLBACK)
public void rollback() {
// handle rollback according to failed method index
}
Check this article. Might come in handy.
The answer to this question is quite broad. There are various ways to do distributed transactions to go through them all here. However, since you are using Java and Spring, your best bet is to use something like JTA (Java Transaction API), which enables a distributed transactions across multiple services/instances/etc.. Fortunately, Spring Boot supports JTA using either Atomikos or Bitronix. You can read the doc here.
One approach to enable distributed transactions is through a message broker such as JMS, RabbitMQ, Kafka, ActiveMQ, etc. and use a protocol like XA transactions (two-phase commit). In the case of external services that do not support distributed, one approach is to write a wrapper service that understands XA transactions to that external service.
This question already has answers here:
How to flush data into db inside active spring transaction?
(4 answers)
Closed 7 years ago.
I'm using Spring / Spring-data-JPA and find myself needing to manually force a commit in a unit test. My use case is that I am doing a multi-threaded test in which I have to use data that is persisted before the threads are spawned.
Unfortunately, given that the test is running in a #Transactional transaction, even a flush does not make it accessible to the spawned threads.
#Transactional
public void testAddAttachment() throws Exception{
final Contract c1 = contractDOD.getNewTransientContract(15);
contractRepository.save(c1);
// Need to commit the saveContract here, but don't know how!
em.getTransaction().commit();
List<Thread> threads = new ArrayList<>();
for( int i = 0; i < 5; i++){
final int threadNumber = i;
Thread t = new Thread( new Runnable() {
#Override
#Transactional
public void run() {
try {
// do stuff here with c1
// sleep to ensure that the thread is not finished before another thread catches up
Thread.sleep(1000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
});
threads.add(t);
t.start();
}
// have to wait for all threads to complete
for( Thread t : threads )
t.join();
// Need to validate test results. Need to be within a transaction here
Contract c2 = contractRepository.findOne(c1.getId());
}
I've tried using the entity manager to, but get an error message when I do:
org.springframework.dao.InvalidDataAccessApiUsageException: Not allowed to create transaction on shared EntityManager - use Spring transactions or EJB CMT instead; nested exception is java.lang.IllegalStateException: Not allowed to create transaction on shared EntityManager - use Spring transactions or EJB CMT instead
at org.springframework.orm.jpa.EntityManagerFactoryUtils.convertJpaAccessExceptionIfPossible(EntityManagerFactoryUtils.java:293)
at org.springframework.orm.jpa.aspectj.JpaExceptionTranslatorAspect.ajc$afterThrowing$org_springframework_orm_jpa_aspectj_JpaExceptionTranslatorAspect$1$18a1ac9(JpaExceptionTranslatorAspect.aj:33)
Is there any way to commit the transaction and continue it? I have been unable to find any method that allows me to call a commit().
I had a similar use case during testing hibernate event listeners which are only called on commit.
The solution was to wrap the code to be persistent into another method annotated with REQUIRES_NEW. (In another class) This way a new transaction is spawned and a flush/commit is issued once the method returns.
Keep in mind that this might influence all the other tests! So write them accordingly or you need to ensure that you can clean up after the test ran.
Why don't you use spring's TransactionTemplate to programmatically control transactions? You could also restructure your code so that each "transaction block" has it's own #Transactional method, but given that it's a test I would opt for programmatic control of your transactions.
Also note that the #Transactional annotation on your runnable won't work (unless you are using aspectj) as the runnables aren't managed by spring!
#RunWith(SpringJUnit4ClassRunner.class)
//other spring-test annotations; as your database context is dirty due to the committed transaction you might want to consider using #DirtiesContext
public class TransactionTemplateTest {
#Autowired
PlatformTransactionManager platformTransactionManager;
TransactionTemplate transactionTemplate;
#Before
public void setUp() throws Exception {
transactionTemplate = new TransactionTemplate(platformTransactionManager);
}
#Test //note that there is no #Transactional configured for the method
public void test() throws InterruptedException {
final Contract c1 = transactionTemplate.execute(new TransactionCallback<Contract>() {
#Override
public Contract doInTransaction(TransactionStatus status) {
Contract c = contractDOD.getNewTransientContract(15);
contractRepository.save(c);
return c;
}
});
ExecutorService executorService = Executors.newFixedThreadPool(5);
for (int i = 0; i < 5; ++i) {
executorService.execute(new Runnable() {
#Override //note that there is no #Transactional configured for the method
public void run() {
transactionTemplate.execute(new TransactionCallback<Object>() {
#Override
public Object doInTransaction(TransactionStatus status) {
// do whatever you want to do with c1
return null;
}
});
}
});
}
executorService.shutdown();
executorService.awaitTermination(10, TimeUnit.SECONDS);
transactionTemplate.execute(new TransactionCallback<Object>() {
#Override
public Object doInTransaction(TransactionStatus status) {
// validate test results in transaction
return null;
}
});
}
}
I know that due to this ugly anonymous inner class usage of TransactionTemplate doesn't look nice, but when for some reason we want to have a test method transactional IMHO it is the most flexible option.
In some cases (it depends on the application type) the best way to use transactions in Spring tests is a turned-off #Transactional on the test methods. Why? Because #Transactional may leads to many false-positive tests. You may look at this sample article to find out details. In such cases TransactionTemplate can be perfect for controlling transaction boundries when we want that control.
I have a Java class which is something like the following:
public class Foo{
public void doSomething(){
StageA a = new StageA();
StageB b = new StageB();
StageC c = new StageC();
a.execute();
b.execute();
c.execute();
}
}
Now, assuming I can't really edit this class itself, could I still use spring AOP to apply logging around the execute methods? (presumably without using aspect4j)
Well you can log method and it's required time(for performance) but I don't think you would be able to log what method is doing.
From Spring Docs :
Around advice: Advice that surrounds a join point such as a method invocation. This is the most powerful kind of advice. Around advice can perform custom behavior before and after the method invocation. It is also responsible for choosing whether to proceed to the join point or to shortcut the advised method execution by returning its own return value or throwing an exception.
If you are using log4j loggers internally in your methods you can log what method is doing by configuring log4j.
(presumably without using aspect4j)
--> Spring internally uses aspectJ
Check here for reference and example
EDITED:
I don't think that it is possible to log execution of the each "execute" method in your case without changing Foo or Stage classes. Because Stage... classes are not managed by container. You can only log when your doSomething method will start execution (if Foo class is managed by Spring container), you cannot control it's execution flow.
If your classes are to be managed by Spring container, then you can easy do this. You should simply write Spring AOP "around" aspect for Stage... classes, not for Foo class.
Here is an example of simple logging aspect:
#Component
#Aspect
#Order(value=2)
public class LoggingAspect {
#Around("execution(* com.blablabla.server..*.*(..))")
public Object logMethod(ProceedingJoinPoint joinPoint) throws Throwable{
final Logger logger = LoggerFactory.getLogger(joinPoint.getTarget().getClass().getName());
Object retVal = null;
try {
StringBuffer startMessageStringBuffer = new StringBuffer();
startMessageStringBuffer.append("Start method ");
startMessageStringBuffer.append(joinPoint.getSignature().getName());
startMessageStringBuffer.append("(");
Object[] args = joinPoint.getArgs();
for (int i = 0; i < args.length; i++) {
startMessageStringBuffer.append(args[i]).append(",");
}
if (args.length > 0) {
startMessageStringBuffer.deleteCharAt(startMessageStringBuffer.length() - 1);
}
startMessageStringBuffer.append(")");
logger.trace(startMessageStringBuffer.toString());
StopWatch stopWatch = new StopWatch();
stopWatch.start();
retVal = joinPoint.proceed();
stopWatch.stop();
StringBuffer endMessageStringBuffer = new StringBuffer();
endMessageStringBuffer.append("Finish method ");
endMessageStringBuffer.append(joinPoint.getSignature().getName());
endMessageStringBuffer.append("(..); execution time: ");
endMessageStringBuffer.append(stopWatch.getTotalTimeMillis());
endMessageStringBuffer.append(" ms;");
logger.trace(endMessageStringBuffer.toString());
} catch (Throwable ex) {
StringBuffer errorMessageStringBuffer = new StringBuffer();
// Create error message
logger.error(errorMessageStringBuffer.toString(), e)
throw ex;
}
return retVal;
}
}
Yes, you can write an #Around advice with execution pointcut targeting methods whose name is execute() as follows:
#Around("execution(* execute(..))")
public Object execute(ProceedingJoinPoint pjp) throws Throwable
{
// Log statements before the call;
Object obj = pjp.proceed();
// Log statements after the call;
return obj;
}
Below is a simplified setup of my application. It has a class Foobar which calls on a facade method for fetching data. The facade then calls on a web service to actually get the data and then manipulates the data a bit and then returns it to Foobar.
Now because the web service might take a good while to run, the method call to the facade needs to be asynchronous. Hence the facade's method doesn't have a return value, but instead, the method uses a callback object. Look at the example and continue reading below.
public class Foobar {
private List<DTO> dtos;
#Autowired
private Facade facade;
public void refresh() {
facade.refreshFoobar(new CallBack() {
public void dataFetched(List<DTO> dtos) {
setDtos(dtos);
}
});
}
public void setDtos(List<DTO> dtos) {
this.dtos = dtos;
}
}
public class Facade {
...
public void refreshFoorbar(CallBack cb) {
// Fetch data from a web service
List<DTO> dtos = webService.getData();
// Manipulate DTOs
....
// call on the callback method
cb.dataFecthed(dtos);
}
}
I have two ways of making the facade's method asynchronous, either by creating a thread manually or by using springs #Async annotation.
public class Facade {
public void refreshFoorbar(CallBack cb) {
new Thread() {
#Override
public void run() {
....
}
}.start();
}
}
// ... OR ...
public class Facade {
#Async
public void refreshFoorbar(CallBack cb) {
....
}
}
My problem is that I now need to write an integration test for this chain of method calls. I think I need to force the async facade call to be synchronous when the integration test is ran, otherwise I won't know for sure when I can do the appropriate asserts. The only idea for making the method call synchronous is to use manually handled threads AND making the threading conditional (so, for testing purposes, I have an if clause which determines if the facade method should be ran in a separate thread or not).
However, I have a feeling that there could be a better solution to my problem, whether it be a better way of forcing the method to me synchronous, eg with spring, or by testing the multithreading on some way.
This is where I need your suggestions, how would you solve my problem? Note, I'm using junit for both unit and integration tests.
Simple solution would be to return a Future object like this,
#Async
public Future<String> refreshFoorbar(CallBack cb) {
yourHeavyLifting(); //asynchronous call
return new AsyncResult<String>("yourJobNameMaybe");
}
And in your test, take the future reference and call the get() method.
future.get(); // if its not already complete, waits for it to complete
assertTrue(yourTestCondition)
This blog post shows a sample.
When JUnit testing stuff like this, I use a testing callback with a CountDownLatch that gets counted down by the callback and await()ed by the test method.
private static class TestingCallback implements Callback {
private final CountDownLatch latch;
public TestingCallback(CountDownLatch latch) {
this.latch = latch;
}
#Override public void onEvent() {
this.latch.countDown();
}
}
#Test
public void testCallback() {
final CountDownLatch latch = new CountDownLatch(1);
classUnderTest.execute( new TestCallback(latch) );
assertTrue(latch.await(30, TimeUnit.SECONDS));
}
If the callback is invoked (asynchronously) by the code under test, the latch returns true and the test passes. If the callback doesn't get invoked, the test times out after thirty seconds and the assertion fails.