How can I use a transaction manager (such as Bitronix, JBoss TS or Atomikos) in a Java SE (not Java EE or Spring) to support the following use case:
Let's assume we have the following class:
public class Dao {
public void updateDatabase(DB db) {
connet to db
run a sql
}
}
and we create a Java Runnable from that, like the following:
public class MyRunnable extends Runnable {
Dao dao;
DB db;
public MyRunnable(Dao dao, DB db) {
this.dao=dao;
this.db = db;
}
public run() throws Exception {
return dao.updateDatabase(db);
}
}
Now in our Service layer, we have another class:
public class Service {
public void updateDatabases() {
BEGIN TRANSACTION;
ExecutorService es = Executors.newFixedThreadPool(10);
ExecutorCompletionService ecs = new ExecutorCompletionService(es);
List<Future<T>> futures = new ArrayList<Future<T>>(n);
Dao dao = new Dao();
futures.add(ecs.submit(new MyRunnable(dao, new DB("db1")));
futures.add(ecs.submit(new MyRunnable(dao, new DB("db2")));
futures.add(ecs.submit(new MyRunnable(dao, new DB("db3")));
for (int i = 0; i < n; ++i) {
completionService.take().get();
}
END TRANSACTION;
}
}
And the client can be a Servlet or any other multi-threaded environment:
public MyServlet extend HttpServlet {
protected void service(final HttpServletRequest request, final HttpServletResponse response) throws IOException {
Service service = new Service();
service.updateDatabases();
}
}
What would be the correct code for BEGIN TRANSACTION and END TRANSACTION parts? Is this even feasible? If not, what needs to be changed? The requirements is to keep the updateDatabases() method concurrent (since it will be accessing multiple databases at the same time) and transactional.
Seems like this can be done using Atomikos using SubTxThread
//first start a tx
TransactionManager tm = ...
tm.begin();
Waiter waiter = new Waiter();
//the code that calls the first EIS; defined by you
SubTxCode code1 = ...
//the associated thread
SubTxThread thread1 = new SubTxThread ( waiter , code1 );
//the code that calls the second EIS; defined by you
SubTxCode code2 = ...
//the associated thread
SubTxThread thread2 = new SubTxThread ( waiter , code2 );
//start each thread
thread1.start();
thread2.start();
//wait for completion of all calls
waiter.waitForAll();
//check result
if ( waiter.getAbortCount() == 0 ) {
//no failures -> commit tx
tm.commit();
} else {
tm.rollback();
}
XA Specification mandates that all XA calls be executed in the same thread context. To elaborate on the reason for this its because the commit could be called before any of the transactional branches are even created in your threads.
if you are just interested in how to execute those three calls in a XA transaction in JBoss TS
First make sure your -ds.xml specifies your datasource as an <xa-datasource>
InitialContext ctx = new InitialContext(parms);
UserTransaction ut = (UserTransaction) ctx.lookup("java:comp/UserTransaction");
ut.begin();
//Some Transactional Code
ut.commit();
Keep in mind with the code above you would not be able to use the ExecutorService to parallelize the calls.
Side Note: I don't know a lot about it but JTS/OTS claims to allow multiple threads to share in a transaction. I think it does this by propagating transactional context similar to ws-coordination/ws-transaction and is supported by JBossTS. Could be a red herring, but if your not under a time crunch it might be worth researching.
How about you
BEGIN_TRANSATION: Connect to all 3 databases in your Service,
pass the Connection objects (instead of db object) to MyRunnable
END_TRANSACTION: invoke commit and close on all 3 connections in
your Service
Related
I am trying to change Quartz Sequential execution to Parallel Execution.
It is working fine, Performance wise, it is seems good but Spawned (created) threads are not destroyed.
It is Still in Runnable State; why and How can I fix that?
Please Guide me.
Code is here :
#Override
protected void executeInternal(JobExecutionContext context) throws JobExecutionException {
logger.error("Result Processing executed");
List<Object[]> lstOfExams = examService.getExamEntriesForProcessingResults();
String timeZone = messageService.getMessage("org.default_timezone", null, Locale.getDefault());
if(lstOfExams!=null&&!lstOfExams.isEmpty()){
ThreadPoolTaskExecutor threadPoolExecuter = new ThreadPoolTaskExecutor();
threadPoolExecuter.setCorePoolSize(lstOfExams.size());
threadPoolExecuter.setMaxPoolSize(lstOfExams.size()+1);
threadPoolExecuter.setBeanName("ThreadPoolTaskExecutor");
threadPoolExecuter.setQueueCapacity(100);
threadPoolExecuter.setThreadNamePrefix("ThreadForUpdateExamResult");
threadPoolExecuter.initialize();
for(Object[] obj : lstOfExams){
if(StringUtils.isNotBlank((String)obj[2]) ){
timeZone = obj[2].toString();
}
try {
Userexams userexams=examService.findUserExamById(Long.valueOf(obj[0].toString()));
if(userexams.getExamresult()==null){
UpdateUserExamDataThread task=new UpdateUserExamDataThread(obj,timeZone);
threadPoolExecuter.submit(task);
}
// testEvaluator.generateTestResultAsPerEvaluator(Long.valueOf(obj[0].toString()), obj[4].toString(), obj[3]==null?null:obj[3].toString(),timeZone ,obj[5].toString() ,obj[1].toString());
// logger.error("Percentage Marks:::::"+result.getPercentageCatScore());
} catch (Exception e) {
Log.error("Exception at ResultProcessingJob extends QuartzJobBean executeInternal(JobExecutionContext context) throws JobExecutionException",e);
continue;
}
}
threadPoolExecuter.shutdown();
}
}
UpdateUserExamDataThread .class
#Component
//#Scope(value="prototype", proxyMode=ScopedProxyMode.TARGET_CLASS)
//public class UpdateUserExamDataThread extends ThreadLocal<String> //implements Runnable {
public class UpdateUserExamDataThread implements Runnable {
private Logger log = Logger.getLogger(UpdateUserExamDataThread.class);
#Autowired
ExamService examService;
#Autowired
TestEvaluator testEvaluator;
private Object[] obj;
private String timeZone;
public UpdateUserExamDataThread(Object[] obj,String timeZone) {
super();
this.obj = obj;
this.timeZone = timeZone;
}
#Override
public void run() {
String threadName=String.valueOf(obj[0]);
log.info("UpdateUserExamDataThread Start For:::::"+threadName);
testEvaluator.generateTestResultAsPerEvaluator(Long.valueOf(obj[0].toString()), obj[4].toString(), obj[3]==null?null:obj[3].toString(),timeZone ,obj[5].toString() ,obj[1].toString());
//update examResult
log.info("UpdateUserExamDataThread End For:::::"+threadName);
}
}
TestEvaluatorImpl.java
#Override
#Transactional
public Examresult generateTestResultAsPerEvaluator(Long userExamId, String evaluatorType, String codingLanguage,String timeZoneFollowed ,String inctenceId ,String userId) {
dbSchema = messageService.getMessage("database.default_schema", null, Locale.getDefault());
try {
//Some Methods
return examResult;
}catch(Exception e){
log.erorr(e);
}
}
I can provide Thread Dump file if needed.
it seems you create a thread pool in the same size of exams which is not quite optimal.
// Core pool size is = number of exams
threadPoolExecuter.setCorePoolSize(lstOfExams.size());
// Max pool size is just 1 + exam size.
threadPoolExecuter.setMaxPoolSize(lstOfExams.size()+1);
You have to consider that:
- If you create a thread pool and started it as many threads as defined in core size started immediately.
The max pool size is only than effective when you submit more than core pool threads can process right now AND when the queue size is full (in this case 100). So that means a new thread will be only then created when the number of submitted tasks exceeded 100+exam size.
In your case I would set the core pool size 5 or 10 (it actually depends on the how many core your target CPU have and/or how IO bound the submitted tasks are).
The max pool size can be double of that but it doesn't effective until the queue is full.
To let the size of live threads decrease after the submitted work done you have to set 2 parameters.
setKeepAliveSeconds(int keepAliveSeconds) : Which let the threads shut down automatically if they are not used along the defined seconds (by default 60 seconds, which is optimal) BUT this is normally only used to shut down threads of non-core pool threads.
To shut down threads of core part after keepAliveSeconds you have to set setAllowCoreThreadTimeOut(boolean allowCoreThreadTimeOut) as true. Which is normally false to keep core pool alive as long as the application is running.
I hope it helps.
I suspect that one of your threads waits indefinitely for an IO request answer. For example, you try to connect to a remote host where you did not set connection timeout and the host does not answer. In this case, you can shutdown all executing tasks forcefully by running shutdownNow method of the underlying ExecutorService then you can analyze InterruptedIOException thrown by the offending threads.
Replace
threadPoolExecuter.shutdown();
with below so you can examine errors.
ExecutorService executorService = threadPoolExecuter.getThreadPoolExecutor();
executorService.shutdownNow();
This will send interrupt signal to all running threads.
The threads do not wait on IO from some remote server, because the executed method on the threads would be in some jdbc driver classes, but they are currently all in UpdateUserExamDataThread.run(), line 37.
Now the question is: what is the code at UpdateUserExamDataThread.java line 37 ?
Unfortunately, the UpdateUserExamDataThread.java given at the moment is incomplete and/or not the version really executed: the package declaration is missing and it ends at line 29.
I suspect the issue is simply that you are calling run() instead of execute() when spawning the task thread using submit(). There is probably some expectation when using submit that threads kill themselves when the task is finished rather than terminating at the end of the run method.
Just Needed to increase the priority of threads and create number of threads as per number of cores in processor.
protected void executeInternal(JobExecutionContext context) throws JobExecutionException {
logger.error("Result Processing executed");
List<Object[]> lstOfExams = examService.getExamEntriesForProcessingResults();
String timeZone = messageService.getMessage("org.default_timezone", null, Locale.getDefault());
int cores = Runtime.getRuntime().availableProcessors();
if(lstOfExams!=null&&!lstOfExams.isEmpty()){
ThreadPoolTaskExecutor threadPoolExecuter = new ThreadPoolTaskExecutor();
threadPoolExecuter.setCorePoolSize(cores);
// threadPoolExecuter.setMaxPoolSize(Integer.MAX_VALUE);
threadPoolExecuter.setBeanName("ThreadPoolTaskExecutor");
// threadPoolExecuter.setQueueCapacity(Integer.MAX_VALUE);
threadPoolExecuter.setQueueCapacity(lstOfExams.size()+10);
threadPoolExecuter.setThreadNamePrefix("ThreadForUpdateExamResult");
threadPoolExecuter.setWaitForTasksToCompleteOnShutdown(true);
threadPoolExecuter.setThreadPriority(10);
threadPoolExecuter.initialize();
for(Object[] obj : lstOfExams){
if(StringUtils.isNotBlank((String)obj[2]) ){
timeZone = obj[2].toString();
}
try {
Userexams userexam=examService.findUserExamById(Long.valueOf(obj[0].toString()));
if(userexam.getExamresult()==null){
UpdateUserExamDataThread task=new UpdateUserExamDataThread(obj,timeZone,testEvaluator);
// threadPoolExecuter.submit(task);
threadPoolExecuter.execute(task);
}
// testEvaluator.generateTestResultAsPerEvaluator(Long.valueOf(obj[0].toString()), obj[4].toString(), obj[3]==null?null:obj[3].toString(),timeZone ,obj[5].toString() ,obj[1].toString());
// logger.error("Percentage Marks:::::"+result.getPercentageCatScore());
} catch (Exception e) {
logger.error("Exception at ResultProcessingJob extends QuartzJobBean executeInternal(JobExecutionContext context) throws JobExecutionException",e);
continue;
}
}
threadPoolExecuter.shutdown();
}
}
I know I'm new to this spring stuff but I've been stuck on this all day. I don't much like asking questions but maybe I'll get an idea.
So here's my problem:
I'm trying to create a Queue for processing stuff on the back end. I did this by creating a static executorservice in a component class with helper methods to run them. it seems to work like i want, and when i wire in classes i can get into those classes, but it seems like when those are running they lose application context (or something this is just my guess).
I'm sure There are better ways to do this, but in the custom framework I am working in there are a number of features that will not work for me. I have no spring-config.xml, cannot use #Configuration
executor service component
#Component
public class FifoComponent {
public static ExecutorService executors = Executors.newSingleThreadExecutor();
private static Lock lock = new ReentrantLock(true);
public static void executeNewTestJob(int i) {
lock.lock();
OrderAllocationTestJob job = new OrderAllocationTestJob(i);
executors.execute(job);
lock.unlock();
}
}
Runnable component - note appdateutils has a method that calls a component that and works fine in my typical tomcat environment
#Component
public class OrderAllocationTestJob implements Runnable {
int i;
public OrderAllocationTestJob(int i) {
this.i = i;
}
#Override
public void run() {
try {
Thread.sleep(100);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.println("Asynchronous task " + i);
System.out.println(AppDateUtils.getCurrentTimeStamp());
}
}
call from a struts 2 action (test) i know I can call the appdateutils.gettime method from
for (int i = 0; i < 50; i++) {
FifoComponent.executeNewTestJob(i);
}
here's the exception i end up with for what it's worth
"Scope 'request' is not active for the current thread"
Exception in thread "pool-15-thread-50" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dateTimestampDao': Scope 'request' is not active for the current thread; consider defining a scoped proxy for this bean if you intend to refer to it from a singleton; nested exception is java.lang.IllegalStateException: No thread-bound request found: Are you referring to request attributes outside of an actual web request, or processing a request outside of the originally receiving thread? If you are actually operating within a web request and still receive this message, your code is probably running outside of DispatcherServlet/DispatcherPortlet: In this case, use RequestContextListener or RequestContextFilter to expose the current request.
"I'm sure There are better ways to do this"
Based on this, you'll need to create/lookup all request and session scoped components before call another thread. Actually, request injection is thread local and can't works in your scenario.
i think if u remove
Thread.sleep(100);
in OrderAllocationTestJob to
job.sleep(100);
in FifoComponent will fix your problem
I solved this solution by extending ConcurrentLinkedQueue for my runnables and keeping them in a manager I instantiated in the initialize method of a ServletContextListener. By overriding the offer() method of the ConcurrentLinkedQueue to continually poll until the queue was empty I was able synchronously process runnables.
Unfortunately this locks down the request thread until the runnable is done and I will have to have my users keep an eye on it and let me know if the pages end up running long, but at least in my test environment the process seems sub-second even when i hit it with 20 at a time so I'm OK for now.
I would still prefer an ExecutorService executed from my Tomcat container but outside the scope of the requests but unless someone can answer the question I'm just going to have to leave it for now
Are you looking something like that?
#Component
public class AsynchronousThread extends Thread {
public static final Logger LOGGER = LoggerFactory
.getLogger(AsynchronousThread.class);
#Autowired
private Writer writer;
private BlockingQueue<IndexContextDTO> blockingQueue = new LinkedBlockingQueue<IndexContextDTO>(
500);
/**
*
*/
public AsynchronousThread() {
super("AsynchronousThread");
}
#PostConstruct
public void init() {
Integer internalQueueSize = 100;
this.blockingQueue = new LinkedBlockingQueue<>(internalQueueSize);
this.start();
}
#Override
public void run() {
while (true) {
// Do stuff
}
}
public void putInQueue(IndexContextDTO message) {
try {
this.blockingQueue.put(message);
} catch (InterruptedException interruptedException) {
// This exception will be thrown in very rare case.
LOGGER.error("An error while putting message in the queue. "
+ message, interruptedException);
}
}
}
I need to develop a web service operation with CXF 3 hosted by Tomcat 7. Our model layer is Spring 3.
This operation calls 16 other web services hosted by distant servers. We need to wait all responses in order to construct the response of our own operation.
We currently call each distant operations sequentially. Of course, we have response time issue. I think we should parallelize our operation inner calls and synchronize the different responses.
What kind of multithreading implementation can be safe? What can we do to make it better?
I'd use Java's generic Futures and a Spring's #Async methods in a #Service.
In short, you call the services sequentially and get all results as Futures, and then you simply check whether all the futures have finished proccessing. You can also do some work with partial data if there is such possibility.
Here's a simple example on how to do it. A sample service from the link:
#Service
public class GitHubLookupService {
RestTemplate restTemplate = new RestTemplate();
#Async
public Future<User> findUser(String user) throws InterruptedException {
System.out.println("Looking up " + user);
User results = restTemplate.getForObject("https://api.github.com/users/" + user, User.class);
// Artificial delay of 1s for demonstration purposes
Thread.sleep(1000L);
return new AsyncResult<User>(results);
}
}
And a method using it:
#Override
public void run(String... args) throws Exception {
// Start the clock
long start = System.currentTimeMillis();
// Kick of multiple, asynchronous lookups
Future<User> page1 = gitHubLookupService.findUser("PivotalSoftware");
Future<User> page2 = gitHubLookupService.findUser("CloudFoundry");
Future<User> page3 = gitHubLookupService.findUser("Spring-Projects");
// Wait until they are all done
while (!(page1.isDone() && page2.isDone() && page3.isDone())) {
Thread.sleep(10); //10-millisecond pause between each check
}
// Print results, including elapsed time
System.out.println("Elapsed time: " + (System.currentTimeMillis() - start));
System.out.println(page1.get());
System.out.println(page2.get());
System.out.println(page3.get());
}
I would use a traditional approach using join() to wait for the threads to finish instead of polling (I don't like polling pattern too much).
Kind of this for a generic thread to replicate:
public class ThreadedWebServiceRetrieval extends Thread {
private List<ResultObject> resultList;
private GenericWebServiceStub stub;
public ThreadedWebServiceRetrieval (List<ResultObject> resultList, GenericWebServiceStub stub) {
this.resultList = resultList;
this.stub = stub;
}
public void run() {
resultList.add(stub.retrieveData());
}
}
And this for the parallel retrieval code:
// ... Controller/Service stuff
List<ResultObject> resultList = new LinkedList<>();//Diamond operator
List<Thread> webServiceList = new LinkedList<>();
webServiceList.add(new ThreadedWebServiceRetrieval(resultList, stub1));
//...
webServiceList.add(new ThreadedWebServiceRetrieval(resultList, stubN));
for (Thread thread : webServiceList) {
thread.start();
}
for (Thread thread : webServiceList) {
thread.join();
}
// resultList is fulfilled
Time of this approach should be +/- longest retrieval.
I made the code VERY generic (overall in the Thread implementation) but it's intentional to fit most cases.
Enjoy!
I am writing an integration test in JUnit for a Message Driven Pojo (MDP):
#JmsListener(destination = "jms/Queue", containerFactory = "cf")
public void processMessage(TextMessage message) throws JMSException {
repo.save(new Entity("ID"));
}
where repo is a spring-data repository
my unit test:
#Test
public void test() {
//sendMsg
sendJMSMessage();
//verify DB state
Entity e = repo.findOne("ID");
assertThat(e, is(notNullValue()) );
}
Now, the thing is that the processMessage() method is executed in a different thread than the test() method, so I figured out that I need to somehow wait for the processMessage() method to complete before verifying the state of the DB. The best solution I could find was based on CountDownLatch. so now the methods look like this:
#JmsListener(destination = "jms/Queue", containerFactory = "cf")
public void processMessage(TextMessage message) throws JMSException {
repo.save(new Entity("ID"));
latch.countDown();
}
and the test
#Test
public void test() {
//set the countdownlatch
CountDownLatch latch = new CountDownLatch(1);
JMSProcessor.setLatch(latch);
//sendMsg
sendJMSMessage();
try {
countDownLatch.await();
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
//verify DB state
Entity e = repo.findOne("ID");
assertThat(e, is(notNullValue()) );
}
So I was very proud of myself and then I run the test and it failed. The repo.findOne("ID") returned null. In the first reaction I set up a breakpoint at that line and proceed with debugging. During the debugging session the repo.findOne("ID") actually returned the entity inserted by the #JMSListenerlistener method.
After scratching my head for a while here's the current theory: Since the spring-data repository is accessed in two different threads, it gets two different instances of EntityManager and therefore the two threads are in a differen't transaction. Eventhough there's some sort of synchronization using the CountDownLatch, the transaction bound to the thread executing the #JMSListener annotated method has not committed yet when the JUnit #Test annotated method starts a new transaction and tries to retrieve the entity.
So my question is:
Is there a way for one thread to wait for the commit of the other.
Can two threads share one transaction in such a synchronized context (ie, the two threads would not access the EntityManager simultaneously)
Is my testing approach a nonsense and there is a better way of doing this
I have a method that is going to call a stored function. I want it to async'ly do its work. This is what I have, but it seems like the .doWork() is never started because when I call getDao.deleteAll(), the stored function does not run.
#Transactional
public void delete()
{
final Session session = (Session) entityManager.getDelegate();
ExecutorService executorService = Executors.newSingleThreadExecutor();
executorService.execute(new Runnable()
{
#Override
public void run()
{
LOGGER.warn("starting");
session.doWork(new Work()
{
#Override
public void execute(Connection connection) throws SQLException
{
try
{
CallableStatement purgeArchived = connection.prepareCall("{call deleteAll()}");
purgeArchived.execute();
}
catch (SQLException exception)
{
LOGGER.warn("Failed to purge archive points. Reason: " + exception);
}
}
});
LOGGER.warn("stopping");
}
});
executorService.shutdown();
}
I see the logger has logged "starting", but it never got to "stopping" why is this happening?
Be aware that #Transaction is moot when you have a separate thread as Transactions are typically thread bound.
You will need to get a new entityManager from the factory inside the run().
Also go for #Async which is much cleaner.
Again be aware of transactionality with #Async
#Async and #Transactional: not working
As a general rule of thumb if you want to make some work async - treat that as a single unit of work and a separate transaction.