I have the following situation. I have a job that:
May time out after a given amount of time, and if so occurs needs to throw an exception
If it does not time out, will return a result
If this job returns a result, it must be returned as quickly as possible, because performance is very much an issue. Asynchronous solutions are hence off the table, and naturally tying up the system by hammering isn't an option either.
Lastly, the system has to conform to the EJB standard, so AFAIK using ordinary threads is not an option, as this is strictly forbidden.
Our current solution uses a thread that will throw an exception after having existed for a certain amount of time without being interrupted by an external process, but as this clearly breaks the EJB standard, we're trying to solve it with some other means.
Any ideas?
Edited to add: Naturally, a job which has timed out needs to be removed (or interrupted) as well.
Edited to add 2:
This issue doesn't seem to have any solution, because detecting a deadlock seems to be mostly impossible sticking to pure EJB3 standards. Since Enno Shioji's comments below reflect this, I'm setting his suggestion as the correct answer.
This is more like a request for clarification, but it's too long to fit as a comment..
I'm not sure how you are doing it right now, since from what you wrote, just using the request processing thread seems to be the way to go. Like this:
//Some webservice method (synchronous)
public Result process(Blah blah){
try{
return getResult(TimeUnit.SECONDS, 10);
}catch(InterruptedException e){
//No result within 10 seconds!
throw new ServiceUnavailableException("blah");
}
}
I'm not sure why you are creating threads at all. If you are forced to use threads because the getResult method doesn't timeout at all, you would have a thread leak. If it timeouts after a longer time and thus you want to "shortcut" your reply to the user, that would be the only case I'd consider using a thread like I imagine how you are using it. This could result in Threads piling up under load and I'd strive to avoid such situation.
Maybe you can post some code and let us know why you are creating in your service at all?
Also, what's your client interface? Sounds like it's a synchronous webservice or something?
In that case, if I were you I would use a HashedWheelTimer as a singleton... this mechanism should work great with your requirement (here is an implementation). However, this unfortunately seem to conflict with the ban on threading AND the ban on singleton in the EJB spec. In reality though there really isn't a problem if you would do this. See this discussion for example. We have also used the singleton pattern in our EJB app. which used JBoss. However, if this isn't a viable choice then I might look at isolating the processing in its own JVM by defining a new web service (and deploy it in a web-container or something), and call that service from the EJB app. This would however obviously incur performance hit and now you would have another whole new app.
With Bean Managed Transaction, the timeout for the specific transaction can be specified by using UserTransaction interface.
Modify the timeout value that is
associated with transactions started
by the current thread with the begin
method.
void setTransactionTimeout(int seconds) throws SystemException
Transaction will timeout after specified seconds & may not get propagated further. If exception is not thrown implicitly, then can throw it explicitly based on the result.
Will return a result on successful completion within specified time.
Can use it with stateless session beans so there may not be a performance issue.
Its EJB standard so that will not be an issue to implement.
With little-bit work around, it should work fine in the given scenario.
Edit : Also can use server specific properties to manage transaction timeout.
JBoss : At either at class or method level annotation #TransactionTimeout(100) can be applied.
Weblogic : Specifying the parameters in weblogic-ejb-jar.xml
<transaction-descriptor>
<trans-timeout-seconds>100</trans-timeout-seconds>
</transaction-descriptor>
GlassFish : Using the optional cmt-timeout-in-seconds element in sun-ejb-jar.xml
Stick the process and it's timeout thread in to a class annotated with #WebService, put that class in to a WAR, then invoke the WebService from your EJB.
WARs don't have the same limitations or live under the same contract that EJBs do, so they can safely run threads.
Yes, I consider this a "hack", but it meets the letter of the requirements, and it's portable.
You can create threads using the commonj WorkManager. There are implementations built into WebSphere and Weblogic as they proposed the standard, but you can also find implementations for other appservers as well.
Basically, the WorkManager allows you to create managed threads inside the container, much like using an Executor in regular Java. Your only other alternative would be to use MDB's, but that would be a 'heavier' solution.
Since I don't know your actual platform, you will have to google commonj with your platform yourself 8-)
Here is a non IBM or Oracle solution.
Note: This is not an actual standard, but it is widely available for different platforms and should suit your purposes nicely.
For EJBs, there is a concept of "Container Managed Transactions". By specifying #TransactionAttribute on your bean, or specific method, the container will create a transaction when ever the method(s) are invoked. If the execution of the code takes longer than the transaction threshold, the container will throw an exception. If the call finishes under the transaction threshold, it will return as usual. You can catch the exception in your calling code and handle it appropriately.
For more on container managed transactions, check out: http://java.sun.com/j2ee/tutorial/1_3-fcs/doc/Transaction3.html and http://download.oracle.com/javaee/5/tutorial/doc/bncij.html
You could use #TimeOut. Something like:
#Stateless
public class TimedBean {
#Resource
private TimerService timerService;
static private AtomicInteger counter = new AtomicInteger(0);
static private Map<Integer, AtomicBoolean> canIRunStore = new ...;
public void doSomething() {
Integer myId = counter.getAndIncrement();
AtomicBoolean canIRun = new AtomicBoolean(true);
canIRunStore.put(myId, canIRun);
timerService.createTimer(1000, 0, myId);
while (canIRun.get() /* && some other condition */) {
// do my work ... untill timeout ...
}
}
#Timeout
#PermitAll
public void timeout(Timer timer) {
Integer expiredId = (Integer) timer.getInfo();
AtomicBoolean canHeRun = canIRunStore.get(expiredId);
canIRunStore.remove(expiredId);
canHeRun.set(false);
}
}
Related
In my service I have an endpoint that creates a resource. After creation the resource needs to be validated and otherwise "prepared" for future processing. To accomplish this my service creates the resource in the database, spawns an asynchronous thread to perform the validation, and then returns to the user.
Entry point:
#Override
public FooDto createFoo(FooDto fooDto) {
FooDto retDto = fooService.createFoo(fooDto); //Annotated with #Transactional
asyncFooService.initializeFooAsync(retDto.getFooId()); //Annotated with #Transactional and #Async
return retDto;
}
Async call
#Transactional
#Async
#Override
public void initializeFooAsync(String foodId) {
Foo foo = fooRepository.findById(foodId);
logger.info("Found foo with id={}", foo.getId())
//More processing which can take a while to perform
}
I was careful to ensure that I have exited the transactional boundaries so that the commit would run before the async call would happen, and that the async method lives in a different bean than the entry method. So logically this should have no issues seeing the data from the first method in the second, and the second should be running asynchronously.
What I have noticed is that the log message in the async call is sometimes throwing a null pointer exception as foo is null. By the time I get notified of this I can see in the database that the wanted foo record exists.
My persistance layer consists of three MySQL or MariaDB replicas (depending on the enviornment) in "master/master" configuration, so what I have derived is that the insert done in fooService.createFoo is going to nodeA, and the select done by initializeFooAsync is going to nodeB which has yet to persist the replication from nodeA. The further evidence I have for this is I have done a patch which, in the initializeFooAsync method checks for a null Food and tries to find it again after 3 seconds. This patch has worked.
I'm looking for other, "cleaner" approaches that don't utilize thread.sleep. The other approach that I thought of was using RMQ (which is available to me) and dead letter exchanges to create a delayed processing queue with limited amount of retries should Foo not be found (so if not found try again in Xms up to Y times). However this approach is being frowned upon by the chief architect.
The other approach I see is to do more of the same, and just do more checks in initializeFooAsync at shorter intervals to minimize wait time. Regardless it would essentially be the same solution using Thread.sleep to deal with replication delay.
Doing the initialization inline with the creation is not possible as this is a specific requirement from product, and the initialization may end up taking what they consider a "significant" amount of time due to coordination.
Is there some other utility or tool in the Spring/Java ecosystem that can help me deliver a cleaner approach? Preferably something that doesn't rely on sleeping my thread.
I am trying to integrate QFJ into a single-threaded application. At first I was trying to utilize QFJ with my own TCP layer, but I haven't been able to work that out. Now I am just trying to integrate an initiator. Based on my research into QFJ, I would think the overall design should be as follows:
The application will no longer be single-threaded, since the QFJ initiator will create threads, so some synchronization is needed.
Here I am using an SocketInitiator (I only handle a single FIX session), but I would expect a similar setup should I go for the threaded version later on.
There are 2 aspects to the integration of the initiator into my application:
Receiving side (fromApp callback): I believe this is straightforward, I simply push messages to a thread-safe queue consumed by my MainProcessThread.
Sending side: I'm struggling to find documentation on this front. How should I handle synchronization? Is it safe to call Session.sendToTarget() from the MainProcessThread? Or is there some synchronization I need to put in place?
As Michael already said, it is perfectly safe to call Session.sendToTarget() from multiple threads, even concurrently. But as far as I see it you only utilize one thread anyway (MainProcessThread).
The relevant part of the Session class is in method sendRaw():
private boolean sendRaw(Message message, int num) {
// sequence number must be locked until application
// callback returns since it may be effectively rolled
// back if the callback fails.
state.lockSenderMsgSeqNum();
try {
.... some logic here
} finally {
state.unlockSenderMsgSeqNum();
}
Other points:
Here I am using an SocketInitiator (I only handle a single FIX session), but I would expect a similar setup should I go for the threaded version later on.
Will you always use only one Session? If yes, then there is no use in utilizing the ThreadedSocketInitiator since all it does is creating a thread per Session.
The application will no longer be single threaded, since the QFJ initiator will create threads
As already stated here Use own TCP layer implementation with QuickFIX/J you could try passing an ExecutorFactory. But this might not be applicable to your specific use case.
I have a method in servlet that inserts tutoring bookings in database. This method has a business rule that checks if the tutor of this session is already busy in that date and hour. The code looks something like this :
class BookingService {
public void insert(Booking t) {
if(available(t.getTutor(), t.getDate(), t.getTime())) {
bookingDao.insert(t);
} else {
// reject
}
}
}
The problem is that multiple users may simultaneously try to book the same tutor on the same date and time, and there is nothing that prevents them both to pass the test and insert their bookings. I've tried making insert() synchronized and using locks, but it doesn't work. How can I prevent concurrent access to this method?
Using synchronized is an inadequate way to try to solve this problem:
First, you will have coded your application so that only one instance can be deployed at a time. This isn’t just about scaling in the cloud. It is normal for an IT department to want to stand up more than one instance of an application so that it is not a single point of failure (so that in case the box hosting one instance goes down the application is still available). Using static synchronized means that the lock doesn’t extend beyond one application classloader so multiple instances can still interleave their work in an error prone way.
If you should leave the project at some point, later maintainers may not be aware of this issue and may try to deploy the application in a way you did not intend. Using synchronized means you will have left a land mine for them to stumble into.
Second, using the synchronized block is impeding the concurrency of your application since only one thread can progress at a time.
So you have introduced a bottleneck, and at the same time cut off operations’ ability to work around the bottleneck by deploying a second instance. Not a good solution.
Since the posted code shows no signs of where transactions are, I’m guessing either each DAO creates its own transaction, or you’re connecting in autocommit mode. Databases provide transactions to help with this problem, and since the functionality is implemented in the database, it will work regardless of how many application instances are running.
An easy way to fix the problem which would avoid the above drawbacks would be to put the transaction at the service layer so that all the DAO calls would execute within the same transaction. You could have the service layer retrieve the database connection from a pool, start the transaction, pass the connection to each DAO method call, commit the transaction, then return the connection to the pool.
One way you could solve the problem is by using a synchronized block. There are many things you could choose as your locking object - for the moment this should be fine:
class BookingService {
public void insert(Booking t) {
synchronized(this) {
if(available(t.getTutor(), t.getDate(), t.getTime())) {
bookingDao.insert(t);
} else {
// reject
}
}
}
}
If you have more than one instance of the servlet, then you should use a static object as a lock.
I work on the backend of a java web application. I added some multithreading to speed up a big data retrieval, and I used a few ExecutorServices I created in the process. However, I've read that it might not be a good idea to be creating threads this way in a web app, and that the 'com.ibm.websphere.asynchbeans.WorkManager' is maybe an option instead. It seems not very friendly to use in the backend though. According to the documentation, the work manager is a 'thread pool created for Java Platform, Enterprise Edition (Java EE) applications that use asynchronous beans'. I, as a very front end ignorant back end guy, don't even entirely know what a bean is. It doesn't really seem like the work manager is what I want, but if it's actually a bad idea to manually create an ExecutorService, I'm not sure what the best approach is.
Creating your own threads in the back end of a webapp is a bad idea for the following reason: Application Server container manages threads for you. So if you create your own threads in your app the container won't know about them and won't be able to manage them. Thus if you mismanage threads your app can cause memory leaks and other thread related problems. In theory the best way is to register some thread pool in your App Server and requests threads from the container using JNDI. But it might be an overkill. So sometimes, you may want to manage your own threads. So, in this case ExecutorService is the best way to go as it provides very nice API to manage your threads. Just make sure that as your application shuts down you shutdown your ExecutorService so no orphan threads are left behind. If you are careful about it then you can create your own threads. Just use them sparingly and be careful that you you shut down your ExecutorService when done or when Application shuts down. BTW there is a similar issue with ThreadLocal variables. You absolutely MUST invoke metod remove() in them when done, otherwise they will remain in the memory even if your app is shutdown, and only Application Server restart will clear them. This is a dangerous memory leak.
Just to expand on Michael Gantman's answer, which is a pretty good explanation, the JavaEE approach to allowing the container manage your threadpool is super easy to actually implement:
#Stateless
public class Foo {
#Asynchronous
#Override
public void doSomething() {
//all calls to this function are asynchronous
}
}
The #Aysnchronous takes care of the magic here, specifying to the container that this function should be called asynchronously and it's the container's responsibility to figure out what that means. Then calling out to your container managed threadpool to execute your function is as simple as:
#Stateless
public class Bar {
#EJB
Foo myFooEJB;
public void businessMethod() {
myFooEJB.doSomething();
}
}
Checkout these for more reading: javax.ejb.Asynchronous, Asynchronous method invocation. Getting maybe a little too into specifics because I know you're using IBM products, in JBoss, your Asynchronous comes as part of a subsystem configuration by specifying <async thread-pool-name="poolName"/> (https://docs.jboss.org/author/display/WFLY8/EE+Subsystem+Configuration)
where poolName should reference a pool such as:
<thread-pools>
<thread-pool name="poolName">
<max-threads count="10"/>
<keepalive-time time="100" unit="milliseconds"/>
</thread-pool>
</thread-pools>
But there is a better way if you're trying to submit, just some task to an ExecutorService which can be managed by the container:
#Resource
ManagedExecutorService executorService;
#Resource
ManagedScheduledExecutorService scheduledExecutorService;
java.io.PrintWriter out = ...;
void scheduleSomeStuff(){
scheduledExecutorService.schedule(new Runnable() {
#Override
public void run() {
out.println("Print out after roughly 15 seconds.")
}
}, 15, TimeUnit.SECONDS);
executorService.submit(new Callable<Boolean>() {
#Override
public Boolean call() throws Exception {
out.println("Print only once, when the task is run by the executor service's discretion.");
}
};
}
These two resources, specifically that I've reference here, run on different thread pools. At that, I originally posted something here using the #Asynchronous thread pool as an executor, but it's unnecessary since the container should already have one for you; hence the updated answer you see here now.
This, again, might be a little specific to JBoss, but in JBoss this is configured as a subsystem managed-scheduled-executor-services and managed-executor-services. These two subsystems have their own distinct thread pools that they utilize.
EDIT April, 2020
So I somehow found this answer again, and reread and am very confused at myself for not calling out the EJB specification explicitly. Not only is it a bad idea to do, you would be in direct violation of the EJB spec if you created your own threads.
JSR 345: Enterprise JavaBeans, Version 3.2 EJB Core Contracts and Requirements
Section 16.2.2 Bean Provider’s Responsibilities, Programming Restrictions:
The enterprise bean must not attempt to manage threads. The enterprise bean must not attempt to start, stop, suspend, or resume a thread, or to change a thread’s priority or name. The enterprise bean must not attempt to manage thread groups.
These functions are reserved for the EJB container. Allowing the enterprise bean to manage threads would decrease the container’s ability to properly manage the runtime environment.
In other words, NEVER attempt to manage your own threads in a web application. This quote is for the EJB spec but even within a ResourceAdapter, for example, the container provides a WorkManager via the BootstrapContext (see ResourceAdapter#start(BootstrapContext), BootstrapContext#getWorkManager()). There's never a reason to attempt to create/manage your own thread in a web application; never do it for any reason.
we have a situation where we want to perform some tasks at the end of a request or transaction. More specifically, we need to collect some data during that request and at the end we use that data to do some automatic database updates.
This process should be as transparent as possible, i.e. users of the EJBs that require this should not have to worry about that.
In addition we can't control the exact call stack since there are multiple entry points to the process.
To achieve our goal, we're currently considering the following concept:
certain low level operations (that are always called) fire a CDI event
a stateless EJB listens for those events and upon receiving one it collects the data and stores it into a scoped CDI bean (either request scope or conversation scope would be fine)
at the end of the request another event is fired which causes the data in the scoped CDI bean to be processed
So far, we managed to get steps 1 and 2 up and running.
However, the problem is step 3:
As I already said, there are multiple entry points to the process (originating from web requests, scheduled jobs or remote calls) and thus we thought of the following approach:
3a. A CDI extension scans all beans and adds an annotation to every EJB.
3b. An interceptor is registered for the added annotation and thus on every call to an EJB method the interceptor is invoked.
3c. The first invocation of that interceptor will fire an event after the invoked method has returned.
And here's the problem (again in the 3rd step :) ):
How would the interceptor know whether it was the first invocation or not?
We thought of the following, but neither worked so far:
get a request/conversation scoped bean
fails because no context is active
get the request/conversation context and activate it (which then should mark the first invocation since in subsequent ones the context should be active)
the system created another request context and thus WELD ends up with at least two active request contexts and complained about this
the conversion context stayed active or was deactivated prematurely (we couldn't yet figure out why)
start a long running conversation and end it after the invocation
failed because there was no active request context :(
Another option we didn't try yet but which seems to be discouraged:
use a ThreadLocal to either store some context data or at least to use invocation context propagatation as described here: http://blog.dblevins.com/2009/08/pattern-invocationcontext-propagation.html
However, AFAIK there's no guarantee that the request will be handled entirely by the same thread and thus wouldn't even invocation context propagation break when the container decides to switch to another thread?
So, thanks to all who endured with me and read through all that lengthy description.
Any ideas of how to solve this are welcome.
Btw, here are some of the software components/standards we're using (and which we can't switch):
JBoss 7.1.0.Final (along with WELD and thus CDI 1.0)
EJB 3.1
Hibernate 3.6.9 (can't switch to 4.0.0 yet)
UPDATE:
With the suggestions you gave so far, we came up with the following solution:
use a request scoped object to store the data in
the first time an object is stored in that object an event is fired
a listener is invoked before the end of the transaction (using #Observes(during=BEFORE_COMPLETION) - thanks, #bkail)
This works so far but there's still one problem:
We also have MBeans that are managed by CDI and automatically registered to the MBean server. Thus those MBeans can get EJB references injected.
However, when we try and call an MBean method which in turn calls an EJB and thus causes the above process to start we get a ContextNotActiveException. This indicates that within JBoss the request context is not started when executing an MBean method.
This also doesn't work when using JNDI lookups to get the service instead of DI.
Any ideas on this as well?
Update 2:
Well, seems like we got it running now.
Basically we did what I described in the previous update and solved the problem with the context not being active by creating our own scope and context (which activated the first time an EJB method is called and deactivated when the corresponding interceptor finishes).
Normally we should have been able to do the same with request scope (at least if we didn't miss anything in the spec) but since there is a bug in JBoss 7.1 there is not always an active request context when calling EJBs from MBeans or scheduled jobs (which do a JNDI lookup).
In the interceptor we could try to get an active context and on failure activate one of those present in the bean manager (most likely EjbRequestContext in that case) but despite our tests we'd rather not count on that working in every case.
A custom scope, however, should be independent from any JBoss scope and thus should not interfere here.
Thanks to all who answered/commented.
So there's a last problem though: whose answer should I accept as you all helped us get into the right direction? - I'll try to solve that myself and attribute those points to jan - he's got the fewest :)
Do the job in a method which is annotated with #PreDestroy.
#Named
#RequestScoped
public class Foo {
#PreDestroy
public void requestDestroyed() {
// Here.
}
}
It's invoked right before the bean instance is destroyed by the container.
What you're looking for is SessionSynchronization. This lets an EJB tie in to the transaction lifecycle and be notified when transactions are being completed.
Note, I am being specific about Transactions, you mention "requests and transactions" and I don't know if you specifically mean EJB Transactions or something tied to your application.
But I'm talking about EJB Transactions.
The downside is that it is only invoked when the specific EJB is invoked, not to "all" transactions in general. But that may well be appropriate anyway.
Finally, be careful in these interim call back areas -- I've had weird things happen with transactional at these lifecycle methods. In the end, ended up putting stuff in to a local, memory based queue that another thread reaped for committing to JMS or whatever. Downside is that they were tied to the transaction at hand, upside was that they actually worked.
Phew, that's a complex scenario :)
From how I understand what you've tried so far you are pretty advanced with the CDI techniques - there is nothing big you are missing.
I would say that you should be able to activate a conversation context at the entry point (you've probably seen the relevant documentaton?) and work with it for the whole processing. It might actually be worthwhile considering implementing an own scope. I did that once in a distantly related scenario where we could not tell whether we've been invoked by HTTP-request or EJB-remoting.
But to be honest, all this feels far too complex. It's a rather fragile construct of interceptors notifying each other with events which all in all seems just too easy to break.
Can it be that there is another approach which better fits your needs? E.g. you might try to hook on the transaction management itself and execute your data accumulation from there?