Develop Reference

Spring/Hibernate: Best option for dealing with Galera/MySql/Mariadb replication delay during asynchronous processing

In my service I have an endpoint that creates a resource. After creation the resource needs to be validated and otherwise "prepared" for future processing. To accomplish this my service creates the resource in the database, spawns an asynchronous thread to perform the validation, and then returns to the user.
Entry point:
#Override
public FooDto createFoo(FooDto fooDto) {
FooDto retDto = fooService.createFoo(fooDto); //Annotated with #Transactional
asyncFooService.initializeFooAsync(retDto.getFooId()); //Annotated with #Transactional and #Async
return retDto;
}
Async call
#Transactional
#Async
#Override
public void initializeFooAsync(String foodId) {
Foo foo = fooRepository.findById(foodId);
logger.info("Found foo with id={}", foo.getId())
//More processing which can take a while to perform
}
I was careful to ensure that I have exited the transactional boundaries so that the commit would run before the async call would happen, and that the async method lives in a different bean than the entry method. So logically this should have no issues seeing the data from the first method in the second, and the second should be running asynchronously.
What I have noticed is that the log message in the async call is sometimes throwing a null pointer exception as foo is null. By the time I get notified of this I can see in the database that the wanted foo record exists.
My persistance layer consists of three MySQL or MariaDB replicas (depending on the enviornment) in "master/master" configuration, so what I have derived is that the insert done in fooService.createFoo is going to nodeA, and the select done by initializeFooAsync is going to nodeB which has yet to persist the replication from nodeA. The further evidence I have for this is I have done a patch which, in the initializeFooAsync method checks for a null Food and tries to find it again after 3 seconds. This patch has worked.
I'm looking for other, "cleaner" approaches that don't utilize thread.sleep. The other approach that I thought of was using RMQ (which is available to me) and dead letter exchanges to create a delayed processing queue with limited amount of retries should Foo not be found (so if not found try again in Xms up to Y times). However this approach is being frowned upon by the chief architect.
The other approach I see is to do more of the same, and just do more checks in initializeFooAsync at shorter intervals to minimize wait time. Regardless it would essentially be the same solution using Thread.sleep to deal with replication delay.
Doing the initialization inline with the creation is not possible as this is a specific requirement from product, and the initialization may end up taking what they consider a "significant" amount of time due to coordination.
Is there some other utility or tool in the Spring/Java ecosystem that can help me deliver a cleaner approach? Preferably something that doesn't rely on sleeping my thread.

Singleton returns null when accessed by threads

As the title states I'm trying to troubleshoot an issue where some threads which read data from a Singleton get a null value. My investigation into our logs read as though its a concurrency issue.
The Singleton is defined as follows:
#Singleton
public class StaticDatabaseEntries {
private static final Map<String,Thing> databaseEntries = new HashMap<>();
#Lock(LockType.READ)
public Thing getThing(String index) {
return databaseEntries.get(index);
}
}
At first I was under the impression that only one element within the data was corrupted as access to the same item is repeatededly returning null. Further access to debug entries show that the issue appears isolated to a specific thread. It's as though once whatever occurs that induces the null return on a thread continues to do so but only on the affected thread.
An earlier version of this class did not apply the LockType.READ so per the specification a LockType.WRITE is assumed. I deployed an update with the correct lock to enable concurrent read. This did not improve the situation.
The data is loaded into the HashMap from a database upon deployment and remains unchange for the duration. Since the class isn't tagged with #Startup the application instead uses a context listener to trigger the loading of the entries from database.
With threads primarily performing a read activity I don't believe a switch to ConcurrentHashMap is benficial. I am considering removing the static final portion as it seems unnecessary when the container is managing concurrent access and the singleton lifecycle. I have experienced side effects when the container cannot subclass/proxy things which are marked as final in EJBs.
The other possibility I've considered is there is some manner of bug in the container software. This is running on a older Java 1.7 and JBOSS 6 EAP. Worst case I'll have to forego the singleton pattern and instead load the entries from the database on demand.

In general: If you work with threads, read activity can cause problems, if you are calling an object method, that isn't thread-save! This is exactly what often leads to undetectable errors in larger projects.
HashMap isn't thread-safe!
You should switch to ConcurrentHashMap.
For further details, have a look at this article: https://www.baeldung.com/java-concurrent-map

Preventing concurrent access to a method in servlet

I have a method in servlet that inserts tutoring bookings in database. This method has a business rule that checks if the tutor of this session is already busy in that date and hour. The code looks something like this :
class BookingService {
public void insert(Booking t) {
if(available(t.getTutor(), t.getDate(), t.getTime())) {
bookingDao.insert(t);
} else {
// reject
}
}
}
The problem is that multiple users may simultaneously try to book the same tutor on the same date and time, and there is nothing that prevents them both to pass the test and insert their bookings. I've tried making insert() synchronized and using locks, but it doesn't work. How can I prevent concurrent access to this method?

Using synchronized is an inadequate way to try to solve this problem:
First, you will have coded your application so that only one instance can be deployed at a time. This isn’t just about scaling in the cloud. It is normal for an IT department to want to stand up more than one instance of an application so that it is not a single point of failure (so that in case the box hosting one instance goes down the application is still available). Using static synchronized means that the lock doesn’t extend beyond one application classloader so multiple instances can still interleave their work in an error prone way.
If you should leave the project at some point, later maintainers may not be aware of this issue and may try to deploy the application in a way you did not intend. Using synchronized means you will have left a land mine for them to stumble into.
Second, using the synchronized block is impeding the concurrency of your application since only one thread can progress at a time.
So you have introduced a bottleneck, and at the same time cut off operations’ ability to work around the bottleneck by deploying a second instance. Not a good solution.
Since the posted code shows no signs of where transactions are, I’m guessing either each DAO creates its own transaction, or you’re connecting in autocommit mode. Databases provide transactions to help with this problem, and since the functionality is implemented in the database, it will work regardless of how many application instances are running.
An easy way to fix the problem which would avoid the above drawbacks would be to put the transaction at the service layer so that all the DAO calls would execute within the same transaction. You could have the service layer retrieve the database connection from a pool, start the transaction, pass the connection to each DAO method call, commit the transaction, then return the connection to the pool.

One way you could solve the problem is by using a synchronized block. There are many things you could choose as your locking object - for the moment this should be fine:
class BookingService {
public void insert(Booking t) {
synchronized(this) {
if(available(t.getTutor(), t.getDate(), t.getTime())) {
bookingDao.insert(t);
} else {
// reject
}
}
}
}
If you have more than one instance of the servlet, then you should use a static object as a lock.

How to create singleton java class for multiple jvm support?

For example I have DBManager.java Singleton Class, which I have to deploy on clustered environment.
It is a web based application, with following deployment stratergy
Apache Load Balancer --> Tomcat 6 (3 Servers in cluster).
I have to maintain single instance of DBManager for 3 tomcat instances.
My code is
package com.db.util;
public class DBManager {
private static DBManager singleInstance;
private DBManager () {}
public static DBManager getSingleInstance() {
if (singleInstance == null) {
synchronized (DBManager.class) {
if (singleInstance == null) {
singleInstance = new DBManager ();
}
}
}
return singleInstance;
}
}
I have been searching a solution to this problem, and found something like JGroups API.
Can this be achieved using JGroups ? Any Idea, How to implement that ?

Java gives you a singleton in each instance, you need some kind of coordination between the instances so at any given time one of them is active, but if the active one dies then a different instance becomes active.
Some app servers have built in capabilities to control such coordinated worker instances, I don't know whether Tomcat has such a function.
Building such functionality yourself is surprisingly difficult, see this question and note that that question gives links to a useful library - which to me looks quite complex to use.
However in your case you have a database, and that gives you a point of coordination. I haven't designed this in detail, but I reckon it's possible to create a reservation scheme using a dedicated row in a control table. It will be a bit tricky to do this efficiently, balancing the speed of detection of an instance death with the overheads of polling the database to see which instance is active, but it seems doable.
The idea is that the record contains a "reservedUntil" timestamp and "processId". Each process reads the record, if it contains it's own id and the timestamp has not yet expired it knows it can work. When the time is nearly expired, the active process updates the timestamp using an optimistic locking style "Update where timestamp == old timestamp" to manage race conditions. Each non active process waits until the timestamp it last read has expired and then attempts to to take control by updating the record, again using an optimistic locking Update where. Usually that attempt to take control will fail, but if it succeeds we now have a new active instance, and due to optimistic locking we can only ever get one active instance.

Singleton ensures only one instance of the class in a given JVM.
What is the issue with multiple DBManagers, one for each JVM, in your case?

Handling a timeout in EJB3 without using threads

I have the following situation. I have a job that:
May time out after a given amount of time, and if so occurs needs to throw an exception
If it does not time out, will return a result
If this job returns a result, it must be returned as quickly as possible, because performance is very much an issue. Asynchronous solutions are hence off the table, and naturally tying up the system by hammering isn't an option either.
Lastly, the system has to conform to the EJB standard, so AFAIK using ordinary threads is not an option, as this is strictly forbidden.
Our current solution uses a thread that will throw an exception after having existed for a certain amount of time without being interrupted by an external process, but as this clearly breaks the EJB standard, we're trying to solve it with some other means.
Any ideas?
Edited to add: Naturally, a job which has timed out needs to be removed (or interrupted) as well.
Edited to add 2:
This issue doesn't seem to have any solution, because detecting a deadlock seems to be mostly impossible sticking to pure EJB3 standards. Since Enno Shioji's comments below reflect this, I'm setting his suggestion as the correct answer.

This is more like a request for clarification, but it's too long to fit as a comment..
I'm not sure how you are doing it right now, since from what you wrote, just using the request processing thread seems to be the way to go. Like this:
//Some webservice method (synchronous)
public Result process(Blah blah){
try{
return getResult(TimeUnit.SECONDS, 10);
}catch(InterruptedException e){
//No result within 10 seconds!
throw new ServiceUnavailableException("blah");
}
}
I'm not sure why you are creating threads at all. If you are forced to use threads because the getResult method doesn't timeout at all, you would have a thread leak. If it timeouts after a longer time and thus you want to "shortcut" your reply to the user, that would be the only case I'd consider using a thread like I imagine how you are using it. This could result in Threads piling up under load and I'd strive to avoid such situation.
Maybe you can post some code and let us know why you are creating in your service at all?
Also, what's your client interface? Sounds like it's a synchronous webservice or something?
In that case, if I were you I would use a HashedWheelTimer as a singleton... this mechanism should work great with your requirement (here is an implementation). However, this unfortunately seem to conflict with the ban on threading AND the ban on singleton in the EJB spec. In reality though there really isn't a problem if you would do this. See this discussion for example. We have also used the singleton pattern in our EJB app. which used JBoss. However, if this isn't a viable choice then I might look at isolating the processing in its own JVM by defining a new web service (and deploy it in a web-container or something), and call that service from the EJB app. This would however obviously incur performance hit and now you would have another whole new app.

With Bean Managed Transaction, the timeout for the specific transaction can be specified by using UserTransaction interface.
Modify the timeout value that is
associated with transactions started
by the current thread with the begin
method.
void setTransactionTimeout(int seconds) throws SystemException
Transaction will timeout after specified seconds & may not get propagated further. If exception is not thrown implicitly, then can throw it explicitly based on the result.
Will return a result on successful completion within specified time.
Can use it with stateless session beans so there may not be a performance issue.
Its EJB standard so that will not be an issue to implement.
With little-bit work around, it should work fine in the given scenario.
Edit : Also can use server specific properties to manage transaction timeout.
JBoss : At either at class or method level annotation #TransactionTimeout(100) can be applied.
Weblogic : Specifying the parameters in weblogic-ejb-jar.xml
<transaction-descriptor>
<trans-timeout-seconds>100</trans-timeout-seconds>
</transaction-descriptor>
GlassFish : Using the optional cmt-timeout-in-seconds element in sun-ejb-jar.xml

Stick the process and it's timeout thread in to a class annotated with #WebService, put that class in to a WAR, then invoke the WebService from your EJB.
WARs don't have the same limitations or live under the same contract that EJBs do, so they can safely run threads.
Yes, I consider this a "hack", but it meets the letter of the requirements, and it's portable.

You can create threads using the commonj WorkManager. There are implementations built into WebSphere and Weblogic as they proposed the standard, but you can also find implementations for other appservers as well.
Basically, the WorkManager allows you to create managed threads inside the container, much like using an Executor in regular Java. Your only other alternative would be to use MDB's, but that would be a 'heavier' solution.
Since I don't know your actual platform, you will have to google commonj with your platform yourself 8-)
Here is a non IBM or Oracle solution.
Note: This is not an actual standard, but it is widely available for different platforms and should suit your purposes nicely.

For EJBs, there is a concept of "Container Managed Transactions". By specifying #TransactionAttribute on your bean, or specific method, the container will create a transaction when ever the method(s) are invoked. If the execution of the code takes longer than the transaction threshold, the container will throw an exception. If the call finishes under the transaction threshold, it will return as usual. You can catch the exception in your calling code and handle it appropriately.
For more on container managed transactions, check out: http://java.sun.com/j2ee/tutorial/1_3-fcs/doc/Transaction3.html and http://download.oracle.com/javaee/5/tutorial/doc/bncij.html

You could use #TimeOut. Something like:
#Stateless
public class TimedBean {
#Resource
private TimerService timerService;
static private AtomicInteger counter = new AtomicInteger(0);
static private Map<Integer, AtomicBoolean> canIRunStore = new ...;
public void doSomething() {
Integer myId = counter.getAndIncrement();
AtomicBoolean canIRun = new AtomicBoolean(true);
canIRunStore.put(myId, canIRun);
timerService.createTimer(1000, 0, myId);
while (canIRun.get() /* && some other condition */) {
// do my work ... untill timeout ...
}
}
#Timeout
#PermitAll
public void timeout(Timer timer) {
Integer expiredId = (Integer) timer.getInfo();
AtomicBoolean canHeRun = canIRunStore.get(expiredId);
canIRunStore.remove(expiredId);
canHeRun.set(false);
}
}