I am using Hibernate to implement the DAO layer (Sybase DB) in a web application running on Jboss5.
The problem I am facing is when the client/UI makes multiple simultaneous HTTP calls - which in-turn calls the DAO insert method - there is a race condition which causes both calls to run the DAO insert method at near same time. What I actually want is
The 1st request calls the DAO method
1st request reads the current db value
Check if new data is valid based on current db value
If valid, insert the new value
AND then the 2nd request to read the current db value
Check if new data is valid
If valid, insert the value...and so on if there are more calls
My DAO layer code looks like so:
#Override
#Transactional
public Set<PartyData> insertPartyData(final Set<PartyData> pData) throws DataServiceException
{
sessionFactory.getCurrentSession().getTransaction().begin();
//code to read the current db value
//validation code to check if new value can be inserted based on what's currently in db
sessionFactory.getCurrentSession().save(pData);
sessionFactory.getCurrentSession().getTransaction().commit();
}
Question?
How can I make sure that the db locks the table for the duration of one transaction so that any other request waits until the previous transaction is complete?
While locking the table will work - the process of locking an entire table just doesn't seem right. There must be other ways to accomplish what you're trying to do, like unique constraints for example.
IMO this is best to be done in the container level rather than the application level (Don't know if it could even be done in the application level).
Based on this article, you can do what you like by configuring two attributes in the Connector.
Set maxThreads to 1 (maxThreads: The maximum number of request processing threads to be created by this connector, which therefore determines the maximum number of simultaneous requests that can be handled. If not specified, this attribute is set to 200.)
This will make sure that you run exactly one request at each time.
Increase the acceptCount : (acceptCount: The maximum queue length for incoming connection requests, when all possible request processing threads are in use. Any requests received when the queue is full will be refused. The default value is 10.)
This should be set relatively high so that you do not deny connections to your service, rather add then to queue untill another request finishes execution.
* ANSWER *
Ok I have tried the below solution which seems to have worked so far.
What seems to happen is each request seems to create a new session and that session runs in its own transaction. So no two request seem to interfere with each other and each transaction runs in one go.
I am not an expert in hibernate so please correct me if this is not the right way to do it.
#Override
#Transactional
public Set<PartyData> insertPartyData(final Set<PartyData> pData) throws DataServiceException
{
final Session session = sessionFactory.openSession();
Transaction tx;
try {
tx = session.beginTransaction();
\\read curren db value and do the validation with new data (throw exception if validation fails else continue)
session.save(pData);
}
tx.commit();
}
catch (final Exception e) {
throw new DataServiceException(e);
}
finally {
session.close();
}
return pData;
}
Related
I have a spring boot web application with the functionality to update an entity called StudioLinking. This entity describes a temporary, mutable, descriptive logical link between two IoT devices for which my web app is their cloud service. The Links between these devices are ephemeral in nature, but the StudioLinking Entity persists on the database for reporting purposes. StudioLinking is stored to the SQL based datastore in the conventional way using Spring Data/ Hibernate. From time to time this StudioLinking entity will be updated with new information from a Rest API. When that link is updated the devices need to respond (change colors, volume, etc). Right now this is handled with polling every 5 seconds but this creates lag from when a human user enters an update into the system and when the IoT devices actually update. It could be as little as a millisecond or up to 5 seconds! Clearly increasing the frequency of the polling is unsustainable and the vast majority of the time there are no updates at all!
So, I am trying to develop another Rest API on this same application with HTTP Long Polling which will return when a given StudioLinking entity is updated or after a timeout. The listeners do not support WebSocket or similar leaving me with Long Polling. Long polling can leave a race condition where you have to account for the possibility that with consecutive messages one message may be "lost" as it comes in between HTTP requests (while the connection is closing and opening, a new "update" might come in and not be "noticed" if I used a Pub/Sub).
It is important to note that this "subscribe to updates" API should only ever return the LATEST and CURRENT version of the StudioLinking, but should only do so when there is an actual update or if an update happened since the last checkin. The "subscribe to updates" client will initially POST an API request to setup a new listening session and pass that along so the server knows who they are. Because it is possible that multiple devices will need to monitor updates to the same StudioLinking entity. I believe I can acomplish this by using separately named consumers in the redis XREAD. (keep this in mind for later in the question)
After hours of research I believe the way to acomplish this is using redis streams.
I have found these two links regarding Redis Streams in Spring Data Redis:
https://www.vinsguru.com/redis-reactive-stream-real-time-producing-consuming-streams-with-spring-boot/
https://medium.com/#amitptl.in/redis-stream-in-action-using-java-and-spring-data-redis-a73257f9a281
I also have read this link about long polling, both of these links just have a sleep timer during the long polling which is for demonstration purposes but obviously I want to do something useful.
https://www.baeldung.com/spring-deferred-result
And both these links were very helpful. Right now I have no problem figuring out how to publish the updates to the Redis Stream - (this is untested "pseudo-code" but I don't anticipate having any issues implementing this)
// In my StudioLinking Entity
#PostUpdate
public void postToRedis() {
StudioLinking link = this;
ObjectRecord<String, StudioLinking> record = StreamRecords.newRecord()
.ofObject(link)
.withStreamKey(streamKey); //I am creating a stream for each individual linking probably?
this.redisTemplate
.opsForStream()
.add(record)
.subscribe(System.out::println);
atomicInteger.incrementAndGet();
}
But I fall flat when it comes to subscribing to said stream: So basically what I want to do here - please excuse the butchered pseudocode, it is for idea purposes only. I am well aware that the code is in no way indicative of how the language and framework actually behaves :)
// Parameter studioLinkingID refers to the StudioLinking that the requester wants to monitor
// updateList is a unique token to track individual consumers in Redis
#GetMapping("/subscribe-to-updates/{linkId}/{updatesId}")
public DeferredResult<ResponseEntity<?>> subscribeToUpdates(#PathVariable("linkId") Integer linkId, #PathVariable("updatesId") Integer updatesId) {
LOG.info("Received async-deferredresult request");
DeferredResult<ResponseEntity<?>> output = new DeferredResult<>(5000l);
deferredResult.onTimeout(() ->
deferredResult.setErrorResult(
ResponseEntity.status(HttpStatus.REQUEST_TIMEOUT)
.body("IT WAS NOT UPDATED!")));
ForkJoinPool.commonPool().submit(() -> {
//----------------------------------------------
// Made up stuff... here is where I want to subscribe to a stream and block!
//----------------------------------------------
LOG.info("Processing in separate thread");
try {
// Subscribe to Redis Stream, get any updates that happened between long-polls
// then block until/if a new message comes over the stream
var subscription = listenerContainer.receiveAutoAck(
Consumer.from(studioLinkingID, updateList),
StreamOffset.create(studioLinkingID, ReadOffset.lastConsumed()),
streamListener);
listenerContainer.start();
} catch (InterruptedException e) {
}
output.setResult("IT WAS UPDATED!");
});
LOG.info("servlet thread freed");
return output;
}
So is there a good explanation to how I would go about this? I think the answer lies within https://docs.spring.io/spring-data/redis/docs/current/api/org/springframework/data/redis/core/ReactiveRedisTemplate.html but I am not a big enough Spring power user to really understand the terminology within Java Docs (the Spring documentation is really good, but the JavaDocs is written in the dense technical language which I appreciate but don't quite understand yet).
There are two more hurdles to my implementation:
My exact understanding of spring is not at 100% yet. I haven't yet reached that a-ha moment where I really fully understand why all these beans are floating around. I think this is the key to why I am not getting things here... The configuration for the Redis is floating around in the Spring ether and I am not grasping how to just call it. I really need to keep investigating this (it is a huge hurdle to spring for me).
These StudioLinking are short lived, so I need to do some cleanup too. I will implement this later once I get the whole thing up off the ground, I do know it will be needed.
Why don't you use a blocking polling mechanism? No need to use fancy stuff of spring-data-redis. Just use simple blocking read of 5 seconds, so this call might take around 6 seconds or so. You can decrease or increase the blocking timeout.
class LinkStatus {
private final boolean updated;
LinkStatus(boolean updated) {
this.updated = updated;
}
}
// Parameter studioLinkingID refers to the StudioLinking that the requester wants to monitor
// updateList is a unique token to track individual consumers in Redis
#GetMapping("/subscribe-to-updates/{linkId}/{updatesId}")
public LinkStatus subscribeToUpdates(
#PathVariable("linkId") Integer linkId, #PathVariable("updatesId") Integer updatesId) {
StreamOperations<String, String, String> op = redisTemplate.opsForStream();
Consumer consumer = Consumer.from("test-group", "test-consumer");
// auto ack block stream read with size 1 with timeout of 5 seconds
StreamReadOptions readOptions = StreamReadOptions.empty().block(Duration.ofSeconds(5)).count(1);
List<MapRecord<String, String, String>> records =
op.read(consumer, readOptions, StreamOffset.latest("test-stream"));
return new LinkStatus(!CollectionUtils.isEmpty(records));
}
I am quite new to hibernate and I was learning about first-level caching in hibernate. I have a concern in first level cache consistency.
Imagine I have two separate Web Applications which can read/write to the same database. Both applications use hibernate. First application consists following code segment.
//First Application code
//Open the hibernate session
Session session = HibernateUtil.getSessionFactory().openSession();
session.beginTransaction();
//fetch the client entity from database first time
Client client= (Client) session.load(Client.class, new Integer(77869));
System.out.println(client.getName());
//execute some code which runs for several minutes
//fetch the client entity again
client= (Client) session.load(Client.class, new Integer(77869));
System.out.println(client.getName());
session.getTransaction().commit();
The second application consists of the following code.
//Second Application code
//Open the hibernate session
Session session = HibernateUtil.getSessionFactory().openSession();
session.beginTransaction();
//fetch the client entity from database first time
String hql = "UPDATE client set name = :name WHERE id = :client_id";
Query query = session.createQuery(hql);
query.setParameter("name", "Kumar");
query.setParameter("client_id", "77869");
int result = query.executeUpdate();
System.out.println("Rows affected: " + result);
session.getTransaction().commit();
Let's say the first application creates the session at 10.00 AM. the first application keeps that session object live for 10 minutes. Meanwhile, at 10.01AM second application do an update to Database (update CLIENT set name = 'Kumar' where id = 77869).
So first level cache in first application is outdated after 10.01AM. am I right? if so, is there any method to avoid this scenario?
There is no implicit way your first application will know about the underlying changes that were triggered by the second application.
One of the possible ways to handle this situation could be the following:
1) Once one of the applications does update / insert it needs to save a flag of some sort in the database for example that the data has been changed.
2) In the other application you just after you start a new session you need to check whether that flag is set and therefore the data has been altered.
If so, you need to set the CacheMode of the session accordingly:
session.setCacheMode(CacheMode.REFRESH);
This would ensure that during this session all the queried entities will not be taken from the cache but from the DB (therefore updating the cache in the process). Most likely this will not update all the changes in the second-level change, you would need to set that session attribute periodically every now and then.
Keep in mind that second-level-caching anything else than dictionary entities that do not really change is tricky in terms of consistency.
I've created a mariadb cluster and I'm trying to get a Java application to be able to failover to another host when one of them dies.
I've created an application that creates a connection with "jdbc:mysql:sequential://host1,host2,host3/database?socketTimeout=2000&autoReconnect=true". The application makes a query in a loop every second. If I kill the node where the application is currently executing the query (Statement.executeQuery()) I get a SQLException because of a timeout. I can catch the exception and re-execute the statement and I see that the request is being sent to another server, so failover in that case works ok. But I was expecting that executeQuery() would not throw an exception and silently retry another server automatically.
Am I wrong in assuming that I shouldn't have to handle an exception and explicitely retry the query? Is there something more I need to configure for that to happen?
It is dangerous to auto reconnect for the following reason. Let's say you have this code:
BEGIN;
SELECT ... FROM tbl WHERE ... FOR UPDATE;
(line 3)
UPDATE tbl ... WHERE ...;
COMMIT;
Now let's say the server crashes at (line 3). The transaction will be rolled back. In my fabricated example, that only involves releasing the lock on tbl.
Now let's say that some other connection succeeds in performing the same transaction on the same row while you are auto-reconnecting.
Now, with auto-reconnect, the first thread is oblivious that the first half of the transaction was rolled back and proceeds to do the UPDATE based on data that is now out of date.
You need to get an exception so that you can go back to the BEGIN so that you can be "transaction safe".
You need this anyway -- With Galera, and no crashes, a similar thing could happen. Two threads performing that transaction on two different nodes at the same time... Each succeeds until it gets to the COMMIT, at which point the Galera magic happens and one of the COMMITs is told to fail. The 'right' response is replay the entire transaction on the server that was chosen for failure.
Note that Galera, unlike non-Galera, requires checking for errors on COMMIT.
More Galera tips (aimed at devs and dbas migrating from non-Galera)
Failover doesn't mean that application doesn't have to handle exceptions.
Driver will try to reconnect to another server when connection is lost.
If driver fail to reconnect to another server a SQLNonTransientConnectionException will be thrown, pools will automatically discard those connection.
If connection is recovered, there is some marginals cases where relaunching query is safe: when query is not in a transaction, and connection is currently in read-only mode (using Spring #Transactional(readOnly = false)) for example. For thoses cases, MariaDb java connection will then relaunch query automatically. In those particular cases, no exception will be thrown, and failover is transparent.
Driver cannot re-execute current query during a transaction.
Even without without transaction, if query is an UPDATE command, driver cannot know if the last request has been received by the database server and executed.
Then driver will send an SQLException (with SQLState begining by "25" = INVALID_TRANSACTION_STATE), and it's up to the application to handle those cases.
Traditionally, we try to avoid LazyInitializationExceptions. However, I need to temporarily allow them to be thrown. This is the pseudo code of what I'm trying to do:
Session session = ...;
Customer customer = (Customer) session.get(Customer.class, 56);
// All eagerly fetched data is now in memory.
disconnectSession(session);
// Attempts to access lazy data throw LazyInitializationExceptions.
// I want to take advantage of this fact to *only* access data that
// is currently in memory.
magicallySerializeDataCurrentlyInMemory(customer);
// Now that the in-memory data has been serialized, I want to re-connect
// the session.
reconnectSession(session);
The magicallySerializeDataCurrentlyInMemory method recursively attempts to serialize in-memory data of customer and its related entities, absorbing LazyInitializationExceptions along the way.
Attempt #1: session.disconnect / session.reconnect
In this attempt, I used this pattern:
Connection connection = session.disconnect();
magicallySerializeDataCurrentlyInMemory(customer);
session.reconnect(connection);
Unfortunately, it didn't throw LazyInitializationExceptions.
Attempt #2: session.close / session.reconnect
In this attempt, I used this pattern:
Connection connection = session.close();
magicallySerializeDataCurrentlyInMemory(customer);
session.reconnect(connection);
Unfortunately, this rendered the session useless after session.reconnect(connection).
How can I temporarily force LazyInitializationExceptions?
There isn't a way to temporarily close the session. What you can do is remove the Entity from the session and then put it back.
session.evict(customer);
//do something will throw lazy load
session.refresh(customer);
connect and reconnect just manually manage what particular JDBC connection is in use. A hibernate Session is a larger concept than a single JDBC connection. The connection is just a resource to it. It can go through using many different physical database connections throughout its lifecycle and not care one bit. If you disconnect then ask the Session to do something, it will just go get another connection.
I am trying to write a mechanism that will manage my save to DB operation.
I send the server a list of objects, it iterates them and saves each one.
Now, if they fail for some strange reason (exception) it saves them to another list that
has a timer that runs every 5 seconds, and tries to re-save them.
I then have a locking problem, which I can solve with another boolean.
My function that saves my lost object is:
private void saveLostDeals() {
synchronized (unsavedDeals) {
if (unsavedDeals.size() > 0) {
for (DealBean unsavedDeal : unsavedDeals) {
boolean successfullySaved = reportDeal(unsavedDeal,false);
if (successfullySaved) {
unsavedDeals.remove(unsavedDeal);
}
}
}
}
}
And my reportDeal() method that's being called for regular reports and for lost deals report:
try {
...
} catch (HibernateException e) {
...if (fallback)
synchronized (unsavedDeals) {
unsavedDeals.add(deal);
}
session.getTransaction().rollback();
} finally {
....
}
Now, when a lost deal is saved - if an exception occurs - the synchronized block will stop it.
What do you have to say about this save fallback mechanism? Are there better design patterns to deal with this common issue?
I would suggest using either a proxy or aspects to handle the rollback/retry mechanism. The proxy could use something like the strategy pattern for advice on what action to take.
If you however don't want to retry immediately, but say in 5 seconds as you propose, I would consider building that into the contract of your database layer by providing asynchronous routines to begin with. Something like dao.scheduleStore(o); or dao.asyncStore(o);.
It depends
For example,
Request to save Entity ---> Exception occurs ---> DB Connection Problem----> In Exception Block Retry to save entity in fallback DB-----> Return the response to request
Request to save Entity ---> Exception occurs ---> DB Connection Problem----> In Exception Block Retry to save entity in in-memory Store of Application -----> Return the response to request
Request to save Entity ----> Exception occurs----> unknown Exception----> In Exception block save entity to XML File store[serialize in XML]---->Return the response mentioning temp saved will be updated later to request
Timer ----> checks the file store for any serialized XML ----> updates the DB
Points to watch out for
Async calls are better in such scenarios rather than making requesting client to wait.
In case of in-memory saving , watch out for amount of data saved in memory in case of prolonged DB failure. That might bring down the whole application
Transactions, whether you want to rollback of save its intermittent state.
consistency of data to be watched for