I have a really weird issue with a project I'm working with. I would appreciate if someone could point me to a right direction here.
// Setup
There are multiple web servers and a loadbalancer is in front of them. Servers are handling requests that might come in multiple parts and parts can be handled by different servers. These multi-part requests should be combined to a one single transaction that is going forward once all the parts are received.
The server that does the final processing doesn't matter, but only one server can do it. Other servers that receive the previous parts should just mark the part received, store the data and give a immediate response back.
For now I'm using database table to handle the synchronization between nodes.
The basic idea is that when a server gets a part it tries to acquire the lock with a transaction id coming with the rquest. This is done by trying to insert a row to a Lock table with the txid as a primary key. If insert is successful, that server gets the lock and processes the part it received, by storing it to database checks if other parts have been received and returns a response immediately if not.
// The Problem
The problem I have is that the threads seem to randomly lock at the database and thus freezing the whole processing. I have debugged it to the point that in a situation where multiple requests come to processing at the same time they just get stuck at trying to acquire the lock and ultimately timeout after 30 seconds. Few of the first requests might get processed or not it seems to be random but even something like 7 concurrent requests block the database.
To me there should not be any way how this could get stuck and I'm fresh out of ideas.
// Information
I am using MySQL with an InnoDB engine. Servers are running Java code and Hibernate is used as a ORM layer to access the DB.
The Lock table:
CREATE TABLE `lock` (
`id` varchar(255) NOT NULL,
`expiryDate` datetime DEFAULT NULL,
`issueDate` datetime DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
The id is the transaction id used to combine the parts.
I have an basic interface that manages the lock accessing.
public interface LockProviderDao {
public boolean lock(String id);
public boolean unlock(String id);
}
And a implementation of that class that uses Hibernate to access database.
#Override
public boolean lock(String id) {
Session session = this.sessionFactory.openSession();
Lock lock = new Lock(id);
Transaction tx = null;
boolean locked = false;
try {
// Try to lock
tx = session.beginTransaction();
session.save(lock);
tx.commit();
locked = true;
} catch(Exception e) {
if(tx != null) {
tx.rollback();
}
} finally {
session.close();
}
return locked;
}
#Override
public boolean unlock(String id) {
Session session = this.sessionFactory.openSession();
boolean status = true;
Transaction tx = null;
try {
Lock lock = (Lock) session.load(Lock.class, id);
tx = session.beginTransaction();
session.delete(lock);
tx.commit();
} catch(Exception e) {
if(tx != null) {
tx.rollback();
}
status = false;
} finally {
session.close();
}
return status;
}
Seems simple enough. Here is the code that does the processing. This thread has a Hibernate session opened already so the Session opened inside the lock and unlock methods is a nested Session, if that makes any difference.
int counter = 0;
boolean lockAcquired = false;
do {
// Try to acquire the lock
lockAcquired = this.lockProviderDao.lock(txId);
if (!lockAcquired) {
// Didn't get it try a bit later
try {
Thread.sleep(defaultSleepPeriod);
} catch (Exception e) {
}
if (counter >= defaultSleepCycles) {
return;
}
counter++;
}
} while (!lockAcquired);
// DO THE PROCESSING HERE ONCE LOCK ACQUIRED
// Release the lock
this.lockProviderDao.unlock(txId);
I would lock after inserting the data. This means, that you would have to change your algorithm to something like this:
Begin transaction
Insert the fragment to database
Commit transaction
Begin transaction
Count number of framgents inserted / exit, if not equal to expected fragment count
Insert a row, that indicates that fragments will be processed (e.g. your lock row). If this fails, fragments have been processed or are being processed (= rollback)
Commit transaction
Begin transaction
Read fragments (and verify that they still exist)
Process fragments
Delete lock and fragments (verify they still exist)
Commit transaction
If you need to increase reliability, you have three options:
1. Use JMS with JTA to control program flow
2. Have your client poll the server for status and start processing, if all parts have been received, but processing has not started yet or has been stalled
3. Create a scheduler that starts processing, if same conditions apply
Related
I have a table called ad_session which logs user sessions. I am using Java to get a list of all successful sessions from that table. I then loop through that list to get the user for each session (which is a foreign key to the ad_user table). I then get the client that belongs to that user, and I add the client to a list. However, one of the users no longer exists, so my code stops running and it gives throws the following exception:
org.hibernate.ObjectNotFoundException: No row with the given identifier exists: [ADUser#76A5C22E6D2446A399AE9AD7C1DED0C7]
This is my original code:
List<Session> sessions = getAllSuccessfulSessionsInTable();
List<Client> clientsForThatDay = new ArrayList<>();
try {
for (Session session : sessions) {
//code fails when trying to get the non-existent user:
User user = session.getCreatedBy();
Client userClient = user.getClient();
clientsForThatDay.add(userClient);
}
} catch (Exception e) {
log.error("Error getting client from user: ", e);
}
I assumed that when getting a non-existent record, it would return null, so this is what I tried:
List<Session> sessions = getAllSuccessfulSessionsInTable();
List<Client> clientsForThatDay = new ArrayList<>();
//Create new user object to stand in place of the non-existent user
User deletedUser = new User();
deletedUser.setName("Deleted User");
//Create new client object to stand in place of the non-existent client
Client deletedUserClient = new Client();
deletedUserClient.setName("Unknown Client");
try {
for (Session session : sessions) {
//check is User is null, if it is, use the deletedUser object, otherwise, use the existing user
User user = session.getCreatedBy() == null ? deletedUser : session.getCreatedBy();
Client userClient = user.getName().equals("Deleted User") ? deletedUserClient : user.getClient();
clientsForThatDay.add(userClient);
}
} catch (Exception e) {
log.error("Error getting client from user: ", e);
}
However, it is not returning null, it's just throwing the exception and then stopping.
How can I get it to return null here so I can deal with the missing record without my code stopping?
Thanks in advance for any suggestions.
It seems that your database is missing a foreign key constraint.
This means that the table mapping User has a reference to a row in the table for Client that no longer exist.
This can only happen if a client has been deleted without updating the user table. The solution would be to add a foreign key constraint between the tables.
Keep in mind that if the data in your tables are not correct, when Hibernate loads the entity User, it will also believe there's a client. This means that User#getClient won't be null, and every place in the code where you have a check like user.getClient() == null is going to fail. A try-catch approach won't help you with this (unless you set the association to null in case of error, I guess).
The solutions I can think of:
Add the foreign key constraint (imho, the best solution)
Don't map the association, map client_id as an attribute and load the client using a second query or find (I would only do this if you cannot update the database)
class User {
#Column(name = "client_id")
Long clientId;
}
User user = ...
Client client = session.find(Client.class, user.getClientId());
You can load the client via session.find(Client.class, user.getClient().getId()) and set the association with the result:
User user = //...
Client client = session.find(Client.class, user.getClient().getId());
user.setClient(client);
Don't map the association at all in User, and run a native SQL query to load the client:
User user = ...
String sql = "select * from Client c join User u on c.id = u.client_id where u.id = :uid";
Client client = session.createNativeQuery(sql, Client.class)
.setParameter("uid", user.getId())
.getSingleResultOrNull();
You can pick what works best for you, but keep in mind that mapping an association without the foreign key constraint, will cause all sort of consistency issues.
I've decided to put option 3 only because, sometimes, people have some impossible situations at work, but I wouldn't recommend it.
I have two applications - Server that has connection with database and Client which doesn't. Server loads entity and sends it to Client. Client edits it and sends it back so that Server can update it. But... there are many Clients and many of them can edit that entity and want to update it with their own value but I want Server to update it only if it hasn't been updated since it had been send to Client. Namely: Clients try to set ticket's status to their own value but only if current status in database is the same that it was when that Client received it. Currently I have it like that:
public boolean changeStatus (Ticket ticket, short newStatus) {
short previousStatus = ticket.getStatus();
boolean result = false;
Session session = sessionFactory.openSession();
Transaction tx = null;
try {
tx = session.beginTransaction();
String sql = "UPDATE ticket SET status = :newStatus WHERE idTicket = :idTicket AND status = :previousStatus";
NativeQuery query = session.createSQLQuery(sql);
query.setParameter("idTicket", ticket.getIdTicket());
query.setParameter("newStatus", newStatus);
query.setParameter("previousStatus", previousStatus);
result = query.executeUpdate() == 1;
but sometimes I also want to set other atributes and check if other atributes hasn't changed and using that method would make it very ugly and I want it to be more object-oriented not plain sql. Any suggestions?
Forgot to mention that each Client has it's own Server thread that might try to update
When I execute:
select * from pg_stat_activity where state ~ 'idle in transact'
I get inappropriate number of rows with state 'idle in transaction'. Some of them idle for a few days. Most of them, are the same simple select query which are executed from one service class (Hibernate 5.1.0.Final, Guice 4.1.0):
public class FirebaseServiceImpl implements FirebaseService {
#Inject
private Provider<FirebaseKeyDAO> firebaseKeyDAO;
#Override
public void sendNotification(User recipient) {
List<FirebaseKey> firebaseKeys = firebaseKeyDAO.get().findByUserId(recipient.getId());
final ExecutorService notificationsPool = Executors.newFixedThreadPool(3);
for (FirebaseKey firebaseKey : firebaseKeys)
notificationsPool.execute(new Runnable() {
#Override
public void run() {
sendNotification(new FirebaseNotification(firebaseKey.getFirebaseKey(), "example");
}
});
notificationsPool.shutdown();
}
}
DAO method:
#Override
#SuppressWarnings("unchecked")
public List<FirebaseKey> findByUserId(Long userId) {
Criteria criteria = getSession().createCriteria(type);
criteria.add(Restrictions.eq("userId", userId));
return criteria.list();
}
Why does it happen? How to avoid this?
UPDATE
Transactions are not commited when I use Guice Provider exampleDAO.get() in a separate thread:
#Inject
Provider<ExampleDAO> exampleDAO;
It usually happens when you use pgbouncer or other pooler/session manager that uses pool_mode = transaction. Eg when client opens a transaction and holds it, not committing nor rolling back. Check if you see DISCARD ALL in query column - if you do this is the case, because pooler has to discard shared session plans, sequences, deallocate statements etc to avoid mixing those for different sessions in pool.
On the other hand any "normal" transaction gives same idle in transaction, eg:
2>select now(),pg_backend_pid();
now | pg_backend_pid
----------------------------------+----------------
2017-05-05 16:53:01.867444+05:30 | 26500
(1 row)
if we check its state we see orthodox idle:
t=# select query,state from pg_stat_activity where pid = 26500;
query | state
--------------------------------+-------
select now(),pg_backend_pid(); | idle
(1 row)
now we start transaction on session 2 >:
2>begin;
BEGIN
2>select now(),pg_backend_pid();
now | pg_backend_pid
----------------------------------+----------------
2017-05-05 16:54:15.856306+05:30 | 26500
(1 row)
and check pg_stat_statements gain:
t=# select query,state from pg_stat_activity where pid = 26500;
query | state
--------------------------------+---------------------
select now(),pg_backend_pid(); | idle in transaction
(1 row)
It will remain this way until statement timeout or end of transaction:
2>end;
COMMIT
t=# select query,state from pg_stat_activity where pid = 26500;
query | state
-------+-------
end; | idle
(1 row)
So it is quite common and ok to have it. If you want to avoid connected sessions, you have to disconnect client. But connection in postgres is expensive, so usually people try reuse existing connections with pool, and so such states appear in pg_stat_activity
Here is a simple hibernate code that inserts a value into a table.
If the row already exists, query the row and return the data.
Most of the time, the code works fine with no issues.
In a very special case, three different clients are trying to insert the exact the same row into the table. Ofcourse, only one row gets inserted. The other two insertions fail and the fall into the try-catch block.
There is a query in the try catch block, which queries the data and sends the value to the client. This results in an error for subsequent operations on the session.
Hibernate throws "ERROR org.hibernate.AssertionFailure - an assertion
failure occured (this may indicate a bug in Hibernate, but is more
likely due to unsafe use of the session)" in the logs.
Here is the code. What would be the right way to handle this scenario?
#Override
public void addPackage(PackageEntity pkg) {
try{
getCurrentSession().save(pkg);
getCurrentSession().flush();
}catch( ConstraintViolationException cve ){
// UNIQ constraint is violated
// query now, instead of insert
System.out.println("Querying again because of UNIQ constraint : "+ pkg);
PackageEntity p1 = getPackage(pkg.getName(), pkg.getVersion());
if( p1 == null ){
// something seriously wrong
throw new RuntimeException("Unable to query or insert " + pkg);
}else{
pkg.setId(p1.getId());
}
}catch (Exception e) {
e.printStackTrace();
}catch (Throwable t) {
t.printStackTrace();
}
}
Primary (or) composite Key makes each row data unique and avoids this error.
If you need the data from all these three requests then create a unique primary key in your table and add it to the entity.
Primary Key could be any unique thing from your data, an auto generated sequence or UUID/GUID.
I have a code looking like this:
Session session = sessionFactory.openSession();
Transaction tx = session.beginTransaction();
try {
for ( Customer customer: customers ) {
i++;
session.update(customer);
if ( i % 200 == 0 ) { //200, same as the JDBC batch size
//flush a batch of inserts and release memory:
session.flush();
session.clear();
}
}
} catch (Exc e) {
//TODO want to know customer id here!
}
tx.commit();
session.close();
Say, at some point session.flush() raises an DataException, because one of the fields did not map into the database column size, one of those batch of 200 customers. Nothing wrong with it, data can be corrupted, it's ok in this case. BUT, I really need to know the customer id which failed. Database returns meaningless error message, not stating what was the params of the statement, etc. Catched exception also does not contain which customer did fail, only the sql statement text, looking like 'update Customer set name=?'
Can I somehow determine it using the hibernate session? Does it store somewhere the information about last entity it tried to save down?