We're using quartz for scheduling jobs with MariaDB on beneath in a setup of a few nodes. We're also using it as a queue system. The most important reason for that is that we already have quartz in our service, and we do not have any queue just yet.
We're getting a lot of requests, very often at the same time with the same id of a business entity, which is used to generate job name for the sake of uniqueness.
try {
quartzServce.scheduleJob(job, trigger);
log.info("job: {} has been scheduled", id);
} catch (ObjectAlreadyExistsException ex) {
log.warn(DUPLICATE_ENTRY_MESSAGE, job.getKey(), trigger.getKey());
} catch (Exception ex) {
throw new RuntimeException(ex);
}
the quartzService.scheduleJobs is simply:
public void scheduleJob(JobDetail jobDetail, Trigger trigger) throws SchedulerException {
schedulerFactory.getScheduler().scheduleJob(jobDetail, trigger);
}
As you can see we're catching ObjectAlreadyExistsException and silence it down on a Warn level, as we do not treat it as errors, but from time to time we still get SQLIntegrityConstraintViolationException wrapped in JobPersistenceException with a message:
Couldn't store job: (conn=435654) Duplicate entry '{our key is over here}' for key 'PRIMARY' [See nested exception: java.sql.SQLIntegrityConstraintViolationException: (conn=435654) Duplicate entry '{our key is over here}' for key 'PRIMARY']
What I assume is that between existence check, and the actual insert the other node manages to insert a row for the same id.
As I'm not really a fan of checking the message in an exception for something like "Duplicated entry" and silence the exception down with a warn level on that condition, I'm looking for another solution, maybe quartz configuration?
You can try checking if the job exists or not before calling quartzServce.scheduleJob.
Using
scheduler.checkExists(...)
Not sure if thats what you were looking for.
But it sounds like you are gonna need a lock of some kind on this business entity id. So only one node at a time tries to schedule it.
You can save the distinct entity ids in a table before scheduling jobs for them. And let your nodes pick them up with a lock so that particular entity id is not visible to other nodes.
Related
I try to build a booking portal. A booking has a checkin datetime and a checkout datetime. The Spring Boot application will run in many replicas.
My problem is to ensure that there is no overlapping possible.
First of all here is my respository method to check if the time is blocked:
#Lock(LockModeType.PESSIMISTIC_READ)
#QueryHints({#QueryHint(name = "javax.persistence.lock.timeout", value = "1000")})
Optional<BookingEntity> findFirstByRoomIdAndCheckInBeforeAndCheckOutAfter(Long roomId, LocalDateTime checkIn, LocalDateTime checkOut);
As you can see I am using findFirstBy and not findBy because the request could get more than 1 result.
With this i can check if this time is blocked (please notice in the call i switch the requested checkin and requested checkout:
findFirstByWorkplaceIdAndCheckInBeforeAndCheckOutAfter(workplaceId, requestCheckOut, requestCheckIn).isPresent();
And everything is happen in the controller:
#PostMapping("")
#Transactional
public void myController(LocalDateTime checkIn, LocalDateTime checkOut, long roomId) {
try {
if (myService.bookingBlocked(checkIn, checkOut, roomId)) {
Log.warning("Booking blocked");
return "no!";
}
bookingService.createBooking(checkIn, checkOut, roomId);
return "good job";
} catch (Exception exception) {
return "something went wrong";
}
}
Everything works fine but I can simulate an overlapping booking if i set a breakpoint after the bookingBlocked-check in two replicas. Following happens:
Replica 1 checks if the booking free (its ok)
Replica 2 checks if the booking free (its ok)
Replica 1 creates new entity
Replica 2 creates new entity
Now I have a overlapping booking.
My idea was to create a #Constraint but this does not possible with hibernate in MySql dialect. Than I tried to create a #Check but this seems also to be impossible.
Now my idea was to create a Lock with Transaction (is already in the code at the top). This seems to work but I am not sure if I have this implemented correct. If I try the same as before with the breakpoints following happen:
Replica 1 checks if the booking free (its ok)
Replica 2 checks if the booking free (its ok)
Replica 1 creates new entity
Replica 2 creates new entity (throws exception and silent rollbacked)
Controller returns a 500.
I am not sure where the exception is thrown. I am not sure what happens when the replica is shutdown durring the lock. I am not sure if i can create a deadlock in the database. And it is still possible to manipulate the database via sql query to create a overlapping.
Could anybody say me if this the correct way? Is there a possibility to create a constraint on database with hibernate (i dont use migration scripts and only for one constraint it will be not cool)? Should I use optimistic locking? Every idea is welcome.
I have a somewhat strange situation which I need to deal with, but can't seem to find a solution.
I need to solve a potential race condition on a customer insertion. We receive the customers through a topic, so they come with an id(we keep it because it's the same id we have in a different database for a different microservice). So, if by some chance, after the same customer is committed to the database before the flush operation is actioned, we should update the record in the database with the one that arrived through the topic, if the last activity field on that one is after the last activity field on the db entry.
The problem we encounter is that, while the flush option is recognizes the newly committed consumer and throws the ConstraintViolationException, when it gets to the find line it returns the customer we try to persist above, not the customer in the database
The code breaks down like this.
try{
entityManager.persist(customer);
//at this point, I insert a new customer in the database with the same id as the one I've persisted
entityManager.flush();
}catch(PersistenceException e){
if(e.getCause() instanceof ConstraintViolationException) {
dbCustomer = Optional.of(entityManager.find(Customer.class,
customer.getId()));
//update DB Customer with data from persisted customer if the last update date on the persisted customer is after the one on the db customer
}
}
I tried different options of transaction propagation, with no success, however, and to use the detach(customer) method before trying to find the db customer, however, in this case, the find function returns Null
Thanks
As soon as a flush fails, the persistence context is essentially broken. If you need to do something with the result of this code block that needs flushing, you need to do that in a new transaction in case of a constraint violation.
I'm working on my personal project, and there I observe a strange behaviour in spring CrudRepository.save. There is a unique constraint in one of the field. When I save a new record with a duplicate value for this field, I didn't get any exception until the request handler method completes. Is this a normal behaviour?
DB is Postgres
RuleSet save = ruleSetRepository.save(convertToRuleSet(request));
fileRepository.createRuleSet(request);
try {
gitRepository.commitAddPush(request.getRuleSetName(), "Added rule set " + request.getRuleSetName(), gitVersion);
} catch (GenericGitException gitException) {
fileRepository.deleteClassDirectory(request.getRuleSetName());
fileRepository.deleteRuleSet(request.getRuleSetName());
throw new CommonRuleCreateException(gitException.getMessage());
}
return new RuleSetResponse(save.getId(), save.getName(), save.getDescription(),save.getPackageName());
This entire method get called without any exception.
What you might be missing is that save method will commit to DB after transaction is completed, generally end of method execution. If you want to save to DB at that time only, use saveAndFlush.
Also if you want, you can make sure that your repo methods are using a new transactions and not same as that of its caller methods. So that when repo method call is completed, it will save transaction data into DB.
I am using spring, hibernate and postgreSQL.
Let's say I have a table looking like this:
CREATE TABLE test
(
id integer NOT NULL
name character(10)
CONSTRAINT test_unique UNIQUE (id)
)
So always when I am inserting record the attribute id should be unique
I would like to know what is better way to insert new record (in my spring java app):
1) Check if record with given id exists and if it doesn't insert record, something like this:
if(testDao.find(id) == null) {
Test test = new Test(Integer id, String name);
testeDao.create(test);
}
2) Call straight create method and wait if it will throw DataAccessException...
Test test = new Test(Integer id, String name);
try{
testeDao.create(test);
}
catch(DataAccessException e){
System.out.println("Error inserting record");
}
I consider the 1st way appropriate but it means more processing for DB. What is your opinion?
Thank you in advance for any advice.
Option (2) is subject to a race condition, where a concurrent session could create the record between checking for it and inserting it. This window is longer than you might expect because the record might be already inserted by another transaction, but not yet committed.
Option (1) is better, but will result in a lot of noise in the PostgreSQL error logs.
The best way is to use PostgreSQL 9.5's INSERT ... ON CONFLICT ... support to do a reliable, race-condition-free insert-if-not-exists operation.
On older versions you can use a loop in plpgsql.
Both those options require use of native queries, of course.
Depends on the source of your ID. If you generate it yourself you can assert uniqueness and rely on catching an exception, e.g. http://docs.oracle.com/javase/1.5.0/docs/api/java/util/UUID.html
Another way would be to let Postgres generate the ID using the SERIAL data type
http://www.postgresql.org/docs/8.1/interactive/datatype.html#DATATYPE-SERIAL
If you have to take over from an untrusted source, do the prior check.
I have an entity named Message with fields: id (PK), String messageXML and Timestamp date. and simple dao to store object into Oracle Database (11g) / MyBatis
Code looks like something like that:
Sevice:
void process throws ProcessException {
Message message = wrapper.getMessage(request);
Long messageId;
try {
messageId = (Long) dao.save(message);
} catch (DaoException e) {
throw ProcessException(e);
}
Dao
private String mapperName = "messageMapper";
Serializable save(Message message) throws DaoException {
try {
getSqlSession().insert(mapperName + ".insert", message);
return message.getPrimaryKey();
} catch (Exception e) {
throw DaoException(e);
}
Simple code. Unfortunately, load of this method process(req) is about 500 req / sec. and sometimes I get a lock on DB during saving message.
To resolve that problem I thought about multiplication table Message, for instance I will be have five table Message1, Message2 ... Message 5 and during saving entity Message i will be drawing (like a round robin algorithm) table - for instance:
private Random generator;
public MessageDao() {
this.generator = new Random();
Serializable save(Message message) throws DaoException {
try {
getSqlSession().insert(getMapperName() + ".insert", message);
return message.getPrimaryKey();
} catch (Exception e) {
throw DaoException(e);
}
private String getMapperName() {
return this.mapperName.concat(String.valueOf(generator.nextInt(5))); //could be more effeciency of course
}
What are you thinking about this solution? Could be efficiently? How can I make that better? Where could I make bottleneck?
Reading between the lines, I guess you have a number of instances of code running serving multiple concurrent requests, hence why you are getting the contention. Or you have 1 server that is firing 500 requests per second and you experience waits. Not sue which of these you mean. In the former case, you might want to look extent allocation - if the table/index next extent sizes are small you will see regularly latency when Oracle grabs the next extent. Size too small and you will get this latency very regularly, size big and when it does eventually run out the wait will be longer. You could do something like calculate the storage per week, and have a weekly procedure to "Grow" the table/indexes accordingly to avoid this during operation hours. I would be tempted to examine the stats and see what the waits are.
If however the cause is concurrency (maybe in addition to extent management), then you're probably getting hot-block contention on the index used to enforce the PK constraints. Typical strategies to mitigate this include REVERSE index (no code change required), or more controversially use partitioning with a weaker unique constraint by adding a simple column to further segregate the concurrent sessions. E.g. add a column serverId to the table and partition by this and the existing PK column. Assign each application server a unique serverId (config/startup file). Amend the insert to include the serverID. Have 1 partition per server. Controversial because the constraint is weaker (down to how partitions work), and this will be an anathema to purists, but this is something I've used on projects with Oracle Consulting to maximise performance on Exadata. So, it's out there. Of course, partitions can be thought of as distinct tables grouped into a super table, so your idea of writing to separate tables is not a million miles from what is being suggested here. The advantage with partitions it is a more natural mechanism for group this data, and adding a new partition will require less work than adding a new table when expanded.