We need to generate sequential numbers for our transactions. We encountered sqlcode=-911, sqlstate=40001, sqlerrmc=2 (deadlock) when concurrent users are trying to book transactions at the same time. Currently deadlock occurs because it is reading and updating to the same record. How can we design this so that deadlock can be prevented?
Create a "seed" table that contains a single data row.
This "seed" table row holds the "Next Sequential" value.
When you wish to insert a new business data row using the "Next Sequential" value. perform the following steps.
1). Open a cursor for UPDATE on the "seed" table and fetch the current row. This gives you exclusive control over the seed value.
2). You will employ this fetched row as the "Next Value"... However before doing so
3). Increment the fetched "Next Value" and commit the update. This commit closes your cursor and releases the seed row with the new "Next Value".
you are now free to employ your "Next Value".
There are a number of ways around this issue, some less performant than others.
Deadlocks can be prevented if all objects are locked in the same hierarchical sequence. [https://en.wikipedia.org/wiki/Deadlock#Prevention]
However, solutions to the Dining Philosophers Problem [https://en.wikipedia.org/wiki/Dining_philosophers_problem] which completely prevent deadlocks are often less performant than simply rolling back the transaction and retrying. You'll need to test your solution.
If you're looking for a data-side solution, an old-fashioned (and potentially under-performant) approach is to force the acquisition of new transaction IDs to be atomic by establishing a rigorous lock sequence.
A quick-ish solution (test this under load before releasing to production!) could be to use TRANSACTION boundaries and have a control row acting as a gatekeeper. Here's a stupid example which demonstrates the basic technique.
It has no error checking, and the code to reclaim ghost IDs is outside the scope of this example:
DECLARE #NewID INTEGER;
BEGIN TRANSACTION;
UPDATE [TABLE] SET [LOCKFLAG] = CURRENT_TIMESTAMP WHERE [ROW_ID] = 0;
SELECT #NewID = MAX([ROW_ID])+1 FROM [TABLE];
INSERT INTO [TABLE] ([ROW_ID]) VALUES (#NewID);
UPDATE [TABLE] SET [LOCKFLAG] = NULL WHERE [ROW_ID] = 0;
COMMIT TRANSACTION;
The idea is to make this atomic, single-threaded, serialized operation very, very short in duration -- do only what is needed to safely reserve the ID and get out of the way.
By making the first step to update row 0, if all ID requests comply with this standard, then competing users will simply queue up behind that first step.
Once you have your ID reserved, go off and do what you like, and you can use a new transaction to update the row you've created.
You'd need to cover cases where the later steps decide to ROLLBACK, as there would now be a ghost row in the table. You'd want a way to reclaim those; a variety of simple solutions can be used.
Related
I have a PostgreSQL table that has millions of record. I need to process every row and for that I am using a column in that table namely 'isProcessed' so by default it's false and when I process it I change it to true.
Now the problem is that there are too many records and due to exceptions code bypasses some records leaving them isProcessed=false and that makes the execution really slow.
I was thinking to use indexing but with boolean it does not help.
Please provide some optimization technique or some better practice.
UPDATE:
I don't have the code, It just a problem my colleagues were asking for my opinion.
Normally an index on a Boolean isn't a good idea, but in PostgreSQL you can do an index where it contains only entries for one value using a partial index http://www.postgresql.org/docs/9.3/interactive/indexes-partial.html. It ends up being a queue of things for you to process, items drop off once done.
CREATE INDEX "yourtable_isProcessed_idx" ON "public"."yourtable"
USING btree ("isProcessed")
WHERE (isProcessed IS NOT TRUE);
This will make life easier when it is looking for the next item to process. Ideally you should be processing more than one at a time, particularly if you can do it in a single query, though doing millions at once may be prohibitive. In that situation, you might be able to do
update yourtable
set ....
where id in (select id from yourtable where isProcessed = false limit 100 )
If you have to do things one at a time, I'd still limit what you retrieve, so potentially retrieve
select id from yourtable where iProcessed = false limit 1
Without seeing your code, it would be tough to say what is really going on. Doing any processing row by row, which it sounds like is what is going on, is going to take a VERY long time.
Generally speaking, the best way to work with data is in sets. At the end of your process, you're going to ultimately have a set of records where isProcessed needs to be true (where the operation was successful), and a set where isProcessed needs to be false (where the operation failed). As you process the data, keep track of which records could be updated successfully, as well as which could not be updated. You could do this by making a list or array of the primary key or whatever other data you use to identify the rows. Then, after you're done processing your data, do one update to flag the records that were successful, and one to update the records that were not successful. This will be a bit more code, but updating each row individually after you process it is going to be awfully slow.
Again, seeing code would help, but if you're updating each record after you process it, I suspect this is what's slowing you down.
Here is approach I use. You should be able to store processing state including errors. It can be one column with values PENDING, PROCESSED, ERROR or two columns is_processed, is_error.
This is to be able skip records which couldn't be successfully processed and which if not skipped slow down processing of good tasks. You may try to reprocess them later or give DevOps possibility to move tasks from ERROR to PENDING state if the reason for failure for example was temporary unavailable resource.
Then you create conditional index on the table which include only PENDING tasks.
Processing is done using following algorithm (using spring: transaction and nestedTransaction are spring transaction templates):
while (!(batch = getNextBatch()).isEmpty()):
transaction.execute( (TransactionStatus status) -> {
for (Element element : batch) {
try {
nestedTransaction.execute( (TransactionStatuc status ) -> {
processElement(element);
markAsProcessed(element);
});
} catch (Exception e) {
markAsFailed(element);
}
}
});
Several important notes:
getting of records is done in batches - this at least saves round trips to database and is quicker then one by one retrieval
processing of individual elements is done in nested transaction (this is implemented using postgresql SAVEPOINTs). This is quicker then processing each element in own transaction but have the benefit that failure in processing of one element will not lose results of processing of others elements in batch.
This is good when processing is complex enough and cannot be done in sql by single query to process batch. If processElement rather simple update of element then whole batch may be updated via single update statement.
processing on elements of the batch may be done in parallel. This requires propagation of transaction to worker threads.
I have a multi-threaded client/server system with thousands of clients continuously sending data to the server that is stored in a specific table. This data is only important for a few days, so it's deleted afterwards.
The server is written in J2SE, database is MySQL and my table uses InnoDB engine. It contains some millions of entries (and is indexed properly for the usage).
One scheduled thread is running once a day to delete old entries. This thread could take a large amount of time for deleting, because the number of rows to delete could be very large (some millions of rows).
On my specific system deletion of 2.5 million rows would take about 3 minutes.
The inserting threads (and reading threads) get a timeout error telling me
Lock wait timeout exceeded; try restarting transaction
How can I simply get that state from my Java code? I would prefer handling the situation on my own instead of waiting. But the more important point is, how to prevent that situation?
Could I use
conn.setIsolationLevel( Connection.TRANSACTION_READ_UNCOMMITTED )
for the reading threads, so they will get their information regardless if it is most currently accurate (which is absolutely OK for this usecase)?
What can I do to my inserting threads to prevent blocking? They purely insert data into the table (primary key is the tuple userid, servertimemillis).
Should I change my deletion thread? It is purely deleting data for the tuple userid, greater than specialtimestamp.
Edit:
When reading the MySQL documentation, I wonder if I cannot simply define the connection for inserting and deleting rows with
conn.setIsolationLevel( Connection.TRANSACTION_READ_COMMITTED )
and achieve what I need. It says that UPDATE- and DELETE statements, that use a unique index with a unique search pattern only lock the matching index entry, but not the gap before and with that, rows can still be inserted into that gap. It would be great to get your experience on that, since I can't simply try it on production - and it is a big effort to simulate it on test environment.
Try in your deletion thread to first load the IDs of the records to be deleted and then delete one at a time, committing after each delete.
If you run the thread that does the huge delete once a day and it takes 3 minutes, you can split it to smaller transactions that delete a small number of records, and still manage to get it done fast enough.
A better solution :
First of all. Any solution you try must be tested prior to deployment in production. Especially a solution suggested by some random person on some random web site.
Now, here's the solution I suggest (making some assumptions regarding your table structure and indices, since you didn't specify them):
Alter your table. It's not recommended to have a primary key of multiple columns in InnoDB, especially in large tables (since the primary key is included automatically in any other indices). See the answer to this question for more reasons. You should add some unique RecordID column as primary key (I'd recommend a long identifier, or BIGINT in MySQL).
Select the rows for deletion - execute "SELECT RecordID FROM YourTable where ServerTimeMillis < ?".
Commit (to release the lock on the ServerTimeMillis index, which I assume you have, quickly)
For each RecordID, execute "DELETE FROM YourTable WHERE RecordID = ?"
Commit after each record or after each X records (I'm not sure whether that would make much difference). Perhaps even one Commit at the end of the DELETE commands will suffice, since with my suggested new logic, only the deleted rows should be locked.
As for changing the isolation level. I don't think you have to do it. I can't suggest whether you can do it or not, since I don't know the logic of your server, and how it will be affected by such a change.
You can try to replace your one huge DELETE with multiple shorter DELETE ... LIMIT n with n being determined after testing (not too small to cause many queries and not too large to cause long locks). Since the locks would last for a few ms (or seconds, depending on your n) you could let the delete thread run continuously (provided it can keep-up; again n can be adjusted so it can keep-up).
Also, table partitioning can help.
I have to create a MySQL InnoDB table using a strictly sequential ID to each element in the table (row). There cannot be any gap in the IDs - each element has to have a different ID and they HAVE TO be sequentially assigned. Concurrent users create data on this table.
I have experienced MySQL "auto-increment" behaviour where if a transaction fails, the PK number is not used, leaving a gap. I have read online complicated solutions that did not convince me and some other that dont really address my problem (Emulate auto-increment in MySQL/InnoDB, Setting manual increment value on synchronized mysql servers)
I want to maximise writing concurrency. I cant afford having users writing on the table and waiting long times.
I might need to shard the table... but still keeping the ID count.
The sequence of the elements in the table is NOT important, but the IDs have to be sequential (ie, if an element is created before another does not need to have a lower ID, but gaps between IDs are not allowed).
The only solution I can think of is to use an additional COUNTER table to keep the count. Then create the element in the table with an empty "ID" (not PK) and then lock the COUNTER table, get the number, write it on the element, increase the number, unlock the table. I think this will work fine but has an obvious bottle neck: during the time of locking nobody is able to write any ID.
Also, is a single point of failure if the node holding the table is not available. I could create a "master-master"? replication but I am not sure if this way I take the risk of using an out-of-date ID counter (I have never used replication).
Thanks.
I am sorry to say this, but allowing high concurrency to achieve high performance and at the same time asking for a strictly monotone sequence are conflicting requirements.
Either you have a single point of control/failure that issues the IDs and makes sure there are neither duplicates nor is one skipped, or you will have to accept the chance of one or both of these situations.
As you have stated, there are attempts to circumvent this kind of problem, but in the end you will always find that you need to make a tradeoff between speed and correctness, because as soon as you allow concurrency you can run into split-brain situations or race-conditions.
Maybe a strictly monotone sequence would be ok for each of possibly many servers/databases/tables?
I have this scenario ,on a airline website (using Java) two separate customers send two requests at same time to book a same seat in same airline
from New York to Chicago. I am using the oracle database and isolation level is read committed.My question here is that does oracle database provide
any solution to deal with this kind of concurrent scenario? what I know is when first transaction DML statement is fired it will get a lock on affected
rows and will release when transaction completes i.e on issuing rollback or commit.But as soon as commit is done and second request will proceed as soon as
first is completed and will override the first one.So it does not help?
Yes in Java I can deal with making my db class as singleton and using synchronized keyword on method which is doing update. But want to know is there
anyway we can this kind of issue at database level itself?Probably isolation level as serializable can help. But not sure?
It will only over write if you allow it. You can try something like
UPDATE seatTable
SET seatTaken = true
WHERE .. find the seat, flight etc.. AND seatTaken = false
This will return 1 row updated the first time and 0 rows updated after that.
As you mention, transanction settings will help you achieving one operation. The best way to enforce this kind of restrictions it to ensure that your relational model is constrained not to accept the 2nd operation once the 1st one succeeds.
Instead of having to do an update on a row, say update .... seat = "taken", create a reservation table (customer, flight, seat) which has a constrain (column:seat = unique) (lookup ora docs to learn the syntax for that on table creation). That way your reservation process becomes an insert in the reservation table and you can rely on the RDBMS to enforce your relational constrains to keep your business model valid.
e.g. Let t1 be the earlier operation time, you'll have:
t1=> insert into reservations(customer1,flight-x,seat-y) // succeeds. Customer 1 reserved the seat-y
t2=> insert into reservations(customer2,flight-x,seat-y) // fails with RDBMS unique constrain violated.
The only way to reserve seat-y again is to first remove the previous reservation, which is probably what your business process wants to achieve.
To handle concurrency in a web site a common practice it to have a column on each record that allows you to check it has not been updated since you got it. Either last update date or a sequential version number (auto incremented by a trigger).
Typically you will read the data (plus the concurrency column)
SELECT seat,etc,version_no
FROM t1
WHERE column = a_value
Then when the user eventually gets round to booking the seat the update will work unless there has been an update.
(the version number or update date will change after every update)
BEGIN
UPDATE t1
SET seatTaken = true
WHERE seatid = .....
AND version_no = p_version
RETURNING version_no INTO p_version;
EXCEPTION WHEN NOT_FOUND THEN
--Generate a custom exception
--concurrency viloation the record has been updated already
END;
the trigger to auto update the version number would look a little like this
CREATE OR REPLACE TRIGGER t1_version
AFTER INSERT OR UPDATE ON t1
FOR EACH ROW
BEGIN
IF :new.version_no IS NULL THEN
:new.version_no := 0;
ELSE
:new.version_no := :old.version_no + 1;
END IF;
END;
Aside from doing everything in a single UPDATE by carefully crafting WHERE clause, you can do this:
Transaction 1:
SELECT ... FOR UPDATE exclusively locks the row for the duration of the transaction.
Check if the returned status of the row is "booked" and exit (or retry another row) if it is.
UPDATE the row and set its "status" to "booked" - it is guaranteed nobody else updated it in the meantime.
Commit. This removes the exclusive lock.
Transaction 2:
SELECT ... FOR UPDATE blocks until Transaction 1 finishes, then exclusively locks the row.
The returned status of the row is "booked" (since Transaction 1 marked it that way), so exit (or possibly retry another row).
I encountered some curious behavior today and was wondering if it is expected or standard. We are using Hibernate against MySQL5. During the course of coding I forgot to close a transaction, I presume others can relate.
When I finally closed the transaction, ran the code and checked the table, I noticed the following. All the times I mistakenly ran my code without closing the transaction, which therefore did not result in actual rows being inserted, nevertheless incremented the auto-increment surrogate primary key value, so that I have a gap (i.e. no rows with id field value of 751 to 762).
Is this expected or standard behavior? Might it vary depending on the database? And/or does Hibernate's own transaction abstraction have some possible effect on this?
Yes that's expected.
If you think about it: what else can the database do? If you increment the column and then use that as a foreign key in other inserts within the same transaction and while you're doing that someone else commits then they can't use your value. You'll get a gap.
Sequences in databases like Oracle work much the same way. Once a particular value is requested, whether or not it's then committed doesn't matter. It'll never be reused. And sequences are loosely not absolutely ordered too.
It's pretty much expected behaviour. With out it the db would have to wait for each transaction that has inserted a record to complete before assigning a new id to the next insert.
Yes, this is expected behaviour. This documentation explains it very well.
Beginning with 5.1.22, there are actually three different lock modes that control how concurrent transactions get auto-increment values. But all three will cause gaps for rolled-back transactions (auto-increment values used by the rolled-back transaction will be thrown away).
Database sequences are not to guarantee id sequence without gaps. They are designed to be transaction-independent, only in such way can be non-blocking.
You want no gaps, you must write your own stored procedure to increase column transactionally, but such code will block other transactions, so you must be carrefull.
You do SELECT CURRVAL FROM SEQUENCE_TABLE WHERE TYPE = :YOUR_SEQ_NAME FOR UPDATE;
UPDATE SEQUENCE_TABLE SET CURRVAL = :INCREMENTED_CURRVAL WHERE TYPE = :YOUR_SEQ.