We use Sybase ASE (15.5) server as our DB and are having strange, intermittent SPID blocking issues that I am trying to detect and mitigate programmatically at the application-layer.
Sybase allows you to schedule so-called "reorgs" which from what I can tell are periodic re-indexes/table compactions, cleanups, etc. Scheduled DB maintenance, basically.
Every once in a while, we get all the planets coming into alignment with each other, where:
A query is executed (creating a SPID in Sybase) and hangs for some reason. This places a (blocking) shared lock on, say, the widgets table; then
The scheduled reorg kicks off, and wants to cleanup the widgets table. The reorg places an exclusive lock request on widgets, but can't obtain the lock because widgets is already locked and blocked by the hanging SPID/query; then
Subsequent queries are executed, each requesting shared locks on widgets; such that
The whole system is now tied up: the reorg can't start until it obtains an exclusive lock on widgets, but widgets is tied up in a blocking shared lock by a hung SPID. And because the reorg has placed an exclusive lock on widgets, all other queries wanting shared locks on widgets have to wait until the reorg is complete (because a newly requested exclusive lock trumps a newly requested shared lock).
I think my ideal strategy here would be to:
Timeout DB queries after say, 2 minutes, which will prevent SPIDs from hanging and thus preventing the reorgs from running; and then
If a query attempts to hit a table that has an exclusive lock on it, detect this and hadle it specially (like schedule the query to run again 1hr later, when hopefully the reorg is complete, etc.)
My questions:
How do I timeout a query to release a shared lock after, say, 2mins?
Is there a way to programmatically (most likely through the Sybase JDBC driver, but perhaps via Sybase command-line, HTTP calls, etc.) determine if a reorg is running? Or, that an exclusive lock exists on a table? That way I could detect the exclusive lock and handle it in a special way.
Thanks in advance!
You can get the commands that are running in the database with the following query:
select cmd from sysprocesses
To find locking information you can join master..syslocks and your_db..sysobjects to find out what locks exist on the object you are trying to access. syslocks.type indicates the kind of lock that is in place, and those possible values can be found here:
select object_name(id), db_name(dbid), type from master..syslocks
where dbid = db_id("your_db")
http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.infocenter.dc36274.1550/html/tables/X16427.htm
Hopefully that helps.
To find if the lock is tied to a reorg, I think you should be able to join syslocks.spid to sysprocesses.spid where cmd = "REORG" or something like:
select p.cmd, p.spid, l.type from master..sysprocesses p, master..syslocks l where CMD = "REORG"
Related
I have a situation where we are acquiring lock on an object from database using SELECT FOR UPDATE. This is necessary for us to insert and delete records from multiple tables in an orderly fashion. The functionality works something like this.
Login -> Acquire lock on unique lock object and insert records to multiple tables and release lock -> Logout -> Acquire lock on same unique lock object and delete records from multiple tables and release lock.
We have synchronization enabled to track users have logged in before logging him out. It is taken care in Java code. However we obtain another lock at database level to make sure the database transactions are synchronized when large number of users are logging in.
Problem: The whole system works perfectly in multi-clustered servers and singleton servers. However, when the number of concurrent users reaches 4000+, we are facing row lock contention (Mode 6) in the database. And few users are not able to login.
Objective: To fix the locking mechanism to enable users to login and logout successfully.
Things tried so far: Added NOWAIT and SKIP LOCKED to SELECT FOR UPDATE query. This doesn't solve my problem because the first one simply throws an error and the second one basically skips the lock which would affect synchronization.
Need suggestions and opinions from Database experts to resolve this issue. TIA.
UPDATE: Just adding one more information. We do not update or do anything with the locked row. It is just used as a mechanism to synchronize other database tasks we do.
Instead of relying on pessimistic locking(your current approach)- use optimistic locking possibly using some ORM.
Developing a Java application that share a single Connection between mutiple threads, the problem of concurrency arise.
If thread A updates record 1 in table T, and simultaneously thread B issues a SELECT on record 1 in table T, how do I ensure thread B reads the updated values of thread A?
java.sql.Connection offers transactions with begin(), commit() and rollback(), but does this process also cover data correctness?
I think I'm missing something.
Two points:
You shouldn't share a jdbc.Connection between threads, at least for any 'seriously production' code, see here. For demo purposes, I think, sharing a Connection is OK;
If a thread reads from DB after relevant DB transaction is committed, it will see data written by another thread.
For your second question
will thread B timeout until the first transaction has commit() or rollback()
-- B will block till A tx is finished (either by commit or rollback) if:
B tries to update/delete same table row which is being updated by A, and ...
A updates that row under DB-level lock, using SELECT ... FOR UPDATE.
You can get this behavior using two consoles (for example, with PostgreSQL psql), each console stands for a thread:
in A console type following:
BEGIN;
SELECT some_col FROM some_tbl WHERE some_col = some_val FOR UPDATE;
now in B console type:
BEGIN;
UPDATE some_tbl SET some_col = new_val WHERE some_col = some_val;
You should see that UPDATE blocks until in A you do either COMMIT or ROLLBACK.
Above explanation uses separate DB connections, just like Java JDBC connection pool. When you share single connection between Java threads, I think, any interaction with DB will block if connection is used by some other thread.
Jdbc is a standard that is broadly adopted but with uneven levels of adherence, it is probably not good to make sweeping statements about what is safe.
I would not expect there is anything to keep statement executions and commits and rollbacks made from multiple threads from getting interleaved. Best case, only one thread can use the connection at a time and the others block, making multithreading useless.
If you don't want to provide a connection to each thread, you could have the threads submit work items to a queue that is consumed by a single worker thread handling all the jdbc work. But it's probably less impact on existing code to introduce a connection pool.
In general if you have concurrent updates and reads then they happen in the order that they happen. Locking and isolation levels provide consistency guarantees for concurrent transactions but if one hasn't started its transaction yet those aren't applicable. You could have a status flag, version number, or time stamp on each row to indicate when an update occurred.
If you have a lot of updates it can be better to collect them in a flat file and execute a bulk copy. It can be much faster than using jdbc. Then with updates out of the way execute selects in jdbc.
I have a J2EE server, currently running only one thread (the problem arises even within one single request) to save its internal model of data to MySQL/INNODB-tables.
Basic idea is to read data from flat files, do a lot of calculation and then write the result to MySQL. Read another set of flat files for the next day and repeat with step 1. As only a minor part of the rows change, I use a recordset of already written rows, compare to the current result in memory and then update/insert it correspondingly (no delete, just setting a deletedFlag).
Problem: Despite a purely sequential process I get lock timeout errors (#1204) and Innodump show record locks (though I do not know how to figure the details). To complicate things under my windows machine everything works, while the production system (where I can't install innotop) has some record locks.
To the critical code:
Read data and calculate (works)
Get Connection from Tomcat Pool and set to autocommit=false
Use Statement to issue "LOCK TABLES order WRITE"
Open Recordset (Updateable) on table order
For each row in Recordset --> if difference, update from in-memory-object
For objects not yet in the database --> Insert data
Commit Connection, Close Connection
The Steps 5/6 have an Commitcounter so that every 500 changes the rows are committed (to avoid having 50.000 rows uncommitted). In the first run (so w/o any locks) this takes max. 30sec / table.
As stated above right now I avoid any other interaction with the database, but it in future other processes (user requests) might read data or even write some fields. I would not mind for those processes to read either old/new data and to wait for a couple of minutes to save changes to the db (that is for a lock).
I would be happy to any recommendation to do better than that.
Summary: Complex code calculates in-memory objects which are to be synchronized with database. This sync currently seems to lock itself despite the fact that it sequentially locks, changes unlocks the tables without any exceptions thrown. But for some reason row locks seem to remain.
Kind regards
Additional information:
Mysql: show processlist lists no active connections (all asleep or alternatively waiting for table locks on table order) while "show engine INNODB" reports a number of row locks (unfortuantely I can't understand which transaction is meant as output is quite cryptic).
Solved: I wrongly declared a ResultSet as updateable. The ResultSet was closed only on a "finalize()" method via Garbage Collector which was not fast enough - before I reopended the ResultSet and tried therefore to aquire a lock on an already locked table.
Yet it was odd, that innotop showed another query of mine to hang on a completely different table. Though as it works for me, I do not care about oddities:-)
Our app is mostly using optimistic locking using Hibernate’ s versioning support. We are planning to implement pessimistic locking in one particular scenario. I don’t have much experience with pessimistic locking so please excuse if this question sounds naïve.
When a user shows intention for updating an entry - we lock the corresponding DB row using “select for update”. Now, if this user takes a long time to commit his changes are forgets about it after locking, how do we unlock this lock using some timeout/rollback mechanism? So that the row doesn’t stays locked for a very long time and disallowing all other users to edit it.
I doubt if this will be handled at Weblogic-JTA-Spring transaction mechanism we are using – where we already have a transaction timeout of 30 mins. (??)
So, should this rollback be handled directly at Oracle level. If yes, then how? Please advise on best way to handle this so that such locks don’t stay lingering around for too long.
Locks will be released only when the transaction ends. The transaction will end either when an explicit commit or rollback is issued to the database or when the database session is terminated (which does an implicit rollback). If your middle tier is already set to rollback any transactions that are open for more than 30 minutes, that would be sufficient to release the locks.
If you have a Java application running in a Weblogic application server, however, it strikes me as unusual for pessimistic locking to be appropriate. First, I assume that you are using a connection pool in the middle tier. If that is the case, then one database connection from the connection pool would need to be held by the middle tier for the length of the transaction (up to 30 minutes in this case). But allowing one session to hold open a particular database session for an extended period of time defeats the purpose of having a connection pool. Normally, dozens if not hundreds of application sessions can share a single connection from the connection pool-- if you are going to allow pessimistic locking, you're now forcing a 1:1 relationship between application sessions and database sessions for those sessions.
There are numerous cases that optimistic locking cannot replace pessimistic locking. Lock timeout is handled in the database. Refer to this page about how to configure it in Oracle
Can Oracle's default object lock timeout be changed?
I have a Java frontend and a MySQL backend scenario, I used a 'LOCK IN SHARE MODE' for SELECT. If I request the same row from another process, it gives the data.. However it does not allow me to update. What I would like to do is inform the user they will only have a READ only copy, so if they wish to see the information they can else they can request it later.. How could I check the status of the ROW so that the user will be informed about this situation?? If I use 'FOR UPDATE', It just waits for until the first user saves the data. I find it less user friendly, if they just have a blank screen or when they click button it does nothing. Any help will be greatly appreciated. Using MySQL 5.5, Java 7.
The short answer is "You can't"!
You may want to take a look at this discussion.
[EDIT]
The answer to that post states:
You can't (check lock's state) for non-named locks!!!! More info:
http://forums.mysql.com/read.php?21,222363,223774#msg-223774
Row-level locks are not meant for application level locks. They are just means to implement consistent reads and writes. That means you have to release them as soon as possible. You need to implement your own application level lock and it's not that much hard. Perhaps a simple user_id field will do. If it is null then there's no lock. But if it's not null, the id indicates who is holding the record. In this case you'll need row-level locking to update the user_id field. And as I said before, you'll have to release MySQL lock as soon as you are done locking / unlocking the record.
The question's entire premise lies in the rather liberal use of RDBMS' row-level locking (which is usually used for short-lived concurrency control) directly for interactive UI control.
But putting that aside and answering the question, one can set the session's innodb_lock_wait_timeout to a very short value, minimum being 1, and catching the resulting Lock wait timeout exceeded; try restarting transaction when unable to lock.
The exception class was com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException when I just tried with mysql-connector-java 5.1.38, but other exception classes has changed over releases so this too may be different in older version of MySQL Connector/J.
The "attempt and fail" method of acquiring locks is the standard way of tackling these types of concurrency situations, as the alternate method of "check before attempting" is an anti-pattern that creates a race-condition between checking and the actual attempt to lock.