I've few scheduled processes which perform on the same row. Actually this row is a SMS needed to be sent. First process is #Asynchronious process which invoked when I press send button and second process is #Scheduled process which invoked once per minute. After I've written a SMS into DB it has status 0. After I've successfully or unsuccessfully sent it, it will have status 2 or 3 accordingly. The problem that when I read a SMS from the DB2 but before I actually sent it the second process can read and send it too. So, my question how can I prevent it? Can I lock specific row for reading? Another way is to add additional status 'processing' and change this status when I read a specific row, but I doubt is is possible to read and to write simultaneously? I though also about global flag which will indicate that this process can't be run, but I'm not only looking for solution I also looking for the most correct solution. Thank you very much!
And sorry for my English guys, if somebody will edit it I'll be grateful.
You cannot reliably prevent other applications from reading a row locked by one application. Queries running with the uncommitted read isolation level will be able to access even exclusively locked rows. Queries running with the cursor stability isolation level using the currently committed semantics will access the previous version of the exclusively locked row.
I think your best choice is to use a special value (like 'processing') in the status column to prevent other sessions from processing the same row.
I'm assuming you're using DB2 for Linux/Unix/Windows, since you don't mention a platform.
There is a way to have DB2 lock a row on a SELECT, see the isolation-clause and lock-request-clause on this Information Center page.
When specifying an isolation level of either RR (repeatable read) or RS (read stability), you can also say that you want to lock the rows that are read, either with a SHARE, UPDATE, or EXCLUSIVE lock.
That statement will lock the row as long as the transaction is active.
Ok, I actually solved it by creating a singleton with following flag public static boolean busy; first method which run change its value to true and while it's true second method doesn't run. While first method completed it changes it back to false.
Related
In my application I have the problem that sometimes SELECT statements run into a java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction exception. Sadly I can't create an example as the circumstances are very complex. So the question is just about general understanding.
A little bit of background information: I'm using MySQL (InnoDB) with READ_COMMITED isolation level.
Actually I don't understand how a SELECT can ever run into a lock timeout with that setup. I thought that a SELECT would never lock as it will just return the latest commited state (managed by MySQL). Anyway according to what is happening this seems to be wrong. So how is it really?
I already read this https://dev.mysql.com/doc/refman/8.0/en/innodb-locking.html but that didn't really give me a clue. No SELECT ... FOR UPDATE or something like that is used.
That is probably due to your database. Usually this kind of problems come from that side, not from the programming side that access it.In my experience with db's, these problems are usually due to that. In the end, the programming side is just "go and get that for me in that db" thing.
I found this without much effort.
It basically explains that:
Lock wait timeout occurs typically when a transaction is waiting on row(s) of data to update which is already been locked by some other transaction.
You should also check this answer that has a specific transaction problem, which might help you, as trying to change different tables might do the timeout
the query was attempting to change at least one row in one or more InnoDB tables. Since you know the query, all the tables being accessed are candidates for being the culprit.
To speed up queries in a DB, several transactions can be executed at the same time. For example if someone runs a select query over a table for the wages of the employees of a company (each employee identified by an id) and another one changes the last name of someone who e.g. has married, you can execute both queries at the same time because they don't interfere.
But in other cases even a SELECT statement might interfere with another statement.
To prevent unexpected results in a SQL transactions, transactions follow the ACID-model which stands for Atomicity, Consistency, Isolation and Durability (for further information read wikipedia).
Let's say transaction 1 starts to calculate something and then wants to write the results to table A. Before writing it it locks all SELECT statements to table A. Otherwise this would interfere with the Isolation requirement. Because if a transaction 2 would start while 1 is still writing, 2's results depend on where 1 has already written to and where not.
Now, it might even produce a dead-lock. E.g. before transaction 1 can write the last field in table A, it still has to write something to table B but transaction 2 has already blocked table B to read safely from it after it read from A and now you have a deadlock. 2 wants to read from A which is blocked by 1, so it waits for 1 to finish but 1 waits for 2 to unlock table B to finish by itself.
To solve this problem one strategy is to rollback certain transactions after a certain timeout. (more here)
So that might be a read on for your select statement to get a lock wait timeout exceeded.
But a dead-lock usually just happens by coincidence, so if transaction 2 was forced to rollback, transaction 1 should be able to finish so that 2 should be able to succeed on a later try.
I have a multi threaded Java program where each thread gets one username for some processing which takes about 10 minutes or so.
Right now it's getting the usernames by a sql query that returns one username randomly and the problem is that the same username can be given to more than one thread at a time.
I don't want a username that is being processed by a thread, to be fetched again by another thread. What is a simple and easy way to achieve this goal?
Step-by-step solution:
Create a threads table where you store the threads' state. Among other columns, you need to store the owner user's id there as well.
When a thread is associated to a user, create a record, storing the owner, along with all other juicy stuff.
When a thread is no longer associated to a user, set its owner to null.
When a thread finishes its job, remove its record.
When you randomize your user for threads, filter out all the users who are already associated to at least a thread. This way you know any users at the end of randomization are threadless.
Make sure everything is in place. If, while working on the feature some thread records were created and should be removed or disposed from its owner, then do so.
There is a lot of ways to do this... I can think of three solution to this problem:
1) A singleton class with a array that contains all the user already in use. Be sure that the acces to the array is synchronized and you remove the unused users from it.
2) A flag in the user table that contains a unique Id referencing the thread that is using it. After you have to manage when you remove the flag from the table.
-> As an alternative, why do you check if a pool of connections shared by all the thread could be the solution to your problem ?
You could do one batch query that returns all of the usernames you want from the database and store them in a List (or some type of collection).
Then ensure synchronised access to this list to prevent two threads taking the same username at the same time. Use a synchronised list or a synchronised method to access the list and remove the username from the list.
One way to do it is to add another column to your users table.this column is a simple flag that shows if a user has an assigned thread or not.
but when you query the db you have to wrap it in a transaction.
you begin the transaction and then first you select a user that doesn't have a thread after that you update the flag column and then you commit or roll back.
since the queries are wrapped in a transaction the db handles all the issues that happen in scenarios like this.
with this solution there is no need to implement synchronization mechanisms in your code since the database will do it for you.
if you still have problems after doing this i think you have to configure isolation levels of your db server.
You appear to want a work queue system. Don't reinvent the wheel - use a well established existing work queue.
Robust, reliable concurrent work queuing is unfortunately tricky with relational databases. Most "solutions" land up:
Failing to cope with work items not being completed due to a worker restart or crash;
Actually land up serializing all work on a lock, so all but one worker are just waiting; and/or
Allowing a work item to be processed more than once
PostgreSQL 9.5's new FOR UPDATE SKIP LOCKED feature will make it easier to do what you want in the database. For now, use a canned reliable task/work/message queue engine.
If you must do this yourself, you'll want to have a table of active work items where you record the active process ID / thread ID of the worker that's processing a row. You will need a cleanup process that runs periodically, on thread crash, and on program startup that removes entries for failed jobs (where the worker process no longer exists) so they can be re-tried.
Note that unless the work the workers do is committed to the database in the same transaction that marks the work queue item as done, you will have timing issues where the work can be completed then the DB entry for it isn't marked as done, leading to work being repeated. To absolutely prevent that requires that you commit the work to the DB in the same transaction as the change that marks the work as done, or that you use two-phase commit and an external transaction manager.
Is it advisable to query a database continuously in a loop, to get any new data which is added to specific table?
I have below a piece of code:
while(true)
try{
// get connection
// execute only "SELECT" query
}
catch(Exception e){}
finally{// close connection
}
//Sleep 5 sec's
}
It is a simple approach that works in many cases. Make sure that the select statement you use doesn't put as little load as possible on the database.
The better (but more difficult to setup) variant would be either to use some mechanism to get actively informed by the database about changes. Some databases can for example can send information with some queuing mechanism, which again could be triggered using a database trigger.
Querying database in loop is not advisable but if you need the same you can daemonize your program.
If longer then 5 s a timer would be appropriate.
For a kind of staying totally up-to-date:
Triggers and cascading inserts/deletes can propagate data inside the database itself.
Otherwise before altering the database, issue messages in a message queue. This not necessarily needs to be a Message Queue (capitals) but can be any kind of queue, like a publish/subscribe mechanism or whatever.
On one hand, if your database has a low change rate then it would be better to use/implement a notification system. Many RDBMS have notification features (Oracle's Database Change Notification, Postgres' Asynchronous Notifications, ...), and if your RDBMS does not have them, it is easy to implement/emulate using triggers (if your RDBMS support them).
On the other hand, if the change rate is very high then your solution is preferable. But you need to adjust carefully the interval time and you must note: reading on intervals to detect changes has a negative collateral effect.
Using/implementing a notification system it is easy to inform the program what has been changed. (A new row X inserted on table A, a new updated row Y on table B, …).
But if you read your data on intervals, it is not easy to determine what has been changed. Then you have two options:
a) you must not only read but load/process all information every interval;
b) or you must not only read but compare database data with memory resident data to determine what has changed every interval.
I have a J2EE server, currently running only one thread (the problem arises even within one single request) to save its internal model of data to MySQL/INNODB-tables.
Basic idea is to read data from flat files, do a lot of calculation and then write the result to MySQL. Read another set of flat files for the next day and repeat with step 1. As only a minor part of the rows change, I use a recordset of already written rows, compare to the current result in memory and then update/insert it correspondingly (no delete, just setting a deletedFlag).
Problem: Despite a purely sequential process I get lock timeout errors (#1204) and Innodump show record locks (though I do not know how to figure the details). To complicate things under my windows machine everything works, while the production system (where I can't install innotop) has some record locks.
To the critical code:
Read data and calculate (works)
Get Connection from Tomcat Pool and set to autocommit=false
Use Statement to issue "LOCK TABLES order WRITE"
Open Recordset (Updateable) on table order
For each row in Recordset --> if difference, update from in-memory-object
For objects not yet in the database --> Insert data
Commit Connection, Close Connection
The Steps 5/6 have an Commitcounter so that every 500 changes the rows are committed (to avoid having 50.000 rows uncommitted). In the first run (so w/o any locks) this takes max. 30sec / table.
As stated above right now I avoid any other interaction with the database, but it in future other processes (user requests) might read data or even write some fields. I would not mind for those processes to read either old/new data and to wait for a couple of minutes to save changes to the db (that is for a lock).
I would be happy to any recommendation to do better than that.
Summary: Complex code calculates in-memory objects which are to be synchronized with database. This sync currently seems to lock itself despite the fact that it sequentially locks, changes unlocks the tables without any exceptions thrown. But for some reason row locks seem to remain.
Kind regards
Additional information:
Mysql: show processlist lists no active connections (all asleep or alternatively waiting for table locks on table order) while "show engine INNODB" reports a number of row locks (unfortuantely I can't understand which transaction is meant as output is quite cryptic).
Solved: I wrongly declared a ResultSet as updateable. The ResultSet was closed only on a "finalize()" method via Garbage Collector which was not fast enough - before I reopended the ResultSet and tried therefore to aquire a lock on an already locked table.
Yet it was odd, that innotop showed another query of mine to hang on a completely different table. Though as it works for me, I do not care about oddities:-)
I have a Java frontend and a MySQL backend scenario, I used a 'LOCK IN SHARE MODE' for SELECT. If I request the same row from another process, it gives the data.. However it does not allow me to update. What I would like to do is inform the user they will only have a READ only copy, so if they wish to see the information they can else they can request it later.. How could I check the status of the ROW so that the user will be informed about this situation?? If I use 'FOR UPDATE', It just waits for until the first user saves the data. I find it less user friendly, if they just have a blank screen or when they click button it does nothing. Any help will be greatly appreciated. Using MySQL 5.5, Java 7.
The short answer is "You can't"!
You may want to take a look at this discussion.
[EDIT]
The answer to that post states:
You can't (check lock's state) for non-named locks!!!! More info:
http://forums.mysql.com/read.php?21,222363,223774#msg-223774
Row-level locks are not meant for application level locks. They are just means to implement consistent reads and writes. That means you have to release them as soon as possible. You need to implement your own application level lock and it's not that much hard. Perhaps a simple user_id field will do. If it is null then there's no lock. But if it's not null, the id indicates who is holding the record. In this case you'll need row-level locking to update the user_id field. And as I said before, you'll have to release MySQL lock as soon as you are done locking / unlocking the record.
The question's entire premise lies in the rather liberal use of RDBMS' row-level locking (which is usually used for short-lived concurrency control) directly for interactive UI control.
But putting that aside and answering the question, one can set the session's innodb_lock_wait_timeout to a very short value, minimum being 1, and catching the resulting Lock wait timeout exceeded; try restarting transaction when unable to lock.
The exception class was com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException when I just tried with mysql-connector-java 5.1.38, but other exception classes has changed over releases so this too may be different in older version of MySQL Connector/J.
The "attempt and fail" method of acquiring locks is the standard way of tackling these types of concurrency situations, as the alternate method of "check before attempting" is an anti-pattern that creates a race-condition between checking and the actual attempt to lock.