I have a requirement in my Java application to execute a set of DB statements in an atomic & isolated fashion. For example, the application needs to read data rows from one table and update a data row in another table.
QueryRunner queryRunner = new QueryRunner(); // DBUtils query runner
Object[] params = new Object[] { param };
Connection conn = null;
try {
conn = ...; // get connection
conn.setAutoCommit(false);
result = queryRunner.query(conn, "select x, y, z from table1 where column1 = ?", new BeanHandler<SomeBean>(SomeBean.class), params);
// logic to get value for update
queryRunner.update(conn, "update table2 set p = ? where q = ?", some_value, some_id);
conn.commit();
} catch (SQLException e) {
//
} finally {
DBUtils.closeQuietly(conn);
}
The transaction management is achieved by setting auto commit to false for the connection and explicitly commit later on, as shown above. But the above code can also be executed in a multi-thread environment and I also want the two DB statements (select & update) to be run as a whole mutual exclusively.
I have some idea to use a shared Java Lock object in that method, depicted below.
In the class,
private Lock lock = new ReentrantLock(); // member variable
In the method,
lock.lock();
try {
conn = ...; // get connection
conn.setAutoCommit(false);
result = queryRunner.query(conn, "select x, y, z from table1 where column1 = ?", new BeanHandler<SomeBean>(SomeBean.class), params);
// logic to get value for update
queryRunner.update(conn, "update table2 set p = ? where q = ?", some_value, some_id);
conn.commit();
} finally {
DBUtils.closeQuietly(conn);
lock.unlock();
}
It seems sort of capable of solving the issue. However, I am wondering if this is the best practice and is there any better alternative (such as frameworks) for this?
My suggestion is to have the database manage those locks for you instead of your application. This handles the case where there are multiple JVM's running the code. The locking mechanims you mentioned can only be effective in a single JVM.
The way to accomplish this is to do a SELECT ... FOR UPDATE. This will place a lock on the selected rows and the lock will be released when your transaction is committed or rolled back. This is better than a table level lock because those rows can still be read by other transactions that just want to read the current value but not update them. If another transaction tries to obtain a FOR UPDATE lock, then it will block until the first one finishes.
The only way you will achieve the atomicy you are requiring is to use a stored procedure in the database to isolate the data and lock it all at once. The locking at the Java level can't do what the locking in the database can as easily.
Another way you can handle problems like this is to use the serializable transaction isolation level for all of your database transactions. This causes any set of transactions to behave as though they were run one at a time, without actually making them run one at a time. If you do this you should be using a framework which catches serialization failures (SQLState 40001) and retries the transactions. The big up-side is that you don't need to worry about particular interactions among transactions -- if a transaction does the right thing when it is the only thing running, it will do the right thing in any transaction mix.
Note that all transactions must be serializable for this to work so simply.
From my understanding, do you just want to make that block of code containing select and update statements as Thread safe? That's what synchronized keyword is used for. Even though this question is asked long back i just want to make a note of it here. put those lines of code under synchronized block.
Related
I'm trying to set a Lock for the row I'm working on until the next commit:
entityManager.createQuery("SELECT value from Table where id=:id")
.setParameter("id", "123")
.setLockMode(LockModeType.PESSIMISTIC_WRITE)
.setHint("javax.persistence.lock.timeout", 10000)
.getSingleResult();
What I thought should happen is that if two threads will try to write to the db at the same time, one thread will reach the update operation before the other, the second thread should wait 10 seconds and then throw PessimisticLockException.
But instead the thread hangs until the other thread finishes, regardless of the timeout set.
Look at this example :
database.createTransaction(transaction -> {
// Execute the first request to the db, and lock the table
requestAndLock(transaction);
// open another transaction, and execute the second request in
// a different transaction
database.createTransaction(secondTransaction -> {
requestAndLock(secondTransaction);
});
transaction.commit();
});
I expected that in the second request the transaction will wait until the timeout set and then throw the PessimisticLockException, but instead it deadlocks forever.
Hibernate generates my request to the db this way :
SELECT value from Table where id=123 FOR UPDATE
In this answer I saw that Postgres allows only SELECT FOR UPDATE NO WAIT that sets the timeout to 0, but it isn't possible to set a timeout in that way.
Is there any other way that I can use with Hibernate / JPA?
Maybe this way is somehow recommended?
Hibernate supports a bunch of query hints. The one you're using sets the timeout for the query, not for the pessimistic lock. The query and the lock are independent of each other, and you need to use the hint shown below.
But before you do that, please be aware, that Hibernate doesn't handle the timeout itself. It only sends it to the database and it depends on the database, if and how it applies it.
To set a timeout for the pessimistic lock, you need to use the javax.persistence.lock.timeout hint instead. Here's an example:
entityManager.createQuery("SELECT value from Table where id=:id")
.setParameter("id", "123")
.setLockMode(LockModeType.PESSIMISTIC_WRITE)
.setHint("javax.persistence.lock.timeout", 10000)
.getSingleResult();
I think you could try
SET LOCAL lock_timeout = '10s';
SELECT ....;
I doubt Hibernate supports this out-of-box. You could try find a way to extend it, not sure if it worth it. Because I guess using locks on a postges database (which is mvcc) is not the smartest option.
You could also do NO WAIT and delay-retry several times from your code.
There is the lock_timeout parameter that does exactly what you want.
You can set it in postgresql.conf or with ALTER ROLE or ALTER DATABASE per user or per database.
The hint for lock timeout for PostgresSQL doesn't work on PostreSQL 9.6 (.setHint("javax.persistence.lock.timeout", 10000)
The only solution I found is uncommenting lock_timeout property in postgresql.conf:
lock_timeout = 10000 # in milliseconds, 0 is disabled
For anyone who's still looking for a data jpa solution, this is how i managed to do it
First i've created a function in postgres
CREATE function function_name (some_var bigint)
RETURNS TABLE (id BIGINT, counter bigint, organisation_id bigint) -- here you list all the columns you want to be returned in the select statement
LANGUAGE plpgsql
AS
$$
BEGIN
SET LOCAL lock_timeout = '5s';
return query SELECT * from some_table where some_table.id = some_var FOR UPDATE;
END;
$$;
then in the repository interface i've created a native query that calls the function. This will apply the lock timeout on that particular transaction
#Transactional
#Query(value = """
select * from function_name(:id);
""", nativeQuery = true)
Optional<SomeTableEntity> findById(Long id);
I ve got a sqlite database with several tables. One of them called studentsession. Inside my code I am calling plenty times the table for select, insert, updates. After a while I am receiving the message:
java.sql.SQLException: [SQLITE_BUSY] The database file is locked (database is locked)
One instance of the calling is the following:
String query3 = "select * from studentssession where id= ? and math= ? and level = ?";
PreparedStatement pst__ = connectionUsers.prepareStatement(query3);
pst__.setString(1, x);
pst__.setString(2, x1);
pst__.setString(3, x2);
ts2 = pst__.executeQuery();
I am trying to figure out if I have or not to close every time the prepared statement, and if there is a case that this is causing my problems.
EDIT: Is it possible to have a check for possible open references in the database, using for example a javafxbutton?
EDIT: Is there a way that I can check in my code whether there is a problem in the references to the table and locate and possible close them?
It's probably due to you having multiple open references to the sqlite database.
I'd start by closing your PreparedStatement in a finally block inside your while loop.
PreparedStatement pst__ = null;
try{
pst__ = connectionUsers.prepareStatement(query3);
pst__.setString(1, x);
pst__.setString(2, x1);
pst__.setString(3, x2);
ts2 = pst__.executeQuery();
}finally{
if(pst__ != null) {
pst__.close();
}
}
You should also close the database connection at the end of everything.
Also it is a bad practice to use multiple connections when connecting to SQLite. See
http://touchlabblog.tumblr.com/post/24474398246/android-sqlite-locking
Set your poolsize maxactive to 1 and try out.
JDBC has been supporting bulk updates for a long time using addBatch and executeBatch. Why isn't there any support for adding a bunch of prepared statements and getting an array of result sets as response?
For example, if I wanted to load customer details, basic account details, basic card details, basic loan details etc. for a single view, I would prefer to create a bunch of prepared statements and append the prepared statements to an ArrayList and execute them as a batch. I would then loop through the result sets and process the data. Hopefully, several network round trips would be saved (assuming my queries are performant).
Sample bunch of queries:
SELECT custid, first, last, age FROM Customer where custid = ?
SELECT custid, acno, accountname, accounttype, status FROM Account where custid = ?
SELECT custid, cardno, cardname, cardtype, status FROM CreditCard where custid = ?
SELECT custid, loanno, principal, rate FROM Loan where custid = ?
I can imagine several hypothetical reasons why it could be a bad idea. But, I am not sure which is most likely true in the real world.
Hypothetical reasons against having bulk-fetch:
There is some fundamental networking/db stack/memory related issue
which prevents a bunch of select queries to be executed on the same
connection and result-sets kept open.
Response handling code would be too cumbersome, as there could be exceptions at call level and individual statement level. And, several statements would have to be closed correctly.
There is no significant performance gain in reducing the number of network-calls. Query execution is the main bottleneck and network round-trip cost is insignificant.
There could be misuse of such a feature. A single non-performant query batched up like this with other queries could cause application to perform poorly.
The reason I ask this is because often I see a lot of Join queries which merge parent-child relationships into a single SQL query, just for the sake of completing the loading in a single call.
However, as the number of tables grows, the query becomes complex. Also, the parent table information is repeated in every row of every child. So, there is huge amount of data redundancy in the single join-ed result set.
Sample join query:
SELECT custid, first, last, age, acno, accountname, accounttype, a.status, cardno, cardname, cardtype, c.status, loanno, principal, rate
FROM Customer cc, Account a, CreditCard c, Loan l
WHERE a.custid=CC.custid(+) and c.custid=CC.custid(+) and l.custid=CC.custid(+)
The JDBC API does support this.
Statement.getMoreResults() can tell you if the SQL statement you executed through execute() produced more than one ResultSet
Quote from the JavaDocs for getMoreResults():
Moves to this Statement object's next result, returns true if it is a ResultSet object, and implicitly closes any current ResultSet object(s) obtained with the method getResultSet.
There are no more results when the following is true:
// stmt is a Statement object<br>
((stmt.getMoreResults() == false) && (stmt.getUpdateCount() == -1))
However it depends on the backend DBMS and the JDBC driver if you can use this. Some JDBC driver simply reject to run more than one statement with a single execute() call (mainly as a means to prevent SQL injenction), others don't.
So in e.g. Postgres you can do something like this:
boolean hasResult = stmt.execute(
"select * from table_1;\n" +
"select * from table_2;");
while (hasResult)
{
rs = stmt.getResultSet();
while (rs.next())
{
// process the result set
}
hasResult = stmt.getMoreResults();
}
This even allows mixing SELECT and e.g. UPDATE statements if you also check for getUpdateCount()
As far as I know you can also do this with SQL Server. It does not work with Oracle.
I haven't tried this with a PreparedStatement though. But as getMoreResults() is defined for Statement it is available for a PreparedStatement as well.
How about to put queries to a procedure and then use CallableStatement to execute that procedure?
A CallableStatement can return one ResultSet object or multiple
ResultSet objects. Multiple ResultSet objects are handled using
operations inherited from Statement.
try
{
CallableStatement stmt = con.prepareCall(/* call procedure */);
boolean results = stmt.execute();
int rsCount = 0;
while (results)
{
ResultSet rs = stmt.getResultSet();
while (rs.next())
{
}
rs.close();
results = stmt.getMoreResults();
}
stmt.close();
}
catch (Exception e) {
e.printStackTrace();
}
Relational databases are designed and optimized for retrieving data through SQL queries that JOIN data from multiple tables. Executing a single query that (correctly) JOINs data is likely always more efficient than getting the same data with separate queries.
When a single query gets too complex, it should be refactored into a VIEW -- from which you can query, joining data from other TABLEs and VIEWs, if required.
Given the above, I don't see a need for bulk queries.
I get the feeling you don't understand what a prepared statement is.
A prepared statement is an object you declare once, then reuse it all the time with different supplied parameters to it.
You're not telling me that you recreate a prepared statement from scratch each time you wish to execute it again?
Say you have four loops. before executing your loops you do this:
PreparedStatement statement1, statement2, statement3,statement4;
try {
con.setAutoCommit(false);//only needed when also doing updates/inserts
statement1 = con.prepareStatement("SELECT custid, first, last, age FROM Customer where custid = ?");
statement2 = con.prepareStatement("SELECT custid, acno, accountname, accounttype, status FROM Account where custid = ?");
// etc....
for (Map.Entry<String, Integer> e : customers.entrySet()) {
statement1.setInt(1, e.getValue().intValue());
ResultSet rs = statement1.executeQuery();
// do what you need to do
statement2.setInt(1, e.getValue().intValue());
ResultSet rs2 = statement2.executeQuery();
// do what you need to do
}
con.commit();//only needed when also doing updates/inserts
}
}
There is no need to recreate the prepared statements. That is why its calleda prepared statement. You just feed it the new values it needs to query.
This way you can add it to lists, itereate it the way you want to itereate it, etc.. and it's all optimised since the database engine will remember the query plans and the optimisations it makes for it. What you do with the prepared statement object is up to you.
It also does this if you recreate the objects constantly because it will remember the query, but you save the overhead of createing new objects over and over and the memory issues that come with that.
So, without a clearer question this is the best answer I can give you.
I've recently started learning to use myBatis.I am now facing such a scenario, I need to constantly fetch a new list of Objects through WebService, then for this list, I need to insert/update each object into the oracle DB table through myBatis.
The tricky part is, I cannot simply do a batch insert every time, because some of the objects might already exist in DB, for these records, I need to update the fields of them instead of a new insertion.
My current solution might be very stupid, using Java, build the list of Object from webservice, loop through each of them, do a myBatis select, if it is not a null(already exists in the db), then do a myBatis update; otherwise, do a myBatis insert for this new object.
The function is achieved. But my technical lead says it is very low-efficient, since doing a for loop using Java and insert/update one by one will consume a lot of system resource. He advised me to do batch insert using myBatis by passing a list of objects in.
Batch insertion in myBatis is straightforward, however, since I am not purely inserting(for existing records I need to do update), I don't think batch insert is appropriate here. I've googled a while for this, and realized maybe I will need to use "merge" instead of "insert" (for Oracle).
The examples I googled out for merge in myBatis is only for one object, not in a batch. Thus I want to find out whether experts could offer me some examples on how to do a batch-merge in MyBatis( The correct way to write a Mapper)?
The accepted answer is not the recommended way of handling batch operations. It does not show true batch statements since the batch executor mode should be used when opening a session. See this post in which a code contributor recommended that the proper way to batch update (or insert) is to open a session in batch mode and repeatedly call update (or insert) for a single record.
Here's what works for me:
public void updateRecords(final List<GisObject> objectsToUpdate) {
final SqlSession sqlSession = MyBatisUtils.getSqlSessionFactory().openSession(ExecutorType.BATCH);
try {
final GisObjectMapper mapper = sqlSession.getMapper(GisObjectMapper.class);
for (final GisObject gisObject : objectsToUpdate) {
mapper.updateRecord(gisObject);
}
sqlSession.commit();
} finally {
sqlSession.close();
}
}
Do not use foreach in your update/insert and ensure that it only updates/inserts a single record. I was running into unsolvable oracle errors by doing it according to the accepted answer (invalid character, statement not ended, etc.). As the linked post indicates, the update (or insert) shown in the accepted answer is actually just a giant sql statement.
In my case also there is same scenario. I used for loop to check whether this record exists in databse or not and then according to that I added this object in to two arraylist for insert or update.
And then used batch for insert and update after for loop for that to list.
here is ex. for update according to different where condition
1] this is for update
<foreach collection="attendingUsrList" item="model" separator=";">
UPDATE parties SET attending_user_count = #{model.attending_count}
WHERE fb_party_id = #{model.eid}
</foreach>
2] this is for insert
<insert id="insertAccountabilityUsers" parameterType="AccountabilityUsersModel" useGeneratedKeys="false">
INSERT INTO accountability_users
(
accountability_user_id, accountability_id, to_username,
record_status, created_by, created_at, updated_by, updated_at
)
VALUES
<foreach collection="usersList" item="model" separator=",">
(
#{model.accountabilityUserId}, #{model.accountabilityId}, #{model.toUsername},
'A', #{model.createdBy}, #{model.createdAt}, #{model.updatedBy}, #{model.updatedAt}
)
</foreach>
</insert>
In dao method declare as
void insertAccountabilityUsers(#Param("usersList") List<AccountabilityUsersModel> usersList);
Update
Here is my batch session code
public static synchronized SqlSession getSqlBatchSession() {
ConnectionBuilderAction connection = new ConnectionBuilderAction();
sf = connection.getConnection();
SqlSession session = sf.openSession(ExecutorType.BATCH);
return session;
}
SqlSession session = ConnectionBuilderAction.getSqlSession();
Actually I already given full example here for this question
In oracle if you want to execute multiple statements at one time you have to enclose your statements in "begin" and "end" block. So try to add attributes to foreach as below. This will definitely work.
<foreach collection="customerList" item="object" open="begin" close=";end;" separator=";">
UPDATE customer SET isActive = #{object.isactive}
WHERE customerId= #{object.customerId}
</foreach>
So, I have a collection of DTOs that I need to save off. They are backed with a temporary table, and they also need to have their data inserted into a "real" table.
I don't have time to do the proper batch process of these records, and the expected number of results, while it can be theoretically very high, is probably around 50 or less anyways. There are several other issues with this application (it's a real cluster**), so I just want to get something up and running for testing purposes.
I was thinking of doing the following psuedocode (in a transaction):
PreparedStatement insert1 = con.prepareStatement(...);
PreparedStatement insert2 = con.prepareStatement(...);
for(DTO dto : dtos) {
prepareFirstInsertWithParameters(insert1, dto);
insert1.executeUpdate();
prepareSecondInsertWithParameters(insert2, dto);
insert2.executeUpdate();
}
FIrst off, will this work as is - can I reuse the prepared statement without executing clearParameters(), or do I have to do a close() on them, or keep getting more prepared statements?
Secondly, aside from batching, is there a more efficient (and cleaner) way of doing this?
This is easy:
conn = dataSource.getConnection();
conn.setAutoCommit( false );
pStatement = conn.prepareStatement( sqlStr );
ListIterator<DTO> dtoIterator = dtoList.listIterator();
while( dtoIterator.hasNext() ) {
DTO myDTO = dtoIterator.next();
pStatement.setInt( 1, myDTO.getFlibble() );
pStatement.setInt( 2, myDTO.getNuts() );
pStatement.addBatch();
}
int[] recordCount = pStatement.executeBatch();
conn.commit();
MetroidFan2002,
I don't know what you mean by 'aside from batching', but I'm assuming you mean executing a single batch SQL statement. You can however, batch the prepared statement calls which will improve performance by submitting multiple calls at a time:
PreparedStatement insert1 = con.prepareStatement(...);
PreparedStatement insert2 = con.prepareStatement(...);
for(DTO dto : dtos) {
prepareFirstInsertWithParameters(insert1, dto);
prepareSecondInsertWithParameters(insert2, dto);
insert1.addBatch();
insert2.addBatch();
}
insert1.executeBatch();
insert2.executeBatch();
// cleanup
Now if your dataset can grow large, like you alluded to, you'll want to put some logic in to flush the batch every N number of rows, where N is a value tuned to the optimal performance for your setup.
JDBC supports Batch Insert/Update. See example here.