I have a spring application which reads data from the Database and sends it to system 'X'. I am using task executors to spin up threads, so there are like 5 threads which are reading the database for rows at the same time. For each thread I need to make sure that unique records are selected.
To achieve this I am using JdbcTemplate and "select for update"
I have written the code but in the logs I am able to see 2 threads picking up the same rows. I am not able to figure out the root cause of this issue.
Does anyone has a suggestion
try {
List<Map<String, Object>> rows = getJdbcTemplate().queryForList(
SELECT_FOR_UPDATE,
new Object[] {a,b,c,d});
for (Map<String,Object> row : rows) {
Header a = new Header();
a.setMailID(((BigDecimal)row.get("mailID")).intValue());
a.setVersion(((BigDecimal)row.get("version")).intValue());
// some other parameters to get
getJdbcTemplate().update(UPDATE_MSG_STATE_VERSION_N_ORIG_MSG_STAT,
x,
a.getVersion()+1,
y),
a.getMailID(),
a.getVersion());
headers.add(a);
}
}
UPDATE_MSG_STATE_VERSION_N_ORIG_MSG_STAT = update MESSAGE set MSG_STAT_CD = ?, VERSION_NBR = ?, ORIG_MSG_STAT_CD=?, LAST_UPD_TS=SYSTIMESTAMP where MESSAGE.MAIL_ID = ? and VERSION_NBR = ?
String SELECT_FOR_UPDATE = "select m.MAIL_ID mailID, m.VERSION_NBR version, m.MSG_STAT_CD state,"
+ "from message m "
+ "and m.MSG_STAT_CD in ('Nerwerw')"
+ " and m.create_ts > (sysdate - ?)"
+ " and mod(mail_id,?) = ?"
+ " and ROWNUM <= ?"
+ " order by mt.MSG_PRIORITY FOR UPDATE";
You need to annotate your class with #Repostitory tag and the #Transactional tag to make sure that all the actions in the same call are handled in one transaction.
If they are not handled in the same transaction then each SELECT_FOR_UPDATE will happen on a different transaction and thus your threads queries will not be syncronized and your select_for_update does not matter.
Have you had transaction control properly set up?
If not, the transaction will only happen for the duration of the update statement, and will be committed automatically (You are using Oracle I believe, base on your syntax).
That means, although you acquired the lock of those records, they are released right-away.
Do you have access to modify the database? If I understand your question correctly I recently had a similar problem and implemented a scheme like this:
Add a new column to your database like "thread_number" or something like that. Set it to some default value like 0. Give each thread a unique identifier. Then you "claim" a record in the database by updating its "thread_number" to the identifier of the thread processing it. Then the other threads will not find it when querying if you include "where thread_number = 0" in the SQL.
I know it's kind of broad, but I hope it helps.
Related
I'm using jdbi (but would prepared to use raw jdbc if needed). My DB is currently Oracle. I have an update that updates the first row matching certain criteria. I want to get the primary key of the updated row from the same statement. Is this possible?
I tried
Integer rowNum = handle
.createUpdate(sqlFindUpdate)
.bind("some var", myVal)
.executeAndReturnGeneratedKeys("id")
.mapTo(Integer.class)
.findOnly();
but I guess this isn't a generated key, as it doesn't find it (illegal state exception, but the update succeeds).
Basically, I have a list of items in the DB that I need to process. So, I want to get the next and mark it as "in progress" at the same time. I'd like to be able to support multiple worker threads, so it needs to be a single statement - I can't do the select after (the status has changed so it won't match anymore) and doing it before introduces a race condition.
I guess I could do a stored procedure that uses returning into but can I do it directly from java?
I'm answering my own question, but I don't think it's a good answer :) What I'm doing is kind of a hybrid. It is possible to dynamically run PL/SQL blocks from jdbi. Technically, this is from Java as I had asked, not via a stored procedure. However, it's kind of a hack, in my opinion - in this case why not just create the stored procedure (as I probably will, if I don't find a better solution). But, for info, instead of:
String sql = "update foo set status = 1 where rownr in (select rownr from (select rownr from foo where runid = :runid and status = 0 order by rownr) where rownum = 1)";
return jdbi.withHandle((handle) -> {
handle
.createUpdate(sql)
.bind("runid", runId)
.executeAndReturnGeneratedKeys("rownr")
.mapTo(Integer.class)
.findOnly();
});
you can do
String sql = "declare\n" +
"vRownr foo.rownr%type;\n" +
"begin\n" +
"update foo set status = 1 where rownr in (select rownr from (select rownr from foo where runid = :runid and status = 0 order by rownr) where rownum = 1) returning rownr into vRownr;\n" +
":rownr := vRownr;\n" +
"end;";
return jdbi.withHandle((handle) -> {
OutParameters params = handle
.createCall(sql)
.bind("runid", runId)
.registerOutParameter("rownr", Types.INTEGER)
.invoke();
return params.getInt("rownr");
});
Like I said, it's probably better to just create the procedure in this case, but it does give you the option to still build the SQL dynamically in java if you need to I guess.
Based on this question, as linked by #APC in the comments, it is possible to use the OracleReturning class without the declare/begin/end.
String sql = "update foo set status = 1 where rownr in (select rownr from (select rownr from foo where runid = ? and status = 0 order by rownr) where rownum = 1) returning rownr into ?";
return jdbi.withHandle((handle) -> {
handle
.createUpdate(sql)
.bind(0, runId)
.addCustomizer(OracleReturning.returnParameters().register(1, OracleTypes.INTEGER))
.execute(OracleReturning.returningDml())
.mapTo(Integer.class)
.findOnly();
});
However, OracleReturning doesn't support named parameters, so you have to use positionals. Since my main reason for using JDBI over plain JDBC is to get named parameter support, that's important to me, so I'm not sure which way I'll go
Pretty hard dependency on it being an Oracle DB you're calling...
Update: enhancement for named parameters in OracleReturning was merged to master, and will be included in 3.1.0 release. Kudos to #qualidafial for the patch
I have looked around for this answer, but I could not find anything that answered my question.
I am working on an application that uses Hibernate and it uses session.merge(object) to do the update or insert of the object. The insert works fine, but the update will fail due to a unique constraint on database fields (A, B, C) if multiple records exist with the same values for (A, B). The Hibernate model only has fields (A, B) defined as the id, it does not have (A, B, C) because when selecting or updating records only the record with a null value of C is wanted to be returned (C is a termination date where null means active and non-null means not active).
In the Hibernate model file (Table.hbm.xml), it has a where clause defined as follows:
<class name="..." table="..." lazy="true" batch-size="10" where="C is null">
That gets inserted when doing selects, but when doing the merge statement, the update statement does not have this as part of the where clause. The generated update is something like:
update table
set ...
where A=?
and B=?
That is fine, but what I would like is to also have the where clause from the Hibernate model file (the C is null clause) to be added to the where clause like it is for select statements.
Does anyone know what I can do to get that added to the update statement?
Thank you for your help.
I do not think it is possible to update the where clause in the update statement from the merge by using the where clause from the Hibernate model.
What I ended up doing to solve the problem was to do something like this:
void update(EntityName entity){
session.evict(entity);
Query update = session.createSQLQuery(
"update table_name "
+ "set c1 = ?,"
+ "c2 = ? "
+ "where A = ? "
+ "and B = ? "
+ "and C is null" //this was the line that was needing to be added
);
//set the parameter values
if(update.executeUpdate() == 0){
session.save(entity);
}
}
So I had to evict the entity to prevent any other updates from triggering an update later. Then an update was performed, and if no rows were updated, an insert is performed.
order item table
order_item_id
order_id
quantity
unit_price
shipping_price
business_id
workflow_id
delivery_id
item_id
Orders table
billing_address_id
shipping_address_id
payment_mode
total_price
shipping_price
customer_id
UPDATE `order_items` t1 INNER JOIN Orders t2 ON t2.order_id = t1.order_id SET t1.workflow_id = ? WHERE t1.order_item_id = ? and t2.order_id = ? and t2.customer_id = ? and t1.delivery_id = ?
UPDATE `order_items` t1 SET t1.workflow_id = ?
WHERE t1.order_item_id = ? and t1.business_id = ? and t1.delivery_id = ?
UPDATE `order_items` t1 INNER JOIN Orders t2 ON t2.order_id = t1.order_id SET t1.workflow_id = ? WHERE t1.order_item_id = ? and t2.order_id = ? and t1.delivery_id = ?"
These queries are fired on different scenarios from my java rest service. (at any point of time, only one query will be used).
Previously I didn't use the inner join in my update sql and it worked well.
Now after I modified the query, it throws the following exception and the query is stuck and doesn't return for a minute.
java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:996)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
UPDATE
This was happening because we forgot to set the autocommit mode to true again in the finally block. After which we didnt see this error.
Point1: You should not update query with join in application instead get primary key and then update the table based on primary key.
Point2: Show your tables structure with index you can get details by "show create table mytable" command, so that it can be checked that your update query is optimized or not.
Point3: If still you want to update based on join due to any specific reason and also your query is optimized then need to change your wait_timeout variable. So check what value is set in this variable on your server. you can check by below command-
SHOW GLOBAL VARIABLES LIKE 'wait_timeout';
A good thing before running UPDATE query is to run same SELECT. I.e.
SELECT * FROM `order_items` t1 INNER JOIN Orders t2 ON t2.order_id = t1.order_id SET t1.workflow_id = ? WHERE t1.order_item_id = ? and t2.order_id = ? and t2.customer_id = ? and t1.delivery_id = ?
Just to make sure, you are updating the right row.
You can also run EXPLAIN on that query to find out how complicated it is for your DB.
I am fairly new in SQL(now working on SQLITE application) and it is a section in my app when i try this piece of Code:
public void addSong(LibrarySong song){
for(int i=0; i<intoPanel.getComponentCount(); i++) //Check if doublicates exist
if(song.getName().equals(intoPanel.getComponent(i).getName()))
return;
intoPanel.add(song);
//Add the song to the database table
try{
Container.DataBase.connection.prepareStatement("INSERT INTO '"
+ Container.libWindow.libInfoWindow.currentLib.getName()+ "'" //Table Name
+ " (PATH,STARS,DATE,HOUR) VALUES ('"
+ song.getName() + "'," + song.stars + ",'"
+ song.dateCreated + "','" + song.hourCreated + "')").executeUpdate();
}catch(SQLException sql){ sql.printStackTrace(); };
}
The Problem:
The above method just add the song to a Jtable and then to database table.The problem is that the performance is too bad for the database.Why might this happen? i use the statement somewhere wrong or i have to to the update with different way?Thanks for reply.
The most expensive part of accessing a database is not the execution of the statement itself, but all the synchronizations done for transactions.
By default, each SQL command is put into an automatic transaction, so you get the overhead for all of them.
If you have multiple updates, you should group them into a single transaction:
Container.DataBase.connection.setAutoCommit(false);
...
for (...)
addSong(...);
Container.DataBase.connection.commit();
Basically the problem boils down to that every write to disc is done by sqlite itself in default mode
you could enable
PRAGMA journal_mode = WAL
PRAGMA synchronous = NORMAL
To make use of the operating system disc buffer.
Just remember to flush/commit everything regularly or at break points after inserts. Only risk is if there is sudden powerloss or reboot, your database might end up corrupted.
https://www.sqlite.org/atomiccommit.html
I think, that the component usage is suboptimal: the first for-loop.
Use a
Set<String> songNames = new HashSet<>();
if (songNames.contains(song.getName())) { return; }
songNames.add(song.getName());
Or maybe a Map for other uses.
And use the prepared statement with placeholders ?. This escapes single quotes too. And is safer.
I have an entity, let's call it X. (the entity has a #Id id).
The entity is mapped to a table, let's call it: X_TABLE.
It also contains a
#ManyToOne
#JoinColumn(name = "JOIN_COLUMN")
YClass joinColumn;//another entity
I'm working with the following code:
X x = new X(xDTO);//just a copy constructor, nothing fancy, without id
x.joinColumn = null;
entityManager.persit(x);
entityManager.flush();
join_column_id = somne_function();//irelevant
Query q = entityManager.createNativeQuery(
"UPDATE X_TABLE t SET t.JOIN_COLUMN=" + join_column_id + " WHERE " + " t.id= " + x.getId());
q.executeUpdate();
q.flush();
The point in doing this, in case anyone is wondering, is that join_column is a foreign key to another table, in a different database, and I'm setting it through a native query because the coresponding table does not exist in the same database containing X_TABLE. But this isn't the problem. The enitity mappings and relations are all fine, I can garantee for that.
The code above is executed for several records in a loop, and it sometimes works fine (saves and updates the table fine) for all records. But sometimes it works fine for all records except for the last one in the loop (in the sense that it saves the record, but does not execute the update in the database). There are no Exceptions thrown and the JBoss server debug log for hibernate shows the UPDATE happening with the right values, with no error. Yet, the table contains null in JOIN_COLUMN for just one record and I have no ideea why. Any thoughts?
PS: The service is a Container Managed Transaction.
Few suggestions:
Check the number of rows affected by executeUpdate()
What is your Transaction policy? Any TransactionManager defined?